|
 
 |
| ORIGINAL ARTICLE |
|
| Year : 2020 | Volume
: 1
| Issue : 2 | Page : 85-91 |
|
Changing paradigm of practical assessment modalities in medical education in the era of COVID-19: A cross-sectional study
Sanjeev Kumar Mittal1, Ranjeeta Kumari2, Rajesh Kathrotia3, Manisha Bisht4, Sunita Mittal3, Prashant Manohar Patil3, Ashish Bhute5, Bandhubaigari Hidayathulla2, Vartika Saxena2, Manoj Gupta6, Ravi Kant7
1 Department of Ophthalmology, AIIMS, Rishikesh, Uttarakhand, India
2 Department of Community and Family Medicine, AIIMS, Rishikesh, Uttarakhand, India
3 Department of Physiology, AIIMS, Rishikesh, Uttarakhand, India
4 Department of Pharmacology, AIIMS, Rishikesh, Uttarakhand, India
5 Department of Forensic Medicine and Toxicology, AIIMS, Rishikesh, Uttarakhand, India
6 Department of Radiation Oncology, AIIMS, Rishikesh, Uttarakhand, India
7 Director, AIIMS, Rishikesh, Uttarakhand, India
| Date of Submission | 11-Aug-2020 |
| Date of Decision | 21-Sep-2020 |
| Date of Acceptance | 30-Sep-2020 |
| Date of Web Publication | 15-Dec-2020 |
Correspondence Address:
Dr. Ranjeeta Kumari
Department of Community and Family Medicine, AIIMS, Rishikesh, Uttarakhand
India
 Source of Support: None, Conflict of Interest: None
DOI: 10.4103/JME.JME_145_20
Introduction: Medical education is a process of continuous learning and assessments, which are essential benchmarks in the whole education process. In view of COVID 19 pandemic, institute took an astute step of conducting video conferencing mode of practical examination (VCMPE) through online video platforms to reduce risk of transmission. This study was designed to undertake a SWOT analysis of VCMPE from the perspective of students and examiners. Methods: A cross sectional descriptive study was conducted from May to July 2020, among all final year postgraduate students and their respective examiners. A total of 95 students and 80 examiners were included in the study. Data collection was done through online google forms. Responses to questions were expected in the format of Likert scale and as open-ended responses, which were grouped under common themes and reported as percentages. Results: Response rate of students was 66.3%, while that of examiners was 28.7%. Interaction with examiners, ease of interactions during exam, ability to understand conversation and overall experience regarding the VCMPE was rated to be good to very good by 77.8% to 92% of the students. The major weakness stated by both students and examiners was related to internet connectivity issues. Examiners reported limitations with respect to complete assessment of all domains and some were concerned with fairness of examination in VCMPE. The main strength reported was the potential of VCMPE to prevent transmission of COVID19 disease and its cost effectiveness. Conclusions: VCMPE is an innovative approach towards summative assessment of practical knowledge and skills of postgraduate students in the era of COVID19 with a primary aim to prevent transmission of COVID 19. VCMPE may be a workable alternative during critical situations like pandemic, genuine reason of candidate's absence to appear in practical examination or in unprecedented absence of one of the external examiner
Keywords: Assessment, medical education, online assessment, SWOT analysis, video conferencing mode of practical examination
How to cite this article:
Mittal SK, Kumari R, Kathrotia R, Bisht M, Mittal S, Patil PM, Bhute A, Hidayathulla B, Saxena V, Gupta M, Kant R. Changing paradigm of practical assessment modalities in medical education in the era of COVID-19: A cross-sectional study. J Med Evid 2020;1:85-91 |
How to cite this URL:
Mittal SK, Kumari R, Kathrotia R, Bisht M, Mittal S, Patil PM, Bhute A, Hidayathulla B, Saxena V, Gupta M, Kant R. Changing paradigm of practical assessment modalities in medical education in the era of COVID-19: A cross-sectional study. J Med Evid [serial online] 2020 [cited 2023 Jun 7];1:85-91. Available from: http://www.journaljme.org/text.asp?2020/1/2/85/303551 |
| Introduction | |  |
Medical education is a process of continuous learning and assessments to evaluate for adequacy in knowledge and skills acquired by a student, to enable him/her to treat a patient efficiently and in an empathetic manner. Medical education in India consists of an undergraduate course of 5½ years' duration including internship. The process of learning of a medical graduate continues during post-graduation and subsequently during further specialisation courses. In fact, a doctor continues to learn throughout his/her life due to the ever-evolving nature of science, to be able to provide the best possible treatment to patients.[1]
Assessments and evaluations of undergraduate and post-graduate (PG) courses in India are essential benchmarks in the whole education process and consist of formative and summative components, covered under both theory and practical aspects of a subject. Theory papers consist of the evaluation of knowledge as cognitive domain and its application in the treatment of patients, whereas, practical examination focuses more on skills, encompassing assessment of process of interaction with patients, cognitive, affective and psychomotor domains, along with the communication skills. Medical practical examination is crucial to assess the knowledge and skills of students regarding various aspects of the subject, communication skills and how they deal with the patient's illness in real world. These skills are an integral component of their future engagements with patients while they provide care and treatment to them.[2]
Since the beginning of modern medical education, this examination was being conducted with a patient in hospital or community settings, and was evaluated by a team of examiners. Traditional mode of practical examination (TMPE) consisted of face-to-face interaction between patient, student and examiner, which was the norm. To undertake an unbiased assessment of student's knowledge and skills, assessments are done by external examiners from other institutes, along with internal examiners. This evaluates student's aptitude, logical thinking and behaviour by observing verbal and nonverbal cues. There was minimal or no use of technology in this pattern of examination.
In January 2020, an unprecedented viral respiratory infection, named COVID-19 emerged as a pandemic. According to the current evidence, it has been reported to spread through respiratory droplets and contact routes and is reported to have high secondary attack rate.[3] To control the spread of this infection, social distancing, lock downs, travel ban and prohibition of public gatherings was implemented by governments worldwide. General OPD in all hospitals were shut down and telemedicine consultations were provided to the patients.[4] These measures affected medical education and practical exams to a great extent, worldwide.[5],[6],[7] Travel ban and social distancing made it impossible for the patient to reach the hospital and for the external examiners to travel to different institutes for examination. On the other hand, PG students were required to complete their tenure of 3 years in a stipulated time and appear for final examination for being eligible to appear in super specialty entrance exams. Conducting exit examination for these students was therefore an utmost priority. In the midst of all these challenges, our institute considered academic activities of the PG medical students and their career as a priority and took an astute step of conducting video conferencing mode of practical examination (VCMPE) through online video platforms such as Google meet and Skype.
Many studies have reported advantages and disadvantages of online conduction of examinations. While few have favored it, others have reservations against this mode of exam.[8],[9],[10],[11] To the best of our knowledge, there is no study, which has explored medical online examination, especially practical examinations in Indian context. This study was therefore, planned with an aim to assess the strengths and weaknesses of VCMPE from the perspective of examinees and the examiners. This study delves into experiences and suggestions of examinees and examiners so that we can explore new opportunities and overcome challenges while conducting examinations using this approach, in future.
| Methodology | | |
Study design
Cross-sectional descriptive study conducted through an online mode.
Study settings
AIIMS Rishikesh.
Study period
May to July 2020.
Participants
All final year PG students appearing in the examination, internal and external examiners.
Inclusion criteria
All final year PG students who took their examination May 2020, internal and external examiners.
Exclusion criteria
Those who did not give consent to participate in the study.
Study size
Google forms were sent to 95 PG students, who had appeared for the exam, on their E-mail ids. Examinations were conducted for twenty PG subjects, with each one of them having two internal and two external examiners. Eighty faculty members were involved in the examination as examiners. Separate Google forms were sent to these faculty members also.
Outcomes
Strengths, weaknesses, opportunities, challenges and experiences of online conduction of examinations were assessed from the perspective of both the examinee and examiners.
Tools of data collection
A questionnaire was prepared to assess the outcomes, after consultation with three experts from within the institute, who were involved in the process of online examination to achieve content validity. The final questionnaire included questions based on recommendations of all three experts. Responses to questions were expected in the format of Likert scale, visual analogue scale, check boxes and as open-ended responses.
Data collection
Data collection was done through online Google forms. An invitation was sent to all the participants with a link of the Google form, requesting their feedback after the examination was over. This invitation also contained their implied consent for participating in the study, considering the ethical issues. We had sent three reminders to the participants for obtaining responses.
Bias
Selection bias was overcome, by including all students who had appeared in the practical examination and all the examiners who were involved in conducting online practical examination. Due to absence of a validated tool to do SWOT analysis of this mode of examination, we could not use a pre-existing validated tool. However, we tried to enhance the content validity of the tool by taking the inputs of three experts. We tried to control for nonresponse bias by sending three reminders for data collection.
Statistical methods
Descriptive statistics included calculation of proportions for categorical variables. Most of the responses were in the form of Likert scale or visual analogue scale, and were reported as proportions. The responses to open ended questions were grouped under common themes and reported as percentages.
Ethical issues and confidentiality of data
Ethical permission to conduct the study was obtained from Institutional Ethics committee (No. 297/IEC/IM/NF/2020). Confidentiality of all the information obtained from participants was maintained.
| Results | | |
Examinee's feedback
Questionnaires were sent to 95 examinees and the response rate was 66.3%. The response rate of pre/para clinical departments was 60%, while that from clinical departments was 65.8% [Table 1]. All the examinees had passed the exam. | Table 1: Department wise distribution of responses obtained from examinees
Click here to view |
It was a novel experience for 97% of examinees, as they had never participated in practical examinations by video conferencing earlier.
Interaction with examiners, ease of interactions during examination, ability to understand conversation and overall experience regarding the VCMPE was rated to be very good by 32% to 38% of the examinees. Another, 43% to 54% of the examinees rated these aspects as good. Only one (1.6%) student rated the examination as bad with respect to the ability to understand the conversation, whereas two (3.2%) reported a bad overall experience. In general, this mode of examination was perceived to be good to very good by most of the examinees [Figure 1]. | Figure 1: Feedback regarding interaction with examiner and overall experience during examination, by examinees (n = 63) (All values are in %)
Click here to view |
The VCMPE was highly reliant on the use of technology for assessment by external examiners, due to their inability to be physically present at the examination venue. The examinees were asked to rate problems related to audio, video and internet connectivity on a visual analogue scale of 5-1 ranging from 'always' to 'never'. A feedback regarding these issues showed that internet connectivity issue, audio and camera problems was never faced by about 22%, 17.5% and 36.5% of the examinees, respectively. Rest of the examinees faced some or the other issues during interaction with the external examiners. However, severe problems were faced by only three to four examinees [Figure 2]. | Figure 2: Connectivity and technology issues during online practical examination as reported by examinees (n = 63) (All values are in %)
Click here to view |
Most of the examinees felt that VCMPE could meet the objectives of practical examination, either completely (37%) or to quite an extent (44%). About 19% of the examinees stated that the objectives of practical examination could not be met adequately with VCMPE [Figure 3]. About 65% of the examinees supported this mode of assessment in future, and another 24% were of the opinion that it may be used; however, the rest 11% did not advocate the use of this mode of assessment. | Figure 3: Rating of examination pattern with respect to achievement of the objectives by examinees (n = 63)
Click here to view |
The examinees were asked to identify the strengths, weaknesses, opportunities and challenges in conduct of online practical examination. Most important strength reported by about a fourth of the examinees was that this is a useful method for assessment in times of pandemics such as COVID 19, which necessitates social distancing and travel bans. It allows us to follow the pandemic protocols to prevent spread of disease, while simultaneously achieving our objectives. Other strengths reported were that this mode of assessment is convenient, obviates the need of travel and hospitality and therefore is cost-effective (18.3%). It was also reported to cause less anxiety/stress (15.5%), lead to more confidence in interaction with examiners (9.9%), seemed more interactive (8.5%), was objective (5.6%), had support of internal examiners (2.8%) and could be recorded for future use (1.4%). Few examinees reported it to be as good as traditional examination (7%).
Commonly identified weaknesses of this mode of examination were that interference due to internet connectivity leading to audio and video problems were frequent (39.7%). Examinees were of the view that the assessment was less complete in this mode of examination and psychomotor skills were not tested adequately (27.9%). Few others reported that they did not have an actual feel of the examination (7.3%) and the examination was totally dependent on the internal examiners (7.3%). Less interaction (5.9%) and increased anxiety amongst examinees, who had no previous exposure (1.5%), were also identified as weaknesses of VCMPE. About 10% of the examinees did not report any weakness.
The opportunities identified by examinees were that this mode of examination is very cost effective and provides an opportunity to involve examiners from around the globe (40.7%), help prepare for examination at international or online platform (12.9%). They also stated that it is a good method for conducting examinations in pandemics, without simultaneously increasing the risks of transmission (12.9%). About one-fourth of examinees could not identify any opportunity. Challenges stated were similar to weakness, with internet connectivity being a major issue (61.9%). This was followed by feeling of incomplete interaction with examiner leading to decreased satisfaction (20.6%) and mental/eyestrain due to increased screen time (12.7%).
The most common recommendation provided by examinees for conducting similar type of examination in future were that there should be a provision of better internet connectivity, large screen with good quality camera and audio facilities (69%). Another 11% examinees recommended prior training of examinees before final examination to allay anxiety. Other recommendations included presence of continuous video surveillance or multiple cameras for proper assessment and prevention of unfair means during assessment (5%). No recommendation was provided by remaining 15% of examinees.
Examiner's feedback
Feedback questionnaire was filled by 23 out of 80 examiners, out of which 11 were internal examiners and 12 were external examiners. The response rate was 28.7%, after sending three reminder E-mails for submission of responses.
Majority of examiners were of opinion that about 60%–80% of objectives of practical examination were achieved. The range varied from 50% to 100% [Figure 4]. | Figure 4: Response to the question ‘What percentage of objectives of practical examination was achieved?' (n = 23)
Click here to view |
Only 39% of the examiners opined that they were able to examine all the domains of assessment in examination. Another 48% examiners said that they could partly assess the three domains [Figure 5]. | Figure 5: Response (%) to the question ‘Could you examine all domains of assessment in examination?' (n = 23)
Click here to view |
Examiners reported that they could generally assess the cognitive domain (91%) through the VCMPE; the assessment of affective and psychomotor domain through VCMPE was limited to 4% and 5%, respectively [Figure 6]. | Figure 6: Domains of learning assessed in online method of assessment based on responses from Examiners (n = 23)
Click here to view |
Majority (88%) of examiners agreed that examination was conducted in fair manner and it was convenient to document results during exam. 55%–60% felt that internet connectivity was appropriate and audio video quality was good during examination. Only a few examiners disagreed with the audio/video quality and maintenance of internet connectivity (4%–9%) during examination [Figure 7]. | Figure 7: Rating of examination process, connectivity and documentation in VCMPE by examiners (n = 23) (All values are in %)
Click here to view |
Based on the experience of online practical examination, 27% of examiners recommended it for future, while a majority (64%) were not very sure of it. Only 9% of the examiners opined against this mode of examination.
Examiners were asked if they could assess the various components of examination in a similar manner to that of TMPE conducted face to face. Forty percent of the examiners were of the opinion that OSCE/case/viva component of practical examination could be conducted in a similar manner as in TMPE. However, the percentages for cognitive component and affective component were only 22.7% and 18%, respectively. 18% examiners suggested that all domains could be evaluated with VCMPE equally well.
The SWOT analysis of online mode of practical examination was conducted by examiners. Convenience for external examiners and saving of time and money was major advantage of VCMPE according to 64% of examiners, while 24% said that the prevention of spread of pandemic is a key strength of VCMPE.12% examiners opined that less stress to examinees is the main advantage of VCMPE.
Complete assessment of all domains of learning (36%), especially, skill/psychomotor assessment (35%) could not be justified with VCMPE and were stated as common weaknesses by examiners. Dependence on internet or technology was considered by 22% as a major drawback of online mode of assessment, while 4% felt that introduction of new system of assessment to student is a constraint.
Dependence on internet and audio-video technology platforms was stated as a key challenge for VCMPE by 59% of examiners. Some examiners (9%) felt that ensuring fairness in conduct of examination is a challenge, while others (9%) felt requirement of more time and resources are main challenges in VCMPE. Dissatisfaction of personal interaction with examinees was also reported by 9% of examiners as a challenge. 13% of examiners did not report any challenges in VCMPE.
| Discussion | | |
Several studies have evaluated the online theory examination; however, studies evaluating online practical examination are scant. This study is therefore a novel attempt with an aim to assess the strengths, weaknesses, opportunities and challenges of online conduction of medical practical examination, based on the feedback of examinees and the examiners. The response rate for examinees in the current study was good (66.3%), but that of examiners was not very encouraging (28.7%), despite the three reminders sent to them. The reason for poor response from examiners could be their engagement in delivering responsibilities related to COVID-19 pandemic at their respective places of work. The residents from department of microbiology and pathology were also engaged in the institute. The lowest rate of responses was from obstetrics and gynecology, ophthalmology and Orthopedics.
The results were quite encouraging from the perspective of examinees, as most of them could have a good interaction with the examiners and reported an overall good experience with the VCMPE. The examinees were mostly satisfied with respect to achievement of objectives of the examination through this mode of examination; however, about 17% rated the assessment as being only partly able to achieve the objectives, which may again be a matter of concern. Despite the issues reported by examinees regarding connectivity and incomplete achievement of objectives, about one-third of examinees supported this mode of assessment for future examinations based on the strengths identified. One of important strengths was that it allayed anxiety, led to more confidence and interaction, while simultaneously drawing support from internal examiners, apart from the obvious strengths of being cost effective and preventive with respect to COVID 19 transmission. A study by Stowell and Bennett has also reported that examinees who normally experience high levels of test anxiety in classroom, experience reduced anxiety in online exams.[8] Hodder et al. have also reported in their study that immediate feedback given in online examination system exerts a calming influence on the stress of examination and appears to make the process more enjoyable, both for examinees and examiners.[9] In the present study, while few examinees reported that this mode of examination allays anxiety, others reported the contrary and expected to have a previous exposure to this mode of assessment before the final assessment. Few examiners also indicated the need to expose the students to this mode of examination. Özden, in their study have also reported that warm atmosphere and prior training is required to reduce the anxiety amongst students during examination.[10] Shen et al. have also reported that students taking online exams, more than once, report more familiarity with the examination process and enjoy the examination process more.[11] The opportunities identified by examinees provide us a prospect of utilising this mode of examination, to help them prepare for international examinations and interviews in future.
The feedback by examiners was not very encouraging with respect to their response rate and the achievement of objectives of practical examination (interquartile range of 60%–80%). They felt that VCMPE had limitations in assessment of affective and psychomotor domains; however, the assessment of OSCE/Case presentation and viva component was achieved satisfactorily. The cognitive domain is assessed, mainly by theory papers, and the inability to assess the affective and psychomotor domains due to the lack of real doctor-patient interactions in VCMPE is a major limitation. These domains could be incorporated in formative assessment for a comprehensive assessment throughout the period of study. In the wake of the current COVID 19 pandemic regulations, doctor–patient interaction for the purpose of assessment was not possible in most of the departments and was an obvious unavoidable limitation. These issues have also been highlighted by other researchers, who have suggested alternate methods of e-assessment in times of COVID-19.[12]
Both, examiners and the examinees have identified that cost-effectiveness and feasibility are important strengths of this mode of examination. Fairness and reliability of assessment were also identified as important attributes by examiners, although some examiners felt that fairness could be compromised in VCMPE. These findings are in concurrence to that reported by other researchers.[13]
Connectivity issues make VCMPE unreliable, as has been reported by Walsh who have stated, 'unreliable systems undermine confidence in the system of assessment amongst both learners and tutors'.[14] Lack of reliability on VCMPE may be due to the hardware, software, or even the power supply. Both the examinees and examiners highlighted these issues as a major limitation. The most common issue reported by the examinees and examiners was related to internet connectivity and its related issues such as video and audio issues, despite the presence of good hardware system, internet connectivity with a backup and a good support of IT team at the current institute for smooth conduction of examination. Issues related to internet connectivity were usually at the end of external examiners, who had connected from all over India. Only 1–2 examinees reported having a bad experience due to this online mode of examination, which may be a matter of concern considering the fact that this assessment is very crucial for a student as it contributes significantly (25%) to their overall marks. This could be overcome by active involvement of internal examiners in explaining and mediating the assessment by external examiners. A more in-depth exploration of issues would help us overcome these issues if we need to use it on a routine basis in future examination. For a country like India, where we do not have good internet connectivity at both ends, this mode of examination may act as a limitation for the student. The connectivity at both ends must be ensured prior to the day of examination by having a test run. However, Inuwa et al. have reported that computer-based practical examinations did not result in significant difference in performance of examinees as compared to the traditional examination, but we could not make such comparison in present study.[15] The examiners generally reported that VCMPE had limitations in assessing the examinees as comprehensively as the TMPE.
| Conclusion | | |
Video conferencing mode of practical examination is an innovative approach toward summative assessment of practical knowledge and skills of PG examinees in the era of COVID-19 with a primary aim to prevent transmission of COVID 19. The greatest advantage of this mode of assessment was its ability to accomplish the task of assessment while minimising the spread of infection, apart from being cost effective, time saving and feasible. However, as nothing in this world is perfect, it had certain weaknesses and challenges. However, considering the imminent need of conducting the examination in times of COVID 19, VCMPE served as a boon for examinees and the examiners. The limitations and challenges highlighted by the examiners and examinees would guide us to formulate strategies to overcome them, if VCMPE needs to be implemented in future also, as COVID-19 is expected to stay for some time. The findings of this study would also guide other institutes to use VCMPE in their assessments. Although the examinees find it more convenient, from the perspective of examiners, concerns related to fairness of conduction of examination, incomplete assessment of all domains and incomplete achievement of objectives are significant limitations in this type of practical assessment. Authors conclude that VCMPE may be a workable alternative during critical situations such as pandemic, genuine reason of candidate's absence to appear in practical examination or in unprecedented physical absence of one of the external examiners.
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
| References | | |
| 1. | Gujar N, Jadhav K. Medical education in India-An overview. J Med Sci 2016;9:76-8.  |
| 2. | Tabish SA. Assessment methods in medical education. Int J Health Sci (Qassim) 2008;2:3-7. |
| 3. | |
| 4. | |
| 5. | Liu BD. Being an American 2 nd year medical student in the COVID-19 pandemic. Int J Med Students 2020;8:77-8. |
| 6. | Sani I, Hamza Y, Chedid Y, Amalendran J, Hamza N. Understanding the consequence of COVID-19 on undergraduate medical education: Medical students' perspective. Ann Med Surg (Lond) 2020;58:117-9. |
| 7. | Lodha R, Kabra SK. COVID-19 Pandemic: The Way Forward. Indian J Pediatr 2020;87:676-9. |
| 8. | Stowell JR, Bennett D. Effects of online testing on student exam performance and test anxiety. J Educ Comput Res 2010;42:161-71. |
| 9. | Hodder RV, Rivington RN, Calcutt LE, Hart IR. The effectiveness of immediate feedback during the objective structured clinical examination. Med Educ 1989;23:184-8. |
| 10. | Özden MY. Students' perceptions of online assessment: A case study. Int J E-Learn Dis Educ 2005;19:77-92. |
| 11. | Shen J, Hiltz SR, Bieber M. Collaborative online examinations: Impacts on interaction, learning, and student satisfaction. IEEE Trans Syst Man Cybernetics-Part A 2006;36:1045-53. |
| 12. | Khan RA, Jawaid M. Technology enhanced assessment (TEA) in COVID 19 pandemic. Pak J Med Sci 2020;36:S108-10. |
| 13. | Shraim K. Online examination practices in higher education institutions: Learners' perspectives. Turkish Online J Dis Educ 2019;20:185-96. |
| 14. | Walsh K. Point of view: Online assessment in medical education–current trends and future directions. Malawi Med J 2015;27:71-2. |
| 15. | Inuwa IM, Taranikanti V, Al-Rawahy M, Habbal O. Anatomy practical examinations: How does student performance on computerized evaluation compare with the traditional format? Anat Sci Educ 2012;5:27-32. |
[Figure 1], [Figure 2], [Figure 3], [Figure 4], [Figure 5], [Figure 6], [Figure 7]
[Table 1]
|
Search Pubmed for