CC BY
3
nw-] JOURNAL OF CLINICAL MEDICINE OF KAZAKHSTAN Original Article
(E-ISSN 2313-1519)
Digitalization of the clinical exam in Covid-19 pandemic: Karaganda Medical University's experience
Gulshat Kemelova1, Ilya Shekhter2, Dinara Aimbetova1, Victor Riklefs1, Yekaterina Yuchnevich1
'Center for Simulation and Educational technologies, Karaganda Medical University, Karaganda, Kazakhstan 2Simulation Operations, Jackson Memorial Hospital Center for Patient Safety, University of Miami, Miami, USA 3Education, Karaganda Medical University, Karaganda, Kazakhstan
4Clinical Pharmacology and Evidence-Based Medicine Department, Karaganda Medical University, Karaganda, Kazakhstan
Abstract
Nowadays, Medical education in the Covid-19 pandemic poses a new challenge to explore approaches for delivering quality distance education, especially in the clinical competence assessments. Assessments of clinical competence approaches to conducting an exam, implementation of a new innovative strategy to the assessment procedure, minimize costs, and efficiency in using existing digital resources, have been revised.
The study aimed to compare paper and electronic checklists used in Objective Structured Clinical Exams and assessors' self-perception of electronic checklists and technologies.
Material and methods: The authors compared three types of checklists (paper, bubble, and electronic checklists) according to the following criteria: ease of use, registration, and authentication of examinees and examiners, processing of exam results, and cost benefits.
Results: In the course of analysis, three versions of checklists were compared, thus identifying the advantages and disadvantages of each one, including the assessors' attitude to new technologies.
Despite the high cost of the tablets compared to paper checklists, the use of tablets at OSCE had many advantages, and assessors mentioned that number of errors significantly dropped.
Conclusion: Thus, digitalization of the assessment procedure helped streamline the exam process, securing evaluation data, and speeding up information processing, which significantly minimized the cost of human resources.
Key words: medical education, checklist, OSCE, assessment, clinical competence, performance
Received: 2022-01-13. Accepted: 2022-03-29
© 0
This work is licensed under a Creative Commons Attribution 4.0 International License
J Clin Med Kaz 2022; 19(2):53-56
Corresponding author: Gulshat Kemelova. E-mail: [email protected]; ORCID: 0000-0002-1326-2363
Introduction
The Covid-19 epidemiological situation has led to a global change in the approaches to teaching medicine both around the world and in Kazakhstan [1]. It is time to be creative to explore approaches for developing and delivering quality distance education. Medical education in Kazakhstan is faced with the challenge of implementing distance learning, because clinical skills cannot be fully taught and assessed online. The pandemic necessitated a transition from traditional educational programs, to online education. All educators had to switch over to distance learning overnight while having different levels of experience in using educational technologies and devices. Assessment of clinical competence is a very important part of the evaluation of educational program outcomes and does not depend on any critical situation.
Nowadays, one of the most important assessment tools is the Objective Structured Clinical Exam (OSCE) which is a comprehensive examination for testing students' clinical skills and competences [2-4]. During the ten-year period of using OSCE at Karaganda Medical University, various methods have been used to assess students' achievements ranging from simple checklists to "advanced" [5]. With the widespread use of available digital resources, the term "digitalization" emerged as gradual transformation of business processes using digital resources. During the pandemic, using digital resources in student assessment accelerated faster but examiners have had some difficulties in using digital technologies. Electronic resources can offer a reasonable solution for assessing students [6,7]. The experience of using paper and electronic assessment checklists and examiners' self-
perception had not been studied in our university, and therefore, this was the goal of our research.
The purpose of this study is to describe and compare student assessment approaches using paper and digital resources during an Objective Structured Clinical Exam and study examiners' perception of new electronic checklists and technologies
Material and methods
This is a descriptive and comparative study. Three types of OSCE checklists were used: a paper checklist (pCHKL, n=7 850), a bubble checklist (bCHKL, n=12 530), and an electronic checklist (eCHKL, n=687). The OSCE is the second part of the clinical skills final examination carried out among senior students. In this study, the investigators attempted to evaluate the effectiveness of tools in assessing clinical competence of students, to identify the advantages and disadvantages of paper and digital resources; and to assess the impact of integrating digital technologies on student assessment (87%) and perception of the examiners (n=70) in using them. To accomplish this goal, the costs for material resources and human resources were assessed, and an online survey of all participants in the assessment was conducted. To evaluate the effectiveness of the digital technologies, methods of relative values and efficiency at the microeconomic level were used.
Results
The three types of OSCE checklists, paper checklist (pCHKL), bubble checklist (bCHKL), and e-checklist (eCHKL) were used, analyzed, and compared for their effectiveness. A single checklist was used to assess the OSCE but the way in which the results were recorded and processed was different.
Paper checklists (pCHKL) is a simple paper version and included criteria of clinical skills algorithms and were fairly simple and easy to fill out by examiners. The OSCE results were transferred to a computer database manually, which required high concentration from the staff to minimize errors that may be related to human factors (inattention, heavy workload), which resulted in processing delays of up to 14 days. With a large number of students, the amount of time spent and the quantity of random errors increased, which delayed the processing and delivery of exam results. The disadvantages of pCHKL were the additional resources related to printing of up to 10,000 copies, staff time to review all materials to identify scoring errors, populate the Excel table for analyzing, and to process and distribute the results. The paper form was easy for examiners to fill, but it was more difficult for staff support personnel to transfer data from paper to Excel.
Bubble checklists (bCHKL) were implemented to further improve the assessment process and paper scanned versions of the checklists were used for automatic data analysis by FormReturn, which, as expected, could allow optimizing the processing of OSCE results. FormReturn allows users to develop their own form of an evaluation sheet template, and then use barcodes to track information across individual checklist categories. The development of a template of the checklist was personalized with the identification number of the examinee, which cannot be changed during OSCE. Preparing all the bCHKL required a long time and preliminary approval of all stages, points, and features of the FormReturn program. All bCHKL filled by examiners were scanned, and the results immediately were loaded into the server database. However, scanned bCHKL have disadvantages. The bCHKL had to be sorted before and after the OSCE day by student ID, cohort, and OSCE stations. It also
took up extra time and staff resources. For example, 10 bCHKL were printed per student, and 1,253 students required at least 12,530 sheets. During one OSCE period, with 3,163 examinees at 10 stations, 64 reams of paper were used, which averages US $220, excluding printer ink and other print shop services. Additional costs during OSCE included office supplies, which also required financial costs associated with the acquisition of a high-resolution printer during scanning. Also, using pCHKL and bCHKL, required up to 10 staff members who were employed for uninterrupted exams and timely processing of OSCE results.
Electronic checklists (eCHKL) are an alternative form for faster and objective student assessment. With the advent of widespread use of available digital resources, the term "digitalization" emerged as the need to transform processes using digital resources [Hochlernet, 2015]. In connection with the development of digitalization strategy, improvement and automation of assessment processes, interest in digital resources arose in the center of practical skills, and an electronic evaluation form was introduced on tablets to streamline the OSCE evaluation process. For this, a tablet with the Android operating system was used. The interface was designed to fill out the fields of the evaluation form conveniently and quickly. For this, the tablets with the Android operating system and eCHKL developed in Google Form (Figure 1), and 687 electronic student results were analyzed.
Figure 1 - e-checklists for Objective Structured Clinical Exam. Electronic checklists for clinical competence assessment on Objective Structured Clinical Examination used
For comparison of three types of checklists, the utilization of resources in OSCE was analyzed. A total of 20 tablets were used in the examination, costing about US $1,700, thus, the cost of one examination period of the OSCE is US $340. The benefits of using tablets and electronic estimating are greater than evaluating using pCHKL and bCHKL. For example, the relative cost-effectiveness of using the tablets showed the profitability of the resource used and the relative cost savings, including labor costs for staff time. For the evaluation of students in OSCE, more than 75 available electronic programs were studied. According to the analysis of the capabilities and effectiveness of the program, Google Forms and Microsoft Forms are superior to other programs in functionality. Tablets can be used for 10 years, and the Google Forms cloud environment does not require financial expenditures, OSCE results are available at any time, and are stored for a long time. The advantages of using Google forms are the simplicity and convenience of creating new forms, the availability of information at any time of the day and on
Table 1
Comparative table of the evaluation sheet in pCHKL, bCHKL, and eCHKL in OSCE with the calculation of savings due to the increase in staff productivity. http://www.antegra.ru/news/experts/_det-experts/4/
Criteria pCHKL bCHKL eCHKL Savings, multiplicity Productivity
Pi ( %)
Filling time per one checklist,min 10 10 5 2,0 100
Identification of ID students,
min 30 60 1 0,03 2900
Time for data processing,
Day/hour/min 7/42/2520 5/30/1800 0/0/60 0,02 3500
The cost of resources per OSCE, $USD 220 4220 340 1,5 120
Number of personnel (n=HR) 10 10 2 0,2 400
Number of OSCE days, days 14 14 7 0,3 500
Staff salary per OSCE period, $USD 1800 1800 370 4,8 400
any Internet-connected device. However, this online service has some drawbacks such as the lack of template variety, which limits the use of more complex evaluation forms. Also, some of the senior examiners were not familiar with tables and had difficulty using them.
Below, the comparative table of the evaluation sheet in paper and electronic versions of OSCE shows the calculation of savings due to the increase in staff productivity in one examination period as an example (Table 1).
The calculation of indirect savings of economy at the expense of increasing the user's labor performance was made using the formula: )- ) 5 ,M , and Pi (b %) - labor productivity, ATi - i-view savings with automation in hours, and Fj - the time that were planned for the j-type work before the introduction of automation in hours. The table shows that with the OSCE automation, labor productivity in hours and HR increased significantly. Despite the high cost of the tablets compared to the pCHKL/bCHKL, the use of the tablet and the N3 have more advantages.
At the end of OSCE, feedback is regularly collected through questionnaires among examiners and students. Seventy out of 114 examiners participated in the online survey, which amounted to 61.4%. When using pCHKL / bCHKL, there was a low satisfaction with filling out paper evaluation sheets (52 %) and there were many comments and recommendations to simplify the assessment procedure. When using an e-checklist, an online survey among examiners was used, obtained by simply clicking on the tablet screen. Thus, online survey shows that 92% of the respondents were satisfied with the innovation, and in the comments section indicated their interest in further use of e-checklists. The survey among 87% students, who took part in the OSCE, was also conducted. The feedback was accepted positively by students, as students are the cohort that is closer to digital technologies and the use of modern smart technologies in the educational process and the examination only give them support and approval. In the introduction of electronic checklists and the automation of the exam process, while piloting the OSCE, fear of technology arose among examiners over 70 years old. To reduce technophobia, examiners were trained, and they were questioned before and after the exam the self-perception exam during the assessment. The results showed that before the training, more than 80% of senior teachers had technophobia, after the training, the phobia decreased up to 5%. Therefore, automation of assessment processes requires planning the digitalization process to reduce stress for examiners and teachers.
Discussion
The COVID-19 emergency triggered a rapid transition from simple forms of assessment to digital electronic assessment formats. With the aim of comparing the effectiveness of the
paper and electronic versions of the evaluation sheets used during the Objective Structured Clinical Exam, a descriptive and comparative study was conducted. To assess the effectiveness of the evaluation procedure in OSCE, the evaluation sheets were divided into three types and compared by the convenience of filling out checklists, registering and authenticating the examinees and examiners, processing the exam results, and studying the indirect economic benefits of using various types of evaluation sheet. The three types of evaluation sheets were used: (i) a simple paper evaluation form, (ii) a bubble version of the evaluation form, and (iii) an electronic evaluation form. The use of digital resources made it possible to optimize the assessment process, secure data and OSCE results, and ensure the efficiency of processing all received information. The use of electronic resources greatly facilitates the collection and processing of test data by automating the process. E-checklists record personal data by IP-addresses, which eliminates the entry into the system of unauthorized persons and allows you to trace the geography of users. To reduce technophobia and stress of examiners, automation of assessment processes should be planned the digitalization process. Feedback from the students, examiners, and administrative staff shows effectiveness of the eCHKL as compared to pCHKL/bCHKL, in generating exam results and minimizing scoring and data processing errors.
Conclusion
In conclusion, the results of the study allowed us to draw the following conclusion: the state of emergency caused by the Covid-19 pandemic accelerated digital implementation of assessment and using digital resources allows faster processing of exam data, to trace the personalization and track each examiner, allows controlling the points of contact of all forms, and increasing the target audience by expanding the geography of IP addresses, and are the main driver for improvement and modernization of OSCE. The effective planning of the digitalization process in education will reduce technophobia and stress of examiners, automation of assessment processes. The charges of resources for digitalization in medical education show appropriate cost-efficacy that necessity for making the decision of the budget of the university.
Disclosures: There is no conflict of interest for all authors.
Acknowledgements: The authors are grateful to all medical teachers and staff of the Centre for Simulation and Educational Technologies who helped collect resources and gave great ideas for writing the article, while no funding was received for this study project.
Funding: None.
References
1. Sivakumar Arunachalam, Jitendra Sharan. COVID-19 Reflections and Team-Based Entrustable Professional Activities for Bioevent Preparedness. https://doi.org/10.15694/mep.2020.000265.1
2. Chacko TV. Simulation-based medical education: Using best practices and curriculum mapping to maximize educational benefits in the context of shift toward competency-based medical education. Archives of Medicine and Health Sciences 2017; 5:9-15. https://doi. org/10.4103/2321-4848.208217
3. Datta R, Upadhyay KK. Simulation and its role in medical education. Medical Journal Armed Forces India. 2012; 68:167-172. https:// doi.org/10.1016/S0377-1237(12)60040-9
4. Dosmagambetova R, Riklefs I, Riklefs V. Particular qualities of medical education in Kazakhstan. Medical Education and Professional Development. 2014; 4:257-289. https://doi.org/10.1080/0142159X.2018.1441989
5. Van Nuland M, Van den Noortgate W, van der Vleuten C et al. Optimizing the utility of communication OSCEs: omit station-specific checklists and provide students with narrative feedback. Patient Educ Couns. 2012; 88:106-12. https://doi.org/10.1016/j.pec.2011.12.015
6. Hochlernet A, Schultz J-H, Moltner A. Electronic acquisition of OSCE performance using tablets. GMS Journal for Medical education. 2015; 32:1-17. https://doi.org/10.3205/zma000983
7. Kemelova G.S., Aimbetova D.B. Rebooting of the simulation training during a pandemic. Virtual Technologies in Medicine. 2020;1(3):98-99. [In Russian] https://doi.org/10.46594/2687-0037_2020_3_1226
8. Papapanou M, Routsi E, Tsamakis K, et alMedical education challenges and innovations during COVID-19 pandemic. Postgraduate Medical Journal. Published Online First: 29 March 2021. http://doi.org/10.1136/postgradmedj-2021-140032
