CC BY
16
CLINICAL SKILLS' ASSESSMENT BASED ON MODIFIED INTEGRATED OBJECTIVE
STRUCTURED CLINICAL EXAMINATION
Irma Manjavidze1, Zurab Vadachkoria2, Dali Chitaishvili3, Khatuna Lomauri4,
Ketevani Kankava5 ©
1MD, PhD; Full Professor, Head of Department of Clinical Skills and Multidisciplinary Simulation,
Tbilisi State Medical University, Tbilisi, Georgia;
MD, PhD; Full Professor, Head of Department of Children and Adolescent Stomatology and Prophylactics of Stomatological (Dental) Diseases; Head of Direction of Children and Adolescent Surgical Stomatology Tbilisi State medical University, Tbilisi, Georgia;
MD, PhD; Assistant Professor, Department of Clinical Skills Multidisciplinary Simulation, Tbilisi
State Medical University, Tbilisi, Georgia; 4MD, PhD; Associate Professor, Department of Pediatrics, Tbilisi State Medical University, Tbilisi,
Georgia;
5MD, MBA Tbilisi State Medical University, Tbilisi, Georgia.
Summary
OSCE is a type of exam, which is highly appreciated due to its reliability and objectiveness. We conducted a modified integrated OSCE with topics from multiple subjects to discuss an exam of this type as a possible part of assessment students' skills at the end of the 5th year at medical university. Students completed tasks at 10 active stations and were assessed by observers with pre-developed checklists. Then examiners and examinees completed questionnaires, which were aimed for evaluation of the organizational part of the exam as well as its functional value. Overall fulfillment of the tasks was satisfactory. The performance picture did not correspond to students' evaluation on the difficulty of the tasks. The weighted average for the student competence according to the observers' opinion was quite high. The stress level was evaluated as moderate or low by majority of students and observers. The study revealed that both students and observers agree, that OSCE itself is an effective, non-stressful tool for skill-assessment and the organizational part of the exam was good enough to consider, that the Department of Clinical Skills is ready to conduct modified integrated OSCEs as a routine exam for students before entering clinical practice.
Keywords: Integrated OSCE, satisfaction, skill-assessment, checklist, medical education
Ключевые слова: интегрированый ОСКЭ, удовлетворенность, оценка навыков, контрольный перечень, медицинское образование
Introduction
Development of clinical skills is one of the principle objectives in medical education. Teaching clinical skills is an important and necessary component in medical schools. Another challenging point is how to assess the skills in accordance with the requirements of the curriculum and expectations of the students and academic staff. At the moment there is enough experience supporting the Objectively Structured Clinical Examinations (OSCEs) as the best technique to test student's competence.
OSCE is a type of exam, which is highly appreciated and trusted due to its reliability and objectiveness. It was introduced in 1970s and is becoming more and more popular thereafter. It consists of several stations, where students are asked to follow certain scenarios and short tasks are assessed by objective marking scheme. The exam is maximally free from examiners' influence. It is based on bare observation and assessment of the examinees' skills according to the strict and uniform criteria.
At Tbilisi State Medical University (TSMU) OSCEs were first introduced several years ago
© Irma Manjavidze, Zurab Vadachkoria, Dali Chitaishvili, Khatuna Lomauri, Ketevani Kankava, 2017 г.
at the Department of Clinical Skills. Teaching Clinical Skills is performed simultaneously at several levels of medical education in TSMU. It happens according to the theoretical knowledge that is already obtained by the student at the time of reaching a certain level. During past years OSCEs have been used to test the skills of 2nd, 4th, 5th and 6th year students. As expected, after completing basic medical education (3 years) students are dealing with OSCEs much better.
Effectiveness of OSCEs in medical educations is well known and well understood (Duerson et al. (2000); Townsend et al. (2001); Tsai et al. (2008)). Researches have provided tentative evidence that a modified OSCE may be an appropriate method for selecting candidates for postgraduate training schemes (van Woerden et al. (2003)). But the process of organizing the examinations to be fair, representative of students' knowledge and skills and at the same time less stressful for participants is still quite a challenge. Several studies focused on evaluation of OSCE success relying on students' and examiners' feedbacks (Branch (2014); Khorashad et al. (2014); Marzieh et al. (2012); Pierre et al. (2004)
Students' feedbacks were analyzed. According to the existing data overall attitude towards OSCE is clearly positive from students' as well as from faculty members' side. Small surveys and personal discussions have shown, that students prefer to have their skills tested within the frames of objectively structured exam with clear, non-ambiguous criteria. The faculty members also accept OSCEs as the best method of testing clinical skills.
But there are some concerning issues as anxiety and seeing preparation to the exam as coping. Despite the fact that there still remain some minor problems the reliability of OSCE-type exams is increasing and the number of disciplines that use this kind of exams become broader. So the value of understanding the advantages and limitations is also increasing. As the students and examiners are the main participants in the exam, their opinion is very valuable and something that should really be taken into account.
The aim of our current study was to conduct a survey about satisfaction with integrated OSCE and its effectiveness at the same time comparing the performance of the students with their satisfaction level, to find out, whether the feedback is real or over/underestimated because of the scores that students get.
In addition to this, we planned to investigate whether modified integrated OSCE could be an effective tool to assess students' clinical competence at the end of 5th year of medical education and their readiness for final part of clinical rotations with a plenty of patient interactions.
Materials and Methods
We planned to carry out a model examination - integrated OSCE with 12 stations; then study the attitude of examinees and observers towards various characteristics of practical side and effectiveness of the examination, as well as their personal impressions. For this purpose the following setup was chosen: OSCE with 10 active and 2 rest stations, which were strategically placed and equipped with refreshments. The stations were designed to test various skills and they were named as follows: Resuscitation/Trauma management; Examination of newborn; Gynecological examination; Emergency care for pediatric patient; Abdominal ultrasound; Pulmonary auscultation; Recording and interpreting ECG; Management of cardiac arrhythmias; Breast examination; Obstetric examination. The specific competencies were estimated for each subject. To minimize the reflection of examiners' personal attitude on the scores that students received, criterion-based scoring was established. For this purpose specific check-lists were designed. Each step of each manipulation was given certain amount of points. This helped to facilitate and standardize the observers' work. The contents and structure of checklists, as well as the clinical scenarios used at the examination were discussed within working groups consisting of the specialists in each discipline and faculty members from the Department of Clinical Skills. Points for each step for completing the manipulation were defined by strict criteria stated on the checklists and they ranged from 0 (omitted or inadequate) to 2 (correct).
We built self-administrative questionnaires for examiners and examinees. They were designed according to the specific topics of interest and proofread by a psychologist to have the
exact and functional verbal content. The questionnaires were aimed for evaluation of the organizational part of the exam as well as its functional value. All questionnaires had space for comments and remarks. The questionnaires of students and examiners did not contain any content, which could allow the identification of their personality. The design of the questionnaires allowed us to calculate, how much the expectations of students did and examiners meet the actual findings regarding the organizational part of the exam.
In July 2015 we conducted a modified integrated OSCE. 24 students took part in the pilot exam. All of them were 5th year students, who had already completed respective clinical rotations and were trained with special course at the Department of Clinical Skills. Several days earlier the students participated in introductory session, which included instructions for the examination and review of the assessed competences. Observers were present at each station to evaluate the performance of the students. Students had 6 minutes to accomplish task on each station and 1 minute was given for changing the station and reading the short assignment (scenario), which was attached to the door at the entrance of each station. The same written assignment was present also in the room, so that students could read it again inside the station. Students were evaluated by the check-lists and the scores collected by them at each station were recalculated into a 4-point scale for homogeneity of the results. At the end of the exam all students and observers were asked to complete questionnaires. Students answered the questions about the complexity of the tasks, level of stressfulness, how well the stations were designed and equipped. In addition students assessed the usefulness of OSCE in comparison to other forms of examinations. The questionnaire of observers was oriented on same topics and in addition it aimed to evaluate their attitude towards the representativeness of the exam and to which extent did the checklist-based result correspond to their personal impression. All results were analyzed in comparison with marks that students got. Participants were informed about anonymity and they were free to express their attitude and impression. After the completion of the questionnaires the representatives of the teaching departments and the head of the Department of Clinical Skills carried out a debriefing session were all participants could discuss any specific details.
Collected data were analyzed with basic statistics calculating means and standard deviations.
Results
All 24 students passed the exam. The fulfillment of all tasks was satisfactory except of two students who failed on the stations "Emergency care for pediatric patient" and "Breast examination". The highest average score (3.84) was seen at the station "Examination of newborn"; the lowest average score was 2.53 for "Breast examination" station (Table 1).
At the same time this last one was evaluated as second easiest station by the students (weighted average for difficulty by 5-point scale - 2.42). The students found "Abdominal Ultrasound" the easiest station (weighted average 2.08) and "Managements of cardiac arrhythmias" - the most difficult one (weighted average 3.58) (Table 2). Though the performance of examinees did not completely correspond to this distribution.
All students gave positive feedbacks to OSCE, stating that this exam is more objective, superior than other types of skill-assessment. They stated that exam was well organized and all stations were well equipped. Most students agreed that their skills were satisfactory for the given tasks (weighted average 4.0).
Students and observers were asked whether they would feel more confident and comfortable with two observers instead of one at each station. 75% of students do not think, that they need the second observer, while 70% of observers state, that they could accept attendance of another observer as a possibility.
All students agree or strongly agree that all tasks were in correspondence with their knowledge (Chart 1) and they had enough time for fulfilling them (Chart 2). The weighted average for the student competence according to the observers' opinion was 4, the same variable for time sufficiency was 4.7. The stress level was evaluated as moderate (75% of students and 43% of observers) or low (25% of students and 57% of observers).
The study revealed that both students and observers agree, that OSCE itself is a wonderful, non-stressful tool for skill-assessment and in this exact settings the exam was very well organized. The evaluation checklists were built in a homogenous, objective manner. They assessed the skills and understanding of theoretical topics. 6 minutes were regarded as enough for completing the task and assessment. The small non-concordance of students' self-assessment and their points happened. The main reason for this was inability of the examinees to follow the steps in task fulfillment. It can be managed by indicating specific study material and if necessary, additional consultations. Slightly complicating the tasks could be useful and acceptable in these certain conditions, as the skills that were tested at given pilot examination were quite basic and easily accomplishable by 5th year students. Overall scores were higher than average, percentage of positive feedbacks and overall satisfaction of participants was much higher than expected probably due to the pilot settings and special training of the students at the Department of Clinical Skills that reduced their anxiety and increased self-confidence.
One of the examiners stated in the comment that scoring of the skill-fulfillment would preferably be more detailed as in some instances it was quite rough. Several students said they would prefer to know their score for each station before moving to the next one. They received the explanations, why it's not possible according to the general principles of OSCE. Only one student said, that at several stations additional time should be given for putting on the gloves. This comment was discussed, but as this was the opinion of only one students and as all of the examinees fulfilled tasks in given time, 6 minutes still remain a standard for the tasks in standard scenarios at our department.
Discussion
The study revealed that both students and observers agree, that OSCE itself is a useful, non-stressful tool for skill-assessment and in this exact settings the exam was very well organized. The evaluation checklists were built in a homogenous, objective manner. They assessed the skills and understanding of theoretical topics. 6 minutes were regarded as enough for completing the task and assessment. The small non-concordance of students' self-assessment and their points happened because they did not follow the steps in task fulfillment. It can be managed by indicating specific study material and if necessary, additional consultations. Slightly complicating the tasks could be useful and acceptable in these certain conditions.
All tasks and corresponding assessment checklists were analyzed together with the representatives of corresponding departments and specialists from the department of clinical skills. Some improvements in scoring system were found to be possible and are already planned for future practice. All the findings are aimed to facilitate future efforts and find an appropriate constellation of tasks, criteria and environment to optimize the effectiveness of the exam.
The key learning points we received from this study are the following:
• Good organizational level of exam ensures satisfaction of both - student and observers independently from the success rate;
• The pilot exam showed, that organizational details were well planned and implemented -this allows modified integrated OSCEs to be used routinely for testing students' competency before clinical practice;
• Students impressions about task complexity don't always match their performance;
• Very high success rates of students allows us to think about more complicated clinical cases to be integrated into exam;
• Modified integrated OSCE is an effective tool for testing students' competency in fulfillment of practical tasks in multiple disciplines;
• Saving money and time by integrating several disciplines into one exam reduces neither the effectiveness, nor satisfaction levels of the exam;
• Modified integrated OSCE can be implemented as a barrier exam before allowing the students into practice in clinical settings.
Table 1
Average scores collected by the students at each station:
Station Average score (4-point scale)
Managements of cardiac arrhythmias 3.27
Gynecological examination 3.41
Emergency care for pediatric patient 2.88
Examination of newborn 3.84
Abdominal ultrasound 3.46
Obstetric examination 3.62
Resuscitation/Trauma management 3.37
Recording and interpreting ECG 3.63
Pulmonary auscultation 3.21
Breast examination 2.53
Table 2
Difficulty of the tasks at each station according to the information collected from student:
Station Weighted average score (4-point scale)
Resuscitation/Trauma management 3.25
Examination of newborn 2.75
Gynecological examination 2.50
Emergency care for pediatric patient 2.92
Abdominal ultrasound 2.08
Pulmonary auscultation 2.58
Recording and interpreting ECG 3.08
Management of cardiac arrhythmias 3.58
Breast examination 2.42
Obstetric examination 2.50
Task was in correspondence with studied material. (1- don'tagree; 5- absolutely agree.)
4.ÖT 4,6- 4,(57
Resuscitation Newborn GYN Exam Emergency Ultrasound Auscultation ECG Airliytiunin s Breast Exam OB exam Exam
Chart 1: Students' opinion on correspondence of their study materials and given
tasks
Given time was enough for completing the tasks, (1- don't agree; 5- absolutely agree.)
Chart 2: Students' answers on question: "Was the given time enough for completing the
tasks?"
References
1. Branch C. (2014). An assessment of students' performance and satisfaction with an OSCE early in an undergraduate pharmacy curriculum. Currents in Pharmacy Teaching and Learning. 6(1):22-31.
2. Duerson MC., Romrell LJ., Stevens CB. (2000). Impacting faculty teaching and student performance: nine years' experience with the Objective Structured Clinical Examination. Teach Learn Med.12(4):176-82.
3. Khorashad AK., SomSalari S. et al. (2014). The Assessment of Undergraduate Medical Students' Satisfaction Levels With the Objective Structured Clinical Examination. Iranian Red Crescent Medical Journal. 16(8):e13088.
4. Marzieh A., Mansoureh N. (2012). The Objective Structured Clinical Examination: A study on satisfaction of students, faculty members, and tutors. Life Science Journal.9(4):4909-4911
5. Pierre RB., Wierenga A., et al. (2004). Student evaluation of an OSCE in paediatrics at the University of the West Indies, Jamaica. BMC Medical Education.4:22.
6. Townsend AH., McLlvenny S. et al. (2001)The use of an objective structured clinical examination (OSCE) for formative and summative assessment in a general practice clinical attachment and its relationship to final medical school examination performance. Med Educ.35(9):841-6.
7. Tsai JC., Liu KM. et al. (2008). Evaluation of the effectiveness of postgraduate general medicine training by objective structured clinical examination---pilot study and reflection on the experiences of Kaohsiung Medical University Hospital. Kaohsiung J Med Sci.24(12):627-33.
8. Van Woerden H., Agbo F., Amso NN., Stokes I. (2003) A Modified OSCE Assessing the Assimilation and Application of Ethical Principles Relevant to Obstetric and Gynaecological Practice. Med Educ Online [serial online]. 8:8.
