CC BY-ND
50
National Research University Higher School of Economics Stognieva, O., & Popov, V. (2021). Assessment under Covid-19: Exploring
Journal of Language & Education Volume 7, Issue 4, 2021 Undergraduate Students' Attitudes towards Their Online Thesis Proposal
Presentations vs. Face-to-face. Journal of Language and Education, 7(4), 139155. https://doi.org/10.17323/jle.2021.11657
Assessment under Covid-19: Exploring Undergraduate Students' Attitudes towards Their Online Thesis Proposal Presentations
vs. Face-to-face
Olga Stognieva, Victor Popov
HSE University
Correspondence concerning this article should be addressed to Olga Stognieva, HSE University, ul.
Shabolovka, 26/5 Moscow, 119049, Russian Federation. E-mail: [email protected]
This study conducted in an EFL setting investigates students' perception of the thesis proposal presentation in an online format due to safety concerns during the Covid-19 pandemic. Fifty-five students aged 20-22 years old, level B2-C1 in English, in their fourth year of a Business Informatics Bachelor's programme at HSE University, Moscow, were surveyed regarding their end-of-course assessment experience, which involved a Skype online presentation, rather than the usual face-to-face presentation. Data were collected using a researcher-made questionnaire consisted of 3 sections and 12 items. A mixed-method approach using quantitative and qualitative data was employed. The findings indicate that emergency transition to the new format did not affect students' satisfaction or results of the presentation. Overall, students were more than satisfied with the online format, and the results were similar, if not better, than in previous years. To minimise the disadvantages of this format, recommendations for teachers and students were offered. This study might offer new insight on the most appropriate and beneficial oral testing system for students and staff.
Keywords: Covid-19, higher education, online education, thesis proposal, online oral assessment
Introduction
At the end of their final year at the university, undergraduate students of the Business and Management Department at HSE University, Moscow, Russia, submit their Bachelor's thesis proposal, which should be written in English, according to the HSE internal educational regulations. English is an important component of any HSE University educational program as undergraduate students are expected to complete at least one professional course in English per year as part of their individual curriculum for the third and fourth years of study. The ability to read professional literature, communicate with foreign professors, write and study in English is one of the key requirements for HSE University students. Additionally, presenting a thesis proposal in English allows assessment of both subject knowledge and linguistic ability, in line with Content and Language Integrated Learning (CLIL) methodology where content and language are taught and tested together.
Through the written paper and oral presentation of the thesis proposals, the undergraduates are assessed on their ability to use English in an academic and professional environment, namely, to plan, report, and present research in public.
The written part of the thesis proposal and its oral presentation are two interconnected, but separate aspects of the academic activity, which demonstrate different academic skills and are graded separately for "talking and writing are two very different modes of communication that mediate the world differently" (Schoultz, Saljo & Wyndhamn, 2001, p. 213). If a well-written paper is poorly presented, the cumulative grade goes down. Therefore, in order to get high grades, the presentation, which introduces the transition from writing to talking about the research, should be well structured, properly rehearsed, and delivered with confidence.
By the end of the first semester of 2019-2020 academic year, the written part of thesis proposals had been graded, and the students were preparing for their oral presentation accompanied by PowerPoint slides, which is
Research Articles
This article is published under the Creative Commons Attribution 4.0 International License.
a 20-minute presentation on campus with two examiners - a teacher from their academic English course and a teacher from the Business and Management Department.
However, the second semester of 2019-20 has been defined by Mohmmed et al. (2020) as 'emergency remote teaching' as it was prepared at very short notice. In March 2020, the HSE University, like many other universities in Russia and worldwide, had to move its courses online in order to promote social distancing during the Covid 19 pandemic. That meant the assessments had to be moved online as well. Along with all the other exams, the delivery mode of the thesis proposal presentation shifted online. In spring and summer 2020, many universities offered students in their final year the opportunity to undertake an online assessment on a 'help not hinder' basis to ensure they can graduate on time with quality assured degree (UNESCO, 2020)1. A well-known telecommunication application, Skype (www.skype.com), that provides video and voice calls between computers, tablets, and mobile devices, was used to administer the video-conferencing version of the presentation because it is freely accessible and it meets the basic requirements for holding and assessing the oral presentations.
Thus, almost overnight, virtual instruction became not just a possible option but also a vital necessity that posed many challenges for both the faculty and students. Teachers had to redesign their courses and assessment to suit the new format of teaching, and the students had to get accustomed to a different mode of learning, which initially could have caused some difficulties.
The aim of this research is to explore undergraduate student attitudes towards their thesis proposal presentation transitioned from face-to-face into online format due to safety concerns. The researchers also hope to highlight some aspects of the online format of presentations that could be improved in the future for use in the university setting.
More specifically, the current study explores the following research questions.
1. What are the students' attitudes towards the online presentation?
2. To what extent are students satisfied with their presentation considering the transition into the new format?
3. Is there any significant correlation between students' attitudes towards the online presentation, their overall satisfaction with their presentation and exam results?
4. What are the positive and negative sides of the online thesis proposal presentation from the students' perspective? How can it be improved in the future?
Literature Review
Oral Assessment in Higher Education
Assessment in higher education permits instructors to make decisions about individuals' progress or the efficiency of programs regarding student learning, the curriculum, and instruction (Nitko, 2004; Reynolds, Livingston, & Wilson, 2006) and finds ways to assure and enhance the quality of the educational provision.
'Oral assessment' is defined as any assessment of student learning that is conducted by word of mouth (Joughin, 2010)2. It can be exclusively oral, or it can be combined with other modes of communication, depending on the nature of the assessment task. Oral assessment may involve not only a student who speaks but also assessors posing questions orally, with varying degrees of spoken interaction as the assessment proceeds.
Oral assessment is applicable to almost any kind of learning outcome. It is particularly useful in relation to students' problem-solving abilities, where they need to apply their knowledge in new situations (Van de Watering et al., 2008). Singh (2011, p.256) argues that oral assessment "enables the learner to identify with,
1 UNESCO. (2020). Exams and assessments in Covid-19 crisis: Fairness at the centre. https://en.unesco.org/news/exams-and-assessments-covid-19-crisis-fairness-centre
2 Joughin, G. (2010). A short guide to oral assessment. Leeds Met Press in association with University of Wollongong
interact with and therefore understand the material better because of the continual interaction and discussion with their peers and the assessors."
Kehm (2001, p.27) describes one of the strengths of oral assessment as its ability "to distinguish superficial from real knowledge through in-depth questioning." Unlike a written exam, assessors can ask the student to elaborate on an answer with carefully graded questions until they have reached the limit of what the student knows as according to Joughin and Collom (2003, p.2), "the opportunity in oral assessment to probe understanding through follow-up questions can encourage deep approaches to learning."
When students are subjected to questions and probing of their understanding, they must rely on their own work, thereby reducing the possibility of plagiarism. As Tinkler and Jackson (2004, p.104)3 points out, "although the presentation serves different purposes [...] it should serve to examine that the candidate has actually undertaken the work presented in the thesis: that is, it should authenticate the thesis."
Shifting Towards an Online Mode of Assessment
Video-conferencing applications are now commonly used for teaching purposes. However, there is little research into the effect of the video-conferencing mode on oral assessment with the exceptions of Kim and Craig (2012), Okada et al. (2015), Nakatsuhara et al. (2017), Akimov and Malin (2020).
Kim and Craig (2012) studied two modes of oral interviews, including face-to-face and video conference. Their findings indicated no significant difference either in performance between test modes or in terms of comfort, computer familiarity, environment, non-verbal linguistic cues, and speaking opportunity, which evidenced the comparability of the video conference and face-to-face interviews.
The findings of Nakatsuharaet et al. (2017) were consistent with the previous research, though some minor differences were observed in test-takers' functional output and the behavior of examiners who served as both raters and interlocutors.
The key benefits of the video-conferencing mode identified from the perspective of assessors and students, according to Okada et al. (2015), were a reliable examination, credible technology, authentic assessment, interactive e-viva, low cost, a scalable process, and practical testing in terms of time, effort and money. Thereby, based on the findings received from the previous research, the video-conferencing version of oral assessment could be used as a relevant substitution to the face-to-face mode of assessment.
Akimov and Malin (2020) described a case study of an online oral examination of postgraduate finance students. According to their research, online oral assessment tests more in-depth knowledge, develops communication skills and the ability to think rapidly, eliminates cheating in assessment, which is a serious concern for courses delivered online.
Before the Covid-19 pandemic, the use of video-conferencing and other online teaching tools was an option, which added value to the teaching-learning process. During the Covid-19 pandemic, the significance of technology for university education increased dramatically. Since 2020, it has become a primary means of remote delivering classes to students. Numerous recent studies have attempted to evaluate the impact of emergency transition into the online mode of teaching on pedagogical approaches and the learners' perceptions(Ajmal et al., 2020; Alfiras, Bojiah, & Yassin, 2020; Crawford et al., 2020; Karim & Hasan, 2020).
Crawford et al. (2020), in their research, found that many universities around the world have responded to the current crisis through rapid redesign of their curriculum to satisfy the remote teaching needs. They also pointed out that there "might be some issues in implementing online learning relating to the availability of the right hardware and software, networks, and storage capacity" (Crawford et al., 2020, p.11).
3 Tinkler, P., & Jackson, C. (2004). The doctoral examination process: A handbook for students, examiners and supervisors: A Handbook for Students, Examiners and Supervisors. McGraw-Hill Education (UK).
Teachers are expected to implement new practical strategies to conduct engaging and effective online classes, which is difficult to achieve without students' interaction. The more students interact with each other in the online classroom, the more motivated for learning they are because they learn not only from their teachers but also from their peers (Ajmal et al., 2020).
The findings of the research conducted in Saudi Arabia in a period of an unplanned shift from traditional learning to emergency e-learning (Karim & Hasan, 2020) are comparable to previous findings of Kim and Craig (2012), Okada et al. (2015), and Nakatsuhara et al. (2017). Exploring undergraduate medical students' preferences regarding the effectiveness of emergency online learning, Karim and Hasan (2020) indicated that the majority of students were satisfied with virtual classes and found them more convenient compared to face-to-face classes; the students felt more confident and believed that remote learning could replace face-to-face mode of learning. That means the curriculum designers should consider the pedagogical practices and student preferences while designing the courses.
However, other authors (Alfiras, Bojiah, & Yassin, 2020) question the usefulness of such an approach. They found that remote classes were more effective when teaching theoretical courses and less effective when teaching practice-oriented courses aimed at training social skills, for example, on a Human Resources Management programme. They believe that a distant mode of teaching and learning might be warranted for a pandemic period only, and faculty, staff, and students should return back to campus and resume face-to-face classes as soon as the university receives approval from state and local government public health authorities.
Overall, these studies highlight the need for further observation and analysis of the current situation in higher education. However, much of the research up to now has been focused on the new pedagogical concepts and modes of delivery of educational content as well as students' and teachers' perceptions in response to Covid-19 lockdown rather than remote assessment practices and especially remote oral assessment in an EFL setting.
The Use of Skype in the EFL Classroom
Skype is a well-known communication platform recognized by many EFL practitioners for remote teaching of foreign languages (Develotte, Guichon, & Vincent, 2010; Eaton, 2010; O'Dowd, 2011; Taillefer& Munoz-Luna, 2014; Kotula, 2016; Yen, Hou, & Chang, 2017; Fedorova, Rasskazova, &Muzafarova, 2018; Dabic, Grkavac, Stojanov, &Suzic, 2019). In this section of the literature review, Skype is considered as a triple-action tool, which is used to promote intercultural experiences for EFL learners, to reinforce the technology skills of language instructors, and to create an interactive environment for online teaching and learning.
The studies by O'Dowd (2011), Taillefer& Munoz-Luna (2014), Fedorova et al. (2018) and Dabic et al. (2019) consider Skype to be a good tool for promoting intercultural experiences for EFL learners as it provides barrier-free communication with people all over the world. "By connecting classrooms and students from all the continents, at the same time, Skype in the Classroom connects different races, religions, socioeconomic classes and cultures as well" (Dabic et al., 2019). As geographical boundaries disappear in the virtual world, there is an opportunity to choose a teacher/learner from anywhere in the world (Fedorova et al., 2018) or to develop speaking skills with peers from other countries by participating in the Skype-mate Language Project (Taillefer& Munoz-Luna, 2014).
Eaton (2010) and Develotte et al. (2010) believe that due to its simplicity, Skype is an excellent tool to help EFL teachers build their technology skills. "Skype is an effective way to experiment while minimizing the risk of things going wrong. [...] For teachers who are reluctant to use technology due to lack of skills or confidence or high levels of anxiety, getting started with simpler tools may be an effective way for them to explore and incorporate new technologies" (Eaton, 2010 p.1). Develotte at al. (2010) posits that EFL teachers should have some skills of using video-conferencing platforms as a part of their professional repertoire, as "they will increasingly be required to exploit the multimodal potentialities of online communication in their teaching" (p. 293).
Skype is also an effective tool to teach languages, as it incorporates a high level of verbal interaction between users (Yen et al., 2017). The research conducted by Dabic et al. (2019) has proven many benefits of using Skype in the classroom. Students improved their speaking and listening skills, built up their vocabulary, their level of
anxiety reduced, and they practised better pronunciation patterns. Collaboration through Skype motivated them to learn more as they practised English in real communication. Due to its multimodal nature (video, audio, text, webcam image, still image), teaching with Skype can achieve interactivity, which qualitatively contributes to comprehension through visual and verbal representations (Develotte et al., 2010).
The use of Skype in educational contexts has been expanding as numerous lessons, and online conferences took place via this application during the coronavirus pandemic. The number of daily active users of Skype has increased to 40 million people from 23 million before the pandemic, which is a 70% increase (Thorp-Lancaster, 2020)4.
What made many EFL teachers choose Skype for their distant classes? The most obvious advantages of Skype mentioned above are the free unlimited calls anywhere in the world (compared to 40 minutes of free calls via Zoom), Skype can be operated on computers, mobile phones, tablets, a web browser, and it is available for all operating systems.
With the updated version released in 2020, Skype became faster to load, more reliable, and interactive. Now it is even easier to use: no sign-up is required. To start a meeting, a user can create a meeting link and share the link with participants. If participants do not have Skype installed, it will open in their browser. The number of the potential participants invited in one call has been increased to 505.
Some useful features for oral assessment in teaching EFL include recording the call and saving it for later review and note-taking, which allows the participants to be engaged in an online meeting without any distractions. The recording is stored in the cloud for up to 30 days. Secondly, the participants can easily share presentations, work materials, or designs in their conference call to provide a collaborative environment for the work revision. Thirdly, if the teacher is working from home, they can choose the pre-installed backgrounds or upload their own to maintain privacy in their home. Finally, the participants may enable live captions and subtitles and read the words that are spoken during an audio or video call or even translate conversations in real-time with a built-in voice translator in up to ten different languages, which may be helpful in some cases of miscomprehension.
With all these interactive features, Skype can help the instructors and learners to create a sense of presence, which is defined by Jonassen et al. as "the degree of salience of the other person in the mediated interaction and the consequent salience of the interpersonal relationship" (Jonassen et al. 2005, p.266). Webcamming creates a sense of presence, establishes a connection between the participants, and enhances the teacher's influence on the learners (Develotte et al., 2010). The basic difference between teaching in the classroom and teaching online lies in the separation of the instructor from the learners and learners from each other (Lehman &Concei^ao, 2010). This separation often leads to a feeling of isolation on the part of the participants and has been a major cause of learner dissatisfaction in the distant learning environment (Palloff& Pratt, 2010). However, the specific nature of contact with the interlocutors (the lack of a possibility to interact in a common space) is argued by Kotula (2016) as one of the limitations of Skype.
Overall, Skype has been recognised as a valuable classroom asset in many studies. In our research, it has been used for the assessment of thesis proposal presentations delivered by undergraduate students and in this particular context; however, Skype has not yet been described and analysed in past studies.
Since the beginning of the quarantine in Moscow almost coincided with appointed dates for the thesis proposal presentation, the administration staff had a very limited time (about two weeks) to organise the transition from face-to-face into an online format. We hypothesized that the students would not be sufficiently prepared for the new format and would experience a number of difficulties during the exam, which would result in a low level of satisfaction with their performance. Therefore, the present study fills this gap in research related to the use of video-conferencing in EFL oral assessment.
4 Thorp-Lancaster, D. 30 Mar 2020 Retrieved from https://www.windowscentral.com/skype-sees-bump-40-million-daily-users-big-increase-calling-min-utes
5 Call up to 50 people at once with Skype! (2019, April 04). Skype. Retrieved from https://www.skype.com/en/blogs/2019-04-group-video-calling/
Methodology
Participants
Fifty-five students (26 male and 29 female) in their fourth year of a Business Informatics Bachelor's programme at HSE University, Moscow, participated in the research based on convenience sampling. Their English proficiency was tested at B2-C1 according to the Common European Framework for Languages (CEFR). The age of the participants was between 20 and 22 years old. Following their online thesis proposal presentation, all the students agreed to fill out the questionnaire.
Instruments
A mixed-method approach using quantitative and qualitative data was employed. We designed a questionnaire in Google forms (given in Appendix 1). The pilot study was conducted to check the clarity of the questions and to identify potential problems. After that, the questionnaire was distributed online among the undergraduate students. The participants were required to provide their informed consent before they started the survey. Respondents were anonymous throughout. To improve the response rate, the instructors sent reminder emails to students to encourage them to fill in the online survey.
The questionnaire consisted of 3 sections and 12 items. In section 1 (items 1-8), we asked the respondents to rate their experience based on eight aspects of the online presentation to measure their attitude according to a five-point Likert scale (1=very poor, 2=poor, 3=okay, 4=good, 5=excellent). In section 2 (item 9), we investigated a self-reported satisfaction with the online presentation according to a 5-point scale, where 5 means completely satisfied and 1 means completely unsatisfied. In section 3 (items 10-12), we asked the respondents to answer three open-ended questions to elicit extra information about the advantages and disadvantages of an online thesis proposal presentation.
Data Analysis
Data were quantitatively analysed by means of IBM SPSS Statistics for Windows, Version 21.0. Spearman's rank-order correlation was used to assess the interdependence of data. The open-ended questions were analysed qualitatively with the use of the Microsoft Excel package.
To analyse responses and interpret qualitative data, thematic analysis has been used to determine common perspectives among participants. The authors implemented an inductive approach by identifying themes in semantic content, which ensures the analysis process was driven by the collected data rather than any analytic preconceptions. The reliability and validity of the results for a given data sample were ensured by using proprietary software products and methodology for the thematic analysis described by Braun and Clarke (2006).
Procedure
The thesis proposal presentations took place in March 2020 in the form of an online video call session with two examiners. One of the examiners was a teacher from the academic English course, and their assessment was mostly focused on such aspects as grammar accuracy, the range of vocabulary, pronunciation, presentation skills, etc. The second examiner was a teacher from the Business and Management Department; their assessment was focused on the presentation content and structure of the thesis proposal. The use of two examiners increased the inter-rater reliability.
Prior to the oral exam, the undergraduates had to familiarise themselves with the format and requirements of the thesis proposal presentation and ensure that the internet connection, required software (Skype), and peripherals worked properly. At the scheduled time, they joined the video call with the examiners, authenticated themselves by showing a student ID card, and delivered their presentation accompanied by slides, sharing their screen with the examiners. The examiners asked some followed-up questions about different aspects of the thesis proposal. Each session lasted for 15 minutes.
To ensure the consistency and fairness of assessment, a rubric (given in Appendix 2) with explicit criteria was used by both examiners. The examiners discussed the grades, calculated the arithmetic mean, and sent the final grades to the students.
Results
The students' feedback on the online questionnaire provided the data regarding the quantity and quality of undergraduate students' perception of the online format of the thesis proposal presentation.
Quantitative Data Analysis
For further quantitative analysis, three variables were taken into consideration. Mean values were calculated and compared to find out if there is any correlation between them.
Analysis Related to Students' Attitudes towards Online Presentation (RQ 1)
Descriptive statistics of students' responses about their attitudes towards online presentation are summarized in the following table.
Table 1
Students' Attitudes towards Online Presentation
Satisfaction parameters Very poor(1) Poor (2) Okay (3) Good (4) Excellent (5) Total Mean D Std
Transition into online format 0% 0 0% 0 16% 9 16% 9 68% 100% 37 55 4.51 0.58 0.76
Exam administration 0% 0 7% 4 20% 11 38% 21 35% 100% 19 55 4.00 0.84 0.91
Skype as a tool 5% 3 7% 4 44% 24 31% 17 13% 100% 7 55 3.38 0.96 0.98
Network connectivity 2% 1 10% 6 20% 11 33% 18 35% 100% 19 55 3.87 1.13 1.06
Length (15 min) 2% 1 2% 1 18% 10 22% 12 56% 100% 31 55 4.29 0.90 0.95
The level of students' comfort 5% 3 13% 7 20% 11 31% 17 31% 100% 17 55 3.69 1.41 1.19
The quality of assessment 2% 1 3% 2 18% 10 36% 20 41% 100% 22 55 4.09 0.88 0.94
Students' satisfaction with their results 7% 4 7% 4 14% 8 30% 16 42% 100% 23 55 3.91 1.50 1.23
Overall, students reported being satisfied with their online thesis proposal presentation experience, with an average satisfaction score (Satl) of 3.97. They were most satisfied with the shift of the exam to an online format (4.51) and least satisfied with Skype as a medium for the presentation (3.38). In addition, students also rated timing of presentation (4.29), the quality of assessment (4.09), administration of oral exam (4.0), satisfaction with their results (3.91), network connectivity (3.87), and the level of comfort during the video call presentation (3.69).
Analysis Related to Students' Self-reported Satisfaction with the Online Presentation (RQ2)
On average, regardless of challenges due to the distant format of presentation, students seem to be satisfied with the new type of assessment experience. Their self-reported satisfaction (Sat 2) with their thesis proposal presentation scored 4.2.
Table 2
Students' Self-reported Satisfaction with the Online Presentation
Satisfaction parameters Completely unsatisfied (1) Unsatisfied (2) Neutral (3) Satisfied (4) Completely satisfied (5) Total Mean D Std
2% 5% 10% 40% 43% 100%
Self-reported satisfaction 1 3 6 22 23 55 4.2 0.89 0.95
with the online presentation
Correlation between students' attitudes, self-reported satisfaction and grades for the online presentation (RQ3)
To answer the third research question, we first presented the result of analysis related to students' grades for the online presentation. The majority of students (75%) successfully presented their thesis proposals and received high grades; 18% generally addressed the task but with some limitations; 7% of the students did not address the requirements of the task in many aspects. The mean value of the grades students received for their presentations (Exam results) is 4.07.
Table 3
Students' Grades for the Online Presentation
Level of achievement Very poor (1) Poor (2) Acceptable (3) Good Excellent (4) (5) Total Mean D Std
The number of students 0% 0 7% 4 18% 10 35% 40% 19 22 5°°% 4.07 0.87 0.94
Figure 1 shows students' satisfaction levels are high. Even challenging experiences scored fairly average in some aspects, were given a high overall score: Students' attitudes towards the online presentation 3.97, students' self-reported satisfaction with the online presentation 4.2, and students' grades for the online presentation 4.07. The lines representing students' attitudes and students' grades are very similar, intersecting at some points and deviating by minor values.
Figure 1
Correlation between Students' Attitudes, Self-reported Satisfaction and Grades for the Online Presentation
Students' attitudes towards the online presentation Students'self-reported satisfaction with the online presentation Students' grades for the online presentation
VERY POOR POOR NEUTRAL GOOD EXELLENT
Further data analysis performed by Spearman rank-order correlation demonstrated that there is no association between the grades students received for their presentations and their overall satisfaction with their online thesis proposal presentation.
Table 4
Spearman Rank-order Correlation
Correlation t-Statistic Probability
Exam results Sat 1 Sat 2 Exam results Sat 1 Sat 2 Exam results Sat 1 Sat 2
Students' grades 1.000000
Students' attitudes 0.116628 1.000000 0.854898 - 0.3965
Students' self-reported satisfaction 0.105461 0.058827 1.000000 0.772075 0.429012 ----- 0.4435 0.6697
Qualitative Data Analysis
In the third part of the questionnaire, we asked the participants to state the negative and positive sides of the online thesis proposal presentation. We also asked them to state their suggestions for improving the online presentation.
Table 5
Percentage of Students' Responses to Advantages, Disadvantages, and Suggestions for Improving the Online Presentation
Advantages Disadvantages Suggestions
(mean percentage of data) (mean percentage of data) (mean percentage of data)
100% 43% 78%
The findings from the quantitative analysis of fifty-five comments (some of them included more than one key point) illustrate that all the students expressed their opinions about the advantages of online presentation (Table 5). More than half of respondents did not cite any disadvantages because they did not find any, and 78% of them had some suggestions for improving the online presentation.
The Advantages of Online Presentation
Participants' comments about the positive side of online presentation were analysed using an inductive approach allowing the data to determine seven themes.
Table 6
Thematic Framework for Students' Positive Comments about Online Presentations
How many times
Theme name ,
it was mentioned
Theme 1 Less stressful 20
Theme 2 Time saving 15
Theme 3 Home environment 12
Theme 4 Time management 11
Theme 5 Safety (no risk of catching a virus) 6
Theme 6 Friendly examiners 4
Theme 7 Innovative format 3
Table 6 summarises frequencies of the themes in the first data set. Most commonly identified advantage was that online exams were less stressful than face-to-face ones. Comments included:
"I wasn't as nervous as if I presented in front of my groupmates and examiners face-to-face." (Student 1) "Staying at home and speaking via Skype is definitely less stressful." (Student 2)
The second most frequently mentioned advantage was that participating online is more time saving compared to face-to-face presentation.
"You don't need to spend time on commuting or waiting for your turn in the university atrium." (Student 3)
"I have more time to prepare for the exam, that is why I felt more confident during my presentation" (Student 4).
"You don't have to wait for your turn since everyone has a specific time slot" (Student 5)
Another beneficial factor noted by the respondents was the opportunity to stay at home, which allowed them to feel more relaxed and not be distracted.
"At home, I could better focus on my own presentation rather than hearing other peoples' rambling discussion of their performances." (Student 6)
Time management of the online format of presentation was perceived as an advantage. "Fifteen minutes is just enough for a student to present their proposal." (Student 7) "The exam was held according to the schedule without any delays or technical troubles." (Student 8)
Other positive factors mentioned by the respondents include safety, the friendliness of the examiners, and the innovative approach to the oral exam.
"There was no contact with other people; because of the coronavirus, this point was crucial." (Student 8)
"The examiners were very friendly! The questions were comprehensible and to the point." (Student 9)
"I liked this format. The same as at university, but I didn't spend time on transport. A very good idea and innovative format"! (Student 10)
Reasons for positive student feedback originate from the general understanding that today's tech-savvy generation finds it easy to adapt to new learning modalities. For them the online format of presentation might not be a challenge but a new experience.
The disadvantages of online presentation
Respondents were also questioned to report on negative aspects of the online thesis proposal presentation. Table 7 illustrates these findings and summarises frequencies of the themes in the second data set.
Table 7
Thematic Framework for Students' Negative Comments about Online Presentations
How many times
Theme name ,
it was mentioned
Theme 1 Anxiety about technical issues 12
Theme 2 Psychological issues 10
Theme 3Technical issues 7
Theme 4 Schedule shifts 6
Theme 5 No feedback from the examiners 5
Almost half of the respondents (twenty-one students) reported nothing negative about the online presentation, which means they quite smoothly and naturally transitioned their presentation skills into the online format without having any negative experiences.
"The online format of the exam was even better than the face-to-face exam which I am used to." (Student 11)
However, some aspects require closer consideration such as comments on the additional pressure felt regarding potential technical issues which was the most frequently mentioned disadvantage. Comments included:
"To the usual stress about the exam, some additional factors were added, such as anxiety about technical problems and the inability to establish eye contact with the examiners." (Student 12)
"Iworried that there would be technical difficulties and there were some." (Student 13)
"I didn't know how well the examiners could hear me. I feel more comfortable presenting face-to-face rather than online." (Student 14)
However, only seven reports cited actual technical problems during their presentation.
"I had a problem with screen sharing. Sometimes, I couldn't hear the examiner well." (Student 15) "I don't use Skype, so I had to spend some time in order to understand how it works." (Student 16) "I couldn't hear the examiners' questions well, because the network quality was poor." (Student 17)
A few students mentioned that online communication with the examiners was more stressful for them due to such psychological issues as lack of non-verbal communication, eye contact, and the inability to establish emotional contact with the examiners.
"I was stressed because using Skype you are not able to establish eye contact with the examiners." (Student 18)
"At the end of the exam, the examiners were exhausted and in a bad mood, which put me under even more pressure." (Student 19)
Some students complained about the changes in the schedule during the online event.
"My exam started six minutes late and the examiners were hurrying me. As a result, I became even more nervous." (Student 20)
"Because of some shifts in the exam schedule, my exam started later than I expected." (Student 21)
After face-to-face presentations, the students can receive feedback from the examiners in the form of verbal and non-verbal cues (Murray, 2015), which is not so obvious in an online format. Students mentioned that having no feedback from the examiners was disconcerting.
"I would prefer to receive feedback on my presentation and understand the examiners' reaction to my answer, which I could not observe on Skype." (Student 22)
Participants' suggestions for improving the online presentation
Based on the experience gained from their online exam the respondents offered a number of tips concerning the exam platform, administration, and their expectations of the examiners' behaviors that could improve the quality of experience in the future (Table 8).
Forty-three comments were available for analyses; twelve respondents did not have any suggestions, because they were fully satisfied with the procedures and the format.
Table 8
Thematic Framework for Students' Suggestions about Online Presentations
Theme name How many times it was mentioned
Theme 1
Use of an alternative conferencing platform 18
Zoom 12
Discord 5
MS Teams 1
Theme 2
To conduct a brief orientation session with the examiners just before the exam 8
Theme 3
To make an online spreadsheet for signing up for the online exam 6
Theme 4
To increase the examination time up to 20 minutes 5
Theme 5
To provide all the students with the feedback about their performance 5
Theme 6
To keep the web cameras turned on during the online presentation 4
Theme 7
To have a warm-up conversation at the beginning of the online presentation 3
The most frequently mentioned suggestion was to substitute Skype for some other conferencing platforms such as Zoom, Discord, or MS Teams. Given that Skype offers all the communication features needed for the oral assessment, it could be that other platforms are more popular and technically more secure for the younger generation. The second frequently mentioned suggestion was that an online session with the examiners explaining the entire procedure just before the exam could considerably reduce stress levels.
Other suggestions indicated by an equal number of respondents included: allowing students to sign up online at a time of their choice, ensuring availability and punctuality; highlighting those who had already presented their proposals, to avoid confusion with the schedule; increasing the examination time to 20 minutes per student instead of 15 minutes; keeping examiners' webcams on; providing examiner feedback to all the students with the feedback about their performance; having a warm-up conversation between a student and the examiners to relax students and help them speak more naturally. As they are based on students' experiences, all the suggestions are valuable and should be taken into consideration when planning online presentations in the future.
Discussion
The main findings of this research report are that undergraduate students rated their experience of online orals examinations positively in terms of Sat 1 (3.97), Sat 2 (4.2), and rated satisfaction with exam results (3.91). Contrary to expectations, the initial hypothesis of the research: the urgency of the transition into online format resulted in the low level of students' satisfaction with their performance at the online presentation was not confirmed. This finding is consistent with that of Okada et al. (2015), whereby students reported positive attitudes towards their experience of an online oral exam, considering it more beneficial for their performance in comparison to the face-to-face alternative. It seems possible that these results are due to the fact that the Business Informatics Bachelor's programme is aimed at training professionals in the field of ICT in business, and the undergraduates were quite advanced in many technical issues, that is why the forced transition to online format did not greatly affect their perception of the online presentation and made it possible to cope with the challenges of the new assessment format.
Another finding, which matches earlier studies (Joughin, 2010), is the good performance in online oral assessment with the mean grade of 4.07. It suggests that students were well prepared for this kind of assessment, which ultimately led to better results. As Joughin (2010, p.5) observes, "assessors often express surprise at how well their students perform in oral assessments - it may be that oral assessment can be particularly good for
probing the upper limits of a student's knowledge." The change of format did not produce a negative effect on the students' performance.
One unanticipated finding was that Skype as a medium for the presentation was rated 3.38, and many students suggested replacing it with other video-conferencing platforms even though it provides all necessary features for holding and assessing the online oral exam such as screen sharing, meeting recording, cloud storing, a whiteboard, file sharing, and joining via call. A possible explanation for this might be the fact that Skype, being launched in 2003, currently has many competitors with the same functions, which are more familiar to the younger generation.
The results reported herein should be considered in the light of some limitations. The first limitation is the number of participants; the second one concerns the data obtained about the students' perception, based on their own reference frames and expectations. Even though this study uses a relatively small sample size, the findings show both instructors and students what may be useful to consider in online oral examining in the future. It is also important to mention that satisfaction is a personal evaluation, "even two people with similar backgrounds can experience and evaluate their service differently" (Study portals, 2019, p.10)6. However, students are typically satisfied when they feel supported by their instructors and when their learning experience is diverse and motivating. Students' opinions are important in creating an overview of the performance assessment procedures universities provide and indicate where improvements are required.
In order to make balanced decisions, it is important to observe the situation as a whole and consider the online assessment procedures from both perspectives - students and instructors, that is why further research could be focused on studying the instructors' satisfaction with the online exam. However, some steps to minimise any possible disadvantages of this format can be taken based on the results of the present research.
Recommendations for the Instructors
The instructors are recommended to provide students with detailed guidelines about different aspects of online presentation in advance concerning the format, requirements, assessment criteria, and equipment. It is important to make sure that all the students are treated fairly and given equal opportunities to display their knowledge and receive equal feedback in terms of the quality, quantity, and timing of feedback. The examiners should record the students' presentations for an unbiased assessment or in case of appeal. Skype allows the recording and storing of audio or video. After the exam, a separate short individual session should be scheduled with each student to provide them with feedback on their presentation.
To reduce anxiety, the chairperson should make introductions, so they feel comfortable in this challenging situation. Examiners should be asked to keep their video cameras on throughout the exam to maintain a connection between the participants, provide a feeling of presence, and improve the quality of the pedagogical relationship.
An administration staff member should be invited to assist the examination board with managing time, resolving queuing/schedule issues, monitoring all the sessions in real-time, and provide technical assistance to examiners and students if required. In the case of an internet malfunction, they should provide the students and examiners with alternative scenarios or make another appointment.
Administration and teaching staff should consider the potential of other video conferencing platforms for the remote online presentation and choose the most appropriate one for their circumstances, and provide relevant teacher training on the chosen platform if needed.
Recommendations for Students
In order to successfully go through the online exam presentation and make the most of this opportunity, the following advice might be useful. It is wise to consult the guidelines on the format and requirements of the oral
6 Studyportals. (2019). Global Student Satisfaction Report: a 2019 Global Overview. https://www.studyportals.com/wp-content/uploads/2019/06/Glob-al-Student-Satisfaction-Report-A-2019-global-overview.pdf
presentation and know what to prepare for it in advance: which software or apps should be installed and if they work properly. A trial call to a groupmate or a family member will help a participant to make sure that the microphone, speakers, or headset are operable. A laptop, computer, or mobile phone should be charged, and the internet connection should be fast enough to take the online exam. A check of the equipment before the online event is essential for students as it could help to reduce stress due to potential technical problems that may occur during the presentation.
To reduce the level of anxiety during the presentation, the students are recommended to rehearse it in an online format in front of other people. Although the 'trial run' cannot fully prepare or protect the candidate from what Wellington (2010, p.138) calls 'unpredictable moments,' it can help to build students' confidence about the online event and make them aware of how they sound, look, and how it feels to speak through the microphone and in front of the camera.
Conclusion
The present study extends the prior results, and in light of students' positive comments, the findings of our research confirm that the online format of the thesis proposal presentation can be effectively used in an EFL setting not only in response to Covid-19 lockdown. The extraordinary pandemic situation has created large disruptions in education systems all over the world, but it has stimulated innovation in teaching and assessment procedures. According to the research of the International Association of Universities (2020)7 in many higher education institutions, the move to distance learning has been an opportunity to expand flexible learning modalities, setting the stage for a permanent shift towards online learning. The transition to the online mode of exams could contribute to this process and support the continuity of learning in the future regardless of the circumstances.
Declaration of Competing Interest
None declared.
References
Ajmal, M., Alrasheedi, S., Keezhatta, M.S. & Yasir, G. M. (2020). Covid-19 and online English language teaching:
Students' response and feedback. The Asian ESP Journal, 16(5.2), 39-56. Akimov, A., & Malin, M. (2020). When old becomes new: a case study of oral examination as an online assessment tool. Assessment & Evaluation in Higher Education, 45(8), 1205-1221. https://doi.org/10.1080/02602938.202 0.1730301
Alfiras, M., Bojiah, B., & Yassin A.A. (2020). Covid-19 pandemic and the changing paradigms of higher education:
A Gulf University perspective. The Asian EFL Journal, 27(5.1), 339-247 Behling, J. H. (1984). Guidelines for preparing the research proposal. University Press of America. Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative research in psychology, 3(2), 77101. https://doi.org/10.1191/1478088706qp063oa Crawford, J., Butler-Henderson, K., Rudolph, J., Malkawi, B., Glowatz, M., Burton, R., & Lam, S. (2020). Covid-19: 20 countries' higher education intra-period digital pedagogy responses. Journal of Applied Learning & Teaching, 3(1), 1-20. https://doi.org/10.37074/jalt.2020.3.1.7 Dabic, T., Grkavac, S., Stojanov, Z., & Suzic, R. (2019, June). Language benefits with Skype in the classroom. In International Conference on Information Technology and Development of Education-ITRO (vol. 2019, pp. 1218). Zrenjanin, Republic of Serbia. Develotte, C., Guichon, N., & Vincent, C. (2010). The use of the webcam for teaching a foreign language in a desktop video-conferencing environment. ReCALL, 22(3), 293-312. https://hal.archives-ouvertes.fr/hal-00806433/
7 International Association of Universities, IAU Global Survey Report: The Impact of Covid-19 on higher education around the world, 2020 https://www. iau-aiu.net/IMG/pdf/iau_covid19_and_he_survey_report_final_may_2020.pdf
Eaton, S. E. (2010). How to use Skype in the ESL/EFL classroom. The Internet TESL Journal, 16(11), 1-14. Fedorova, E., Rasskazova, T., & Muzafarova, A. (2018). Teaching English viaSkype: Challenges and opportunities. In 12th International Technology, Education and Development Conference (INTED) (pp. 663-667). IATED. https:// doi.org/10.21125/inted.2018.1119 Jonassen, D. H., Lee, C. B., Yang, C. C., & Laffey, J. (2005). The collaborative principle in multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 247-270). Cambridge University Press. https://doi.org/10.1017/CB09780511816819.017 Joughin, G., & Collom, G. (2003). Oral assessment. Biomedical Scientist, 47(10), 1078-1080. http://dx.doi.
org/10.1007/s10459-008-9111-9 Karim, M.R., & Hasan, M. (2020). Virtual classes during Covid 19 pandemic in tertiary level in Saudi Arabia:
Challenges and prospects from the students'. The Asian EFL Journal, 27(5.1), 205-219. Kehm, B. (2001). Oral examinations at German universities. Assessment in Education, 8, 25-31. http://dx.doi.
org/10.1080/09695940120033234 Kim, J., & Craig, D. A. (2012). Validation of a video-conferenced speaking test. Computer Assisted Language
Learning, 25(3), 257-275. https://doi.org/10.1080/09588221.2011.649482 Kotula, Y. K. (2016). Teaching a foreign language in a desktop videoconferencing environment. Teaching English
with Technology, 16(3), 37-51. Lehman, R. M., & Concei^ao, S. C. (2010). Creating a sense of presence in online teaching: How to "be there" for
distance learners (vol. 18). John Wiley & Sons. Mohmmed, A. O., Khidhir, B. A., Nazeer, A., &Vijayan, V. J. (2020). Emergency remote teaching during Coronavirus pandemic: the current trend and future directive at Middle East College Oman. Innovative Infrastructure Solutions, 5(3), 1-11. http://dx.doi.org/10.1007/s41062-020-00326-7 Murray, R. (2015). How to survive your viva: Defending a thesis in an oral examination (3rd ed.). Open University Press.
Nakatsuhara, F., Inoue, C., Berry, V., & Galaczi, E. (2017). Exploring the use of video-conferencing technology in the assessment of spoken language: A mixed-methods study. Language Assessment Quarterly, 14(1), 1-18. http://dx.doi.org/10.1080/15434303.2016.1263637 Nitko, A. J. (2004). Educational assessment of students (4th ed). Pearson Education Inc.
O'Dowd, R. (2011). Intercultural communicative competence through telecollaboration. In The Routledge handbook of language and intercultural communication (pp. 340-358). Routledge. http://dx.doi. org/10.4324/9780203805640.ch21 Okada, A., Scott, P., & Mendona, M. (2015). Effective web video-conferencing for proctoring online oral exams:
A case study at scale in Brazil. Open Praxis, 7(3), 227-242. http://dx.doi.org/10.5944/openpraxis.7.3.215 Palloff, R. M., & Pratt, K. (2010). Collaborating online: Learning together in community (vol. 32). John Wiley & Sons. Paltridge, B. 1997. Thesis and dissertation writing: Preparing ESL students for research. English for Specific
Purposes, 16(1), 61-70. https://doi.org/10.1016/S0889-4906(96)00028-2 Reynolds, C. R., Livingston, R. B., & Wilson, V. (2006). Measurement and assessment in education. Pearson Education Inc.
Schoultz, A., Saljo, R., & Wyndhamn, J. (2001) Conceptual knowledge in talk and text: What does it take to understand a science question? Instructional Science 29, 213-236. https://doi.org/10.1023/A:1017586614763 Singh, P. (2011). Oral assessment: Preparing learners for discourse in communities of practice. Systemic Practice
and Action Research, 24(3), 247-259. https://doi.org/10.1007/s11213-010-9184-2 Taillefer, L., & Munoz-Luna, R. (2014). Developing oral skills through Skype: A language project analysis.
Procedia - Social and Behavioral Sciences, 141, 260-264. http://dx.doi.org/10.1016Zj.sbspro.2014.05.045 Van de Watering, G., Gijbels, D., Dochy, F., & Van der Rijt, J. (2008). Students' assessment preferences, perceptions of assessment and their relationships to study results. Higher Education, 56(6), 645-658 http:// dx.doi.org/10.1007/s10734-008-9116-6 Wellington, J. (2010). Weaving the threads of doctoral research journeys. In P. Thomson, & M. Walker (Eds.), The
Routledge doctoral student's companion (pp. 128-42). Routledge. Yen, Y. C., Hou, H. T., & Chang, K. E. (2017). Exploring the effect of VoIP tools as a foreign language instruction on learners' English speaking errors. International Journal of Continuing Engineering Education and Life Long Learning, 27(4), 339-357. http://dx.doi.org/10.1504/IJCEELL.2017.087134
OLGA STOGNIEVA, VICTOR POPOV Appendix 1
A survey on undergraduate satisfaction with the online thesis proposal presentation
Rate your level of satisfaction with the following aspects of your online presentation
Very poor Poor Okay Good Excellent
Rate the idea of the thesis proposal presentation transition to an online format due to the threat of the coronavirus □ □ □ □ □
Rate the administration of the online thesis proposal presentation. How well were you informed about the new format and procedures? □ □ □ □ □
Rate Skype as a tool for the thesis proposal presentation □ □ □ □ □
Rate network connectivity during the online exam □ □ □ □ □
Rate timing (15 min per student) □ □ □ □ □
Rate the level of comfort while presenting your thesis proposal presentation online □ □ □ □ □
Rate the quality of the assessment of your presentation □ □ □ □ □
Rate how you feel about the results of your online presentation □ □ □ □ □
1. Rate your overall satisfaction with your online thesis proposal presentation
Completely unsatisfied Unsatisfied Neutral Satisfied Completely satisfied
1 2 3 4 5
□ □ □ □ □
In addition, answer three open-ended questions:
1. What are the positive sides of the online thesis proposal presentation?
2. What are the negative sides of the online thesis proposal presentation?
3. What suggestions do you have to improve various aspects of online thesis proposal presentation?
Appendix 2
Oral presentation rubric
Excellent Good Acceptable Poor Very poor
(5) (4) (3) (2) (1)
Organization The presentation The research content The presentation The presentation The student
and structure content, structure is presented with some does not fully does not reflect the provides little of
and style deliver the imperfections. reflect the research research content. the information
message effectively. The speaker content. The speaker does required or fails
The speaker demonstrates The speaker not demonstrate to complete the
demonstrates full understanding of the demonstrates poor understanding assignment.
understanding of research done. understanding of of the research
the research done. There are certain the research done. done. Presentation
Presentation and deviations from There are quite a etiquette and
academic discourse presentation etiquette few deviations from academic discourse
etiquette is observed. and academic discourse conventions. presentation etiquette and academic discourse conventions. conventions are not observed.
Language The lexical and The lexical and The lexical and Numerous Mistakes in usage
grammatical resources grammatical resources grammatical pronunciation, are pervasive,
are adequate. are adequate. resources are not lexical, and distort meaning and
Functional clichés are Functional clichés are quite relevant. grammatical prevent effective
employed. employed. Functional clichés inaccuracies impede communication.
There are no any There are minor are virtually not communication. The student's
lexical, grammatical lexical, grammatical used. There are no Functional clichés language
and pronunciation and pronunciation more than 3 cases are either absent or is basically
inaccuracies. Terms inaccuracies which of term misuse. used incorrectly. incomprehensible.
are generally used do not impede There are more than
correctly. communication. Terms are generally used correctly. 3 instances of term misuse.
Delivery Natural, confident Clear voice, generally Low voice, Mumbles the words, Constant
delivery that does effective delivery; occasionally audience members in hesitations and
not just convey the minimal distracting inaudible; some the back can't hear problems with
message but enhances gestures, etc., but distracting filler well enough; too pronunciation
it; excellent use of somewhat monotone. words and gestures; many filler words; cause
volume, pace etc. articulation mostly, but not always, clear. distracting gestures; communication breakdown.
Mechanics The slides are clear, The slides are relevant The slides Presentation Used no visuals.
well-designed and and designed well are irrelevant, etiquette is not
not overloaded with enough. overloaded with observed. The
information. There are minor text student attempts to
The slides fully reflect imperfections or not informative, read off the paper
the presentation regarding font, color, not meeting and/ or slides.
message. images, and visual academic The slides
There are no factual effects. presentation are irrelevant,
mistakes on the slides. conventions. uninformative or absent.
Timing The timing is The timing is observed The timing is The timing is The timing is not
observed and balanced and balanced in observed but poor, e.g., only the observed.
in accordance with accordance with the imbalanced. introduction was
the presentation. presentation. presented.
