CC BY
0УДК 378.14
Kobicheva A.M.
PhD, associate professor Peter the Great St. Petersburg Polytechnic University (Saint-Petersburg, Russia)
CHATBOT TECHNOLOGY EFFICIENCY IN LEARNING FOREIGN LANGUAGES
Аннотация: the aim of this study is to analyze the effectiveness of integrating chatbot technology into a blended learning environment in foreign language learning. The study examined students' academic performance and engagement in the learning process, which were assessed in traditional blended learning (N=104) and chatbot-enabled blended learning (N=107). Mean scores, standard deviations (a), and Student's t-test for independent samples were usedfor comparison. We also calculated the Pearson correlation coefficient for all analyzed indicators. The results of the current study confirmed the positive impact of using artificial intelligence in blended learning, as it increases students' emotional engagement, which directly and significantly affects their academic performance.
Ключевые слова: chatbot technology, blended learning, engagement level, student performance.
Introduction.
The COVID-19 pandemic has forced modern education to quickly and effectively transform teaching in higher education using distance learning [1]. The problem concerns all areas without exception - technical, pedagogical, medical, as evidenced by numerous articles by teachers in various foreign journals. The educational process initially slowed down, but did not stop, it turned out that the university, staff and students were quite ready for distance learning, but the results of such learning were different [2-3]. In 2020/21, the majority (88.5%) of universities switched to the "blended learning" format. Russian universities have also switched to the new format of education [4-5].
In order to improve the teaching and learning process, we have developed an integrated blended learning model based on an artificial intelligence chatbot for the professional English course (for law students). The user interface was developed using LandBot.io (https://landbot.io/). The main menu of the chatbot included the following elements: class schedule (schedule of face-to-face classes and classes in Teams with a link to a virtual room), materials (information about the main topics that students will learn in the course) and two assignments (detailed description of each assignment, instructions for completing the assignment, information about the deadlines for submitting assignments and the form of assessment). After reading the material or watching videos on the topics, students were required to take a test to verify their understanding of the topic. Different methods were used for this, such as O/X quiz, multiple choice, and open-ended questions. Feedback was provided after each question, and it varied depending on whether the answer was correct or not. If the answer is incorrect, the learning content is rearranged so that students can study again. After successfully completing a topic test, students can move on to the next topic. Students could also enter questions and sthe answers in the chatbot through the user interface. When a question is entered in natural language, the natural language processing engine recognizes the purpose and nature of the question, and the most appropriate answer is selected and provided from the accumulated learning outcomes database. The main objective of this study is to evaluate the effectiveness of a chatbot-based course in terms of the impact of this course on student engagement and academic performance. Therefore, this paper addresses two main research questions:
1. Is the engagement level different in traditional blended learning courses and chatbot-based blended learning courses?
2. Is the academic performance of learners different in traditional blended learning classes and chatbot-based blended learning classes?
Methodology.
The respondents included 211 undergraduate students who studied professional English through a blended learning approach in the fall semesters of2021-2022. From a statistical point of view, the sample consisted of 97 males (45.97%) and 114 females
(54.03%) aged 21 to 24. A total of 8 groups of students studying law took part in the experiment. 4 groups of students studied professional English through traditional blended learning (control group, N=104), and another 4 groups studied the same course through blended learning based on an artificial intelligence chatbot (experimental group, N=107). Initial testing was conducted to identify the level of professional English and divide the groups accordingly. The testing was conducted partly via the online platform Moodle, developed for the Saint Petersburg Polytechnic University (Listening, Reading, Writing), and partly via seminars (Speaking). The last testing was at the end of the semester.
This study measured the scale of students' academic engagement using the thrmost common parameters identified by researchers (behavioural, emotional and cognitive engagement). We measured behavioural engagement (BE) through the records of students' attendance at lectures and the number of completed tasks in MS Teams (the results are presented on a 10-point scale). To identify emotional engagement (EE), we used a motivation questionnaire. For this, specific statements were created to clarify students' perceptions of the proposed teaching and learning, defining five indicators: (a) desire to study after university, (b) anxiety, (c) positive attitude towards learning, (d) self-esteem and (e) self-discipline. Responses were assessed on a five-point Likert scale. To examine students' cognitive engagement (CE), we administered a three-item survey measuring cognitive criteria that reflect the extent to which students pay attention and expend mental effort on the learning tasks assigned to them. Responses were also assessed using a five-point Likert scale.
Results.
The results of academic performance are presented in Table 1.
Table 1. Results of students' testing.
Indicators Control group Experimental group t-test results
M SD M SD
Listening 15.58 2.02 16.71 1.94 t=2.24*
Reading 16.88 1.85 17.44 1.97 t=1.78*
Writing 17.2 1.94 17.27 1.88 t=0.76
Speaking 16.91 1.78 17.02 1.95 t=1.01
According to the results, the experimental group students wrote the final test in professional English significantly better than the control group students. The experimental group students' results in listening and reading were significantly higher than the control group students' results (p < 0.05), so it can be concluded that the AI chatbot also had a positive effect on these indicators of students' professional English proficiency. The engagement level results are presented in Table 2.
Table 2. Results of students' testing.
Indicators Control group Experimental group t-test results
M SD M SD
Behavioral engagement 15.58 2.02 16.71 1.94 t=2.24*
Emotional engagement 16.88 1.85 17.44 1.97 t=1.78*
Cognitive engagement 17.2 1.94 17.27 1.88 t=0.76
The analysis of the level of engagement showed that the students in the experimental group were more engaged in the educational process, in particular, their behavioral and emotional engagement rates were significantly higher than those of the students in the control group, which also suggests that the use of AI chatbot technology had a positive impact on these indicators, confirming its effectiveness in the field of education.
Conclusion.
The purpose of this study was to study the level of student engagement and their academic performance in the implementation of an educational course based on an intelligent chatbot and compare the results with those of students studying in a traditional blended learning environment.
The contribution of this study to the field of education is its comprehensive study of AI-based blended learning and its role in the educational process. The results of the current study confirmed the positive impact of using artificial intelligence (chatbot technology) during blended learning, as it increases the level of student engagement, in particular, their attendance and completion of midterm assignments in the online environment, as well as the level of student motivation, which directly and significantly positively affects academic performance, the indicators of which reflect the effectiveness of the educational process. As for the limitations, the sample included students from only one country and humanities, while other groups of students may have their own characteristics, and the results of their level of engagement in the blended environment may differ. In the future, we plan to conduct a similar study with students from several foreign universities to confirm or expand on the current results.
СПИСОК ЛИТЕРАТУРЫ:
1. E. M. Aucejo, J. French, M. P. U. Araya, and B. Zafar, The impact of COVID-19 on student experiences and expectations: Evidence from a survey, Journal of Public Economics, vol. 191. doi:10.1016/j.jpubeco.2020.104271;
2. T. A. Baranova, A. M. Kobicheva, and E. Y. Tokareva, Effects of an integrated learning approach on students' outcomes in St. Petersburg Polytechnic University, ACM International Conference Proceeding Series, pp. 77-81, 2019. doi.org/10.1145/3369199.3369245;
3. M. Adnan and K. Anwar, Online learning amid the COVID-19 pandemic: Students' perspectives, Journal of Pedagogical Sociology and Psychology, vol. 2, no. 1, pp. 45-51, 2020. https://doi.org/10.33902/JPSP. 2020261309;
4. T. K. Burki, COVID-19: consequences for higher education, The Lancet Oncology, vol. 21, no. 6, p. 758, 2020;
5. 5. 10. Ait Baha, T., El Hajji, M., Es-sdy, Y., & Fadili, H. (2022). Towards highly adaptive Edu-Chatbot. Procedia Computer Science, 198, 397-403. https://doi.org/10.1016/j.procs.2021.12.260.
