CC BY
8
Original article
DOI: 10.14529/ped230203
REFLECTIONS ON ACHIEVEMENTS AND SHORTCOMINGS IN THE IELTS EXAM IN A TECHNICAL UNIVERSITY OVER AN EIGHT-YEAR PERIOD
O.G. RossikhinaB, [email protected], https://orcid.org/0000-0001-7569-0139 N.V. Chernushkina, [email protected], https://orcid.org/0000-0001-7261-6435 P.V. Ermakova, [email protected], https://orcid.org/0000-0003-3169-4917 National University of Science and Technology MISiS, Moscow, Russia
Abstract. IELTS exam is an internationally recognized tool to measure students' proficiency in academic English, and in anglophone countries it often plays a role of a gatekeeper to a university. At National University of Science and Technology MISiS (NUST MISiS) IELTS exam, on the contrary, completes a four-year long English course for all majors. IELTS exam was used as the unique KPI of the program effectiveness, enabling the department to monitor graduates' proficiency throughout eight years, track its dynamics against the placement test levels, and oversee the factors which affect final results. The data obtained from 5600 engineering students revealed that no more than 50% could achieve the target B2/C1. There was a strong correlation between the entrance levels and final IELTS results, except for the COVID-19 year (KPI = 38%). Trajectories drawn for each starting level statistically confirm the primacy of this factor. The entrants with the initial A1 level could hardly achieve B2 (11%), while A2 cohort were in the zone of perspective growth (57%). Among other factors that impede setting higher KPI the authors name the nature of the university environment with the high flexibility of a student population, and their motivation to take IELTS. These findings demonstrate that actual effectiveness of a program can differ significantly from theoretical expectations, and this should be taken into account by course designers.
Keywords: IELTS, program efficiency, levels of proficiency in English, individual trajectories, motivation
For citation: Rossikhina O.G., Chernushkina N.V., Ermakova P.V. Reflections on achievements and shortcomings in the ielts exam in a technical university over an eight-year period. Bulletin of the South
Ural State University. Ser. Education. Educational Sciences. 2023;15(2):29-41. DOI: 10.14529/ped230203
Научная статья
УДК 372.881.111.1
DOI: 10.14529^230203
ОБЗОР РЕЗУЛЬТАТОВ МЕЖДУНАРОДНОГО ЭКЗАМЕНА 1Е^ В ТЕХНИЧЕСКОМ УНИВЕРСИТЕТЕ ЗА ВОСЬМИЛЕТНИЙ ПЕРИОД
О.Г. Россихинав, [email protected], https://orcid.org/0000-0001-7569-0139 Н.В. Чернушкина, [email protected], https://orcid.org/0000-0001-7261-6435 П.В. Ермакова, [email protected], https://orcid.org/0000-0003-3169-4917 Национальный исследовательский технологический университет МИСиС, Москва, Россия
Аннотация. Международный экзамен ГЕЬТ8 широко используется в мире для подтверждения готовности студента учиться в англоязычном университете; в нем оцениваются как языковые, так и академические навыки. В Национальном исследовательском технологическом университете МИСиС (НИТУ МИСиС), сдача ГЕЬТ8 всеми выпускниками бакалавриата, напротив, завершает четырехлетний курс английского языка. В статье описывается опыт использования ГЕЬТ8 как уникального показателя эффективности программы языковой подготовки, что дало возможность оценить потенциал программы и факторы, влияющие на конечный результат. Данные, полученные от 5600 студентов-бакалавров за восемь лет наблюдений, показали, что в реальности не более 50 % выпускников смогли достичь заявленный в программе уровень В2. При этом в течение всего периода наблюдений отмечался статистически значимый рост показателей ГЕЬТ8 параллельно с повышением входных
© Россихина О.Г., Чернушкина Н.В., Ермакова П.В., 2023
уровней, за исключением лет COVID-19. Начальный уровень владения языком во многом определял и индивидуальную траекторию студента. Первокурсники с входным уровнем А1 лишь в редких случаях смогли достичь В2 (11 %) в конце обучения, в то время как студентов с А2 можно отнести к перспективной группе (до 57 %). На сегодня реалистичным является показатель эффективности в 50 %, что авторы связывают с волатильностью студенческого контингента и их недостаточной мотивированностью к сдаче экзамена. Полученные в статье данные показывают, что реальные достижения студентов на программе могут существенно отличаться от теоретических ожиданий, и это необходимо учитывать разработчикам университетских программ по английскому языку.
Ключевые слова: IELTS, эффективность учебной программы, уровни владения языка, индивидуальные траектории, мотивация
Для цитирования: Rossikhina O.G., Chernushkina N.V., Ermakova P.V. Reflections on achievements and shortcomings in the ielts exam in a technical university over an eight-year period // Вестник ЮУрГУ. Серия «Образование. Педагогические науки». 2023. Т. 15, № 2. С. 29-41. DOI: 10.14529/ped230203
Introduction
In the global higher educational landscape, the issues concerning English proficiency are being widely discussed as the language of instruction in many universities worldwide is English. It imposes certain language demands on non-native English speakers majoring in their disciplines in the EMI (English as a medium of instruction) environment. The academic performance expressed as Grade Point Average (GPA) is in most cases higher if students have B2 or higher level of English proficiency [9, 18] as compared with intermediate students. Though this correlation might be weak for higher levels, for students with intermediate English it can become a major hindrance in their studies [5, 7, 16].
There are two internationally recognized tools to measure students' proficiency in academic English and their ability to study in an EMI university, which are the Test of English as a Foreign Language (TOEFL) and the International English Language Testing System (IELTS). IELTS is taken by people who aim at working or studying abroad or in an English-speaking environment at home. It was designed by the examination committee of the University of Cambridge in cooperation with the British Council and IDP Education Australia, and currently totals more than 3.5 million test takers in the world (IELTS Guide for teachers, 2022). In anglophone universities IELTS plays the role of a gatekeeper, letting in only entrants with a suitable level of English. Although still debatable, there seems to be a consensus view that Band 6.0 (B2) is the minimal requirement, enabling students to study and graduate in the chosen field successfully.
The situation in our country differs from that in Australia, the UK, the USA, the UAE, New Zealand and even some former Soviet republics
such as Kazakhstan [1], Uzbekistan or Ukraine as universities here operate in the Russian-speaking environment (Russia was not even represented in the 2014 British Council survey of countries where EMI was reported as allowed) [6]. In view of this, though IELTS used to be the most popular exam in academic English in Russia until recently, its role in students' motivation to master English is not so obvious.
This research is conducted in the National University of Science and Technology MISiS (MISiS), which is one of the leading higher educational institutions in the fields of material science, nanotechnology mining, and metallurgy in Russia. The university had been the part of the Competitiveness Enhancement Program 5-100 and had an opportunity to become a part of international educational environment. The state project was aimed at the internalization of the education in the Russian Federation, so that Russian universities will be able to become top world institutions, attracting students, scientists and academics from other countries, producing high quality research and demonstrating good standards in teaching.
MISiS is a technical university, so our applicants do not have to submit the results of a state exam in English and are not selected by the language proficiency. Yet, the involvement into the Competitiveness Enhancement Program 5-100 led the university to revise many aspects of teaching, and the English language program in particular. Now the course is unified across all specialist departments, covers seven semesters and ends with a standardized international exam IELTS.
The article presents the experience of the Department of Modern Languages and Communication (IYAKT) with the introduction
of the compulsory IELTS exam for all undergraduates over the period from 2014 to 2021. As all MISiS students regardless of their major have to take IELTS at the end of a 4-year course, our department receives a unique tool to assess the efficiency (KPI) of the English language program.
Having such a large dataset, we were able to analyze the results with respect to objective factors such as the entry level of English, and other impacts including the volatility of the student population and their motivation to have an internationally recognized certificate in English proficiency.
Literature Review
The International English Language Testing System (IELTS) is an English language proficiency test that serves as a gatekeeping mechanism for students to enter an international university in many countries where English is used as the language of instruction. Petrishcheva [14] writes about international students from China, Vietnam, Japan, Portugal and Russia who came to New Zealand to continue their tertiary studies, Thorpe et al [18] explore the wide geography of students in UK universities, which comprises Africa, Americas, East, West and South Asia, and Europe.
University departments set a certain proficiency level at which the risk of failure is minimized, and in this regard much research deals with the predictive validity of IELTS for the future academic success expressed in GPA [7, 9, 16, 18].
There are no set requirements for the acceptable level expressed in IELTS bands, and each institution makes its own decision about admission expectations. However, higher IELTS bands (7.5 or higher) are generally recommended for the 'linguistically more demanding' courses, such as business, law, arts, humanities, and technical disciplines, such as engineering, are considered 'less demanding' with Band 6.5 as a pass. Lower entry IELTS score (5,0 or lower) might lead to a poor academic record in an EMI university even in a non-English-speaking country [16].
In Russia the incentive for students to study English and to pass IELTS successfully is more ambiguous as the language of instruction in the overwhelming majority of non-linguistic universities remains Russian. In the article on needs analysis conducted at MISiS in 2015-2016, we note that in some specialties students can get a degree without ever using English in their studies or work [15]. Having said this, most of
students admitted using English on a regular basis for reading research papers and listening to lectures and tutorials online.
Meanwhile, the role of the English language in the Russian educational landscape has increased dramatically since the Russian government announced the course to the internationalization of university education while bolstering their positions in the global education market. First, The Russian Academic Excellence Project launched in 2013 gathered twenty-one top Russian universities, then in 2021 the list was expanded to 43 participating members under the federal academic leadership program Priority 2030. One of the stated aims of both projects is 'to become the leaders in creating new scientific knowledge, technology, and developments for introduction into the economy and social sphere in Russia'.
After 2013 all top universities introduced master's degree programs in English. For example, in 2017 MISiS announced nine EMI postgraduate courses in materials science, engineering materials, nanotechnology and public relations (MISiS). Many universities offered double degree master programs, predominately with European and Asian universities (www.hse.ru; www.dvfu.ru; https://itmo.ru; www.tpu.ru;www.etu.ru). The essential prerequisite for entering international programs is a certain level of language proficiency, and the most acceptable certificate is IELTS score 5.5 or higher.
On a brief examination of the 5\100 members websites, one can find information about six international programs in collaboration with Finland and Germany in ITMO university, or several double degree projects with leading French engineering schools ParisTech in Novosibirsk State University (www.nsu.ru).
Undergraduate programs in EMI are much less common in the Russian Federation and very small cohorts of bachelors have taken part so far. In this regard we can note London Educational Project, which Financial University in Moscow implemented jointly with the London School of Economics [10], and three double degree programs realized in Higher School of Economics together with London and Seoul universities. The language requirements in both courses are IELTS Bands 6.0 or higher.
Despite the fact that IELTS is a gatekeeper for many international educational projects, to the best of our knowledge only two tertiary institutions in Russia make IELTS test compulsory
for all undergraduates (HSE). In other universities those students who need the exam certificate either attend fee-paying extracurricular courses, or prepare by themselves. The two universities where IELTS is taken by all bachelor's degree students are Higher School of Economics and MISiS. The declared aim of both English language programs is achieving B2 level CEFR, or Band 5.5 or higher IELTS.
In Higher School of Economics (HSE) the exam is taken in the second year of studies, in our university IELTS culminates a 4-year long course of English as a final external standardized control. According to the publication of Baidel-dinova [2], the results in HSE and Financial Academy are high, with around 90% of test-takers achieving at least Band 5.5, but some issues need clarification. The initial levels of students who study in the program are not specified, but it may affect the score dramatically; what is more, the results are not presented in dynamics, which can lead to overlooking some important trends.
It would be interesting to examine the issue of motivation as in Russia the relationship between effective language skills and academic achievement or career prospects is not always obvious. Motivation is widely acknowledged as a significant factor in learning, alongside with hard work, effective study strategies and supportive environment [8, 20]. Researchers distinguish two motivational orientations, namely intrinsic and extrinsic. The intrinsic motivation refers to doing something because of the satisfaction derived from the process, while the extrinsic motivation refers to doing something in order to get external outcomes [4, 17].
Many students derive enjoyment from the process of learning English, the knowledge they acquire or the sense of accomplishment, that is they have the intrinsic motivation to advance [4, 17]. External regulation comes from external rewards or punishments and is displayed when a student sees the tasks as a way to the goal to be reached. It seems obvious that the necessity to 'have one eye on the grade' in the view of increasing competition for jobs after graduation remains one of the main incentives [19]. Such categories as finance, career, job or educational program were distinguished as codes by Kember [13].
If we accept the view of a tertiary student as a 'conscientious consumer' who weighs the benefits and efforts [11], they have a good idea of what they are capable of achieving without jeo-
pardizing other things. The final IELTS exam can be considered as such a goal, and the motivation to succeed in it is a significant factor affecting the results.
In this article the authors present the reflection on a longitudinal study of the results of IELTS taken at the end of the English course by all bachelor's degree students at a technical university. The proportion of B2/C1 graduates transcribed in IELTS bands serves as a clear-cut KPI for the English course. The research investigates the processes underlying the final achievements in the English language course with the view of enabling course designers to set feasible goals and considering possible ways to improve the results.
Methodology
Participants
The data were obtained from the entire cohorts of students who entered the bachelor's program in the years from 2011 to 2018. All entrants were streamed into four trajectories, A1, A2, B1 and B1+/B2, and they moved along their trajectory during seven semesters, or 600 contact hours, of English studies.
According to CEFR Guided Learning Hours [3], this number is sufficient for a beginner to achieve B2. All MISiS institutes, College of Engineering, College of New Materials, College of Computer Technologies and College of Economics, have the same English language program, which ends with the final IELTS exam.
IELTS procedure
In 2014 and 2015 all bachelors regardless of their English achievements were examined by an official test center BKC and, consequently, received certificates with IELTS bands ranging from 2.0 to 7.0. Achieving of B2 level (Band 5.5) was stated by the department as a compulsory requirement of the English program, and undera-chievers had to go through retake procedures.
Starting from 2016 and up to 2021 students with preliminary scores 5.0 and lower were excluded from the certified exam and took a mock IELTS exam administered completely by the IYAKT teachers. In 2017 the status of IELTS was downgraded to a certain number of points in the score-rating system, so a student could be attested without achieving B2.
The other change in the IELTS procedure occurred in 2020 and 2021 due to COVID-19 pandemic restrictions, when all classes and, more importantly, exams, were taken online. Those candidates who were selected for a certified exam
did a computer-based test in the premises of IELTS Students International, while the others took a test from home on the university platform Canvas. Needless to say, the results from a certified center were credible as they eliminated cheating, whereas the online in-house test could not guarantee complete integrity.
Quantitative analysis
For statistical analysis of the data, we used a free open-source program JASP. The strength of the relationship between the placement test and IELTS results by years was measured by means of the correlation analysis. Both data sets were not normally distributed, so we used Spearman's rank correlation coefficient rho. The /»-value was used in order to quantify the statistical significance of a claim, with p < 0,05 regarded as statisically significant. The same indicators were calculated for the number of students who missed the final exam and its overall results.
Mean values for different trajectories were compared using descriptive statistics methods, and the paired samples t-test allowed us to prove that there are statistically significant differences between the final IELTS results for trajectories A1, A2, B1 and B2. The test of normality demonstrated that the difference scores were not normally distributed, so the Wilcoxon signed-rank test was applied, with p < 0,05 signified that the differences were reliable.
Qualitative data
The factors which affected students' motivation to take IELTS or the reasons to abstain from it were examined with questionnaires. High achievers, who took the certified exam in IELTS Students International test center, were asked by their English teachers to answer a few questions:
Why did you take a certified IELTS?
How are you going to use your certificate?
We received 72 responses out of 281 (25%), in which we identified the categories and counted the number of voices in each.
In the group of underachievers, who failed to get Band 5.5, 167 students did not attend any part of the exam, and 29 skipped the oral part. It turned out that 41 were factually expelled or on the academic leave, but still were registered for IELTS on paper. A small proportion from this group were attested in the semester despite IELTS failure, and the others had to take IELTS on a MISiS electronic platform.
The tests opened only upon completion of a short questionnaire, and this way we obtained
69 answers to open and closed questions. The ones to be discussed further are as follows:
Why did you fail to attend the exam?
How do you assess your chanced to get a passing score now?
Results
IELTS against placement test
The situation at our university was unique for Russia, as all undergraduates in a technical university had to take a certified IELTS exam at the end of the English language course. As there were no 'shoulders to stand on' and to base predictions on the proportion of graduate with B2 or C1, at first we were guided by CEFR, which states that 600 contact hours is sufficient for an A1 to get to B2 (Cambridge English Support Site, 2019). Consequently, in the ideal conditions we could expect at least up to 80 percent of upper-intermediate graduates. The actual results of IELTS for all years are shown in Fig. 1. As can be seen, the proportion of B2/C1 students had never been higher than 50%, with the best score in 2019 of 49,2% and the lowest number in 2014 of just 28%.
From 2014 to 2019 the proportion of B2/C1 graduates gradually increased alongside with the betterment of student intake. The placement test graph shows the percentage of entrants with B1 and B2 levels by year.
If for the 2014-2016 cohort the overwhelming majority of entrants were beginners or elementary (71 and 66%, respectively), for 2020 and 2021 graduates only around half were A1/A2 at the start of the program. A noticeable decline is observed in the COVID years, when the figures were no more than 45% in 2020 and 38% in 20211. The trend looks like a pullback to the early years of the program, when we taught much less proficient students, and the teachers at IYAKT were less experienced in IELTS training. Whatever the reasons, it can be clearly seen that in the COVID pandemic years, with learning online and taking an exam online, fewer opportunities for academic mobility and employment abroad, the efficiency of the program suffered significantly. At the same time, the proportion of high-achievers (C1) rose throughout the whole period, allegedly because of the increasing number of B1+/B2 students at the starting point.
1 In COVID years the graph presents only the bands from the certified exam from the IELTS test center. The real results from the in-house online test were up to 1,3 times higher due to cheating.
60
2014 2015 201Б 2017 2018 2019 2020 2021
Years
B2 ■ CI -Placement Test
Fig. 1. IELTS against Placement Test over the period from 2014 to 2021
Table 1
Correlation between final IELTS and placement test results
Variable Placement B2/C1 in 2014-2021 B2/C1 in 2014-2019
Placement Spearman's rho -
p-value -
B2/C1 in 2014-2019 Spearman's rho 0.893 1.000 -
p-value 0.012 < 0.001 -
B2/C1 in 2014-2021 Spearman's rho 0.717 -
p-value 0.037 -
Visually there is a strong correlation between the proportion of upper-intermediate and advanced students at the end and at the start of the English course. This trend persisted from 2014 to 2019, but then the results deteriorated despite the rising trend for the placement test.
In order to check the hypothesis that the correlation is strong, we calculated Spearman's coefficient for nonparametric data. Table 1 shows that there is a strong correlation for 2014-2019 (p = 0,893; p-value 0,012), but the last two years disrupted the linear trend and resulted in a lower coefficient (p = 0,717; p-value 0,037).
The conclusion here is that in the current situation, with existing students' intake and administrative procedures, the realistic KPI for the final IELTS exam is just 50%. The first explanation might be derived from looking into a 'grey zone', namely the population of students who simply skipped the exam. For traditional paper IELTS, which the department organized in 2014-2019, sitting an exam meant coming to exam
premises at the appointed time. In COVID-19 years the exam performed on Canvas LMS required even much less effort as it could be done from any place with the Internet access.
However, Table 2 demonstrates that not all students managed to take the exam, with one-fifth on average skipping it. These absentees, who amounted in some years to a quarter or more (Table 2), is a reason for lower than theoretically possible KPI. However, if we compare results by year, there is no statistically significant correlation between IELTS score and absentee percentage.
Trajectories for each starting level
IELTS as a standardized objective tool for assessing individual achievements opens many doors for researchers. For instance, we were able to trace each student from the placement test to his or her final IELTS and draw trajectories for each CEFR entrance level. Two sets of trajectories for a year with low final IELTS scores (2015) and the most successful year (2019) are shown in Fig. 2 and Fig. 3.
Table 2
Correlation between the number of students who did not appear in the exam and the final IELTS results
Year 2014 2015 2016 2017 2018 2019 2020 2021
Absentees,% 112.7 15.0 9.5 16.0 15.8 21.1 16.4 25.6
IELTS B2/C1,% 28.0 35.0 34.0 43.7 45.8 49.2 45.0 38.0
Spearman's rho 0.433
p-value 0.250
— -A1 - A2 •••• B1 = B2/C1
l\
/ \
/ \
• / • • • • 1---v \
ABS 2.0 3.0 4.0 5.0 5.5 6.0 6.5 7.0 7.5 8.0
IELTS Bands in 2015
Fig. 2. Trajectories for students who started the program in 2012 at A1, A2, B1 or B2,
and completed in 2015
45
40
35
<u 30
tn 25
5 20
J3
£ 15
10
-—A1 — A2 •••• B1 «=»B2/C1 t'\
"X"
i ' I \ \
\ \ ** N • 9V \\
V- • • Arm \\ ' "V S '• \
•Л • •
•Хч • \
Nfc--...
ABS 2.0 3.0 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0
IELTS Bands in 2019
Fig. 3. Trajectories for students who started the program in 2016 at A1, A2, B1 or B2,
and completed in 2019
In 2015 low level students basically failed to achieve B2, or Band 5,5 IELTS, and the peaks were at Band 4,0 for both A1 and A2. The difference between the two levels was that some students who started at A2 were able to achieve B2, but A1 starters generally finished with elementary level bands. Those who already had B1+ or B2 at the start typically achieved C1, and the starters with B1 demonstrated the whole array of final scores, but with a peak at Band 6.0.
In the 2019 graph the peaks for all levels except A1 shift towards higher bands. The most perspective trajectory seems to be A2 as it is feasible for the majority to achieve Band 5,5 or even higher. Unfortunately, 2019 saw a large percentage of absentees, and that fact pulled down the results for all trajectories.
Overall, students who started with B1 and B2 could all achieve the target level, provided they did not miss the whole exam or a speaking part. The area of concern is beginners, who struggle to move beyond the intermediate level. The most promising cohort is A2 starters, who demonstrated the potential for achieving the target level and beyond.
Despite the fact that IELTS results fluctuate for each trajectory, the mean values have a rising trend (Table 3). According to t-test, the differences between trajectories are significant with a high probability value of p = 0,008 (Table 4).
Table 3
B2/C1 achievers (%) in each trajectory
Trajectory Mean SD
A1 11.59 6.18
A2 37.21 13.51
B1 66.65 12.39
B2 81.27 14.07
Note. N = 8 years.
Table 4
Paired samples t-test
Trajectory 1 Trajectory 2 W p
A1 A2 0.000 0.008
A2 B1 0.000 0.008
B1 B2 0.000 0.008
Insight into the program efficiency in a COVID year
The standard external international exam IELTS is a unique objective instrument for assessing the program KPI. According to it, the program efficiency in 2020 and 2021 declined as
compared to the previous years (45% and 38% B2/C1, respectively). Although there might be various reasons for that, the main change in COVID period was online format of studies. 2021 graduates spent two out of four years in front of their computers, without live interaction, and they took IELTS in an online format too. The diagram in Fig. 1 is a vivid proof that face-to-face classes are more efficient.
In order to understand what happened in 2021, especially in view of high starting results in 2018 (51% B1/B2), we conducted two surveys: with high achievers, who took a certified exam, and underachievers, who came to the retake in order to be attested.
Seventy-two students (25%) from the first group completed a questionnaire specifying the reasons for taking IELTS. The typical responses to the question "Do you think you can use the certificate soon?" were as follows:
• In the current situation, no. It's hard to get a visa in any desirable country now. Maybe few months later it will change. Although that's a good proof of my English level to include in my resume.
• It is difficult to answer, but the certificate will not be superfluous.
• I suppose yes. I have plans to work in a country where it is quoted.
• Unfortunately, in the current situation I doubt it, but the experience was interesting anyway.
• I very much hope so.
The quotes above imply that IELTS certificates had rather a potential than practical value for their holders, and nobody was going to apply them just after graduation.
However, 281 students took and passed the exam successfully, and it is unlikely that they were motivated only by marks in their diploma. Fig. 4 represents responses to the question, "Why did you take IELTS?"
Although 'getting a certificate' was the most popular answer, this response does not look like a strong incentive, but more like 'just in case'. Surprisingly, only two students reported 'mark' as the only reason, while others outlined factors commonly associated with the intrinsic motivation:
• I took IELTS in order to prove myself that I studied English for four years not in vain.
• I was interested to check my real knowledge and know my real level of proficiency.
• For the sake of experience.
Fig. 4. Incentives to take IELTS in 2021
Fig. 5. The reasons for not attending the exam
In the second group, hereafter called 'unde-rachievers', 155 students failed to attend the whole exam or an oral part. The department ceased the opportunity to question them. Before getting access to IELTS retake materials on Canvas electronic platform, each student had to complete a short compulsory questionnaire on the same platform. We have collected and analyzed 69 answers, which are grouped into six categories in Fig 5.
Low English proficiency is cited as a reason by 26% of respondents:
• The reason is my individual poor preparation during the course, lack of knowledge.
• I blame lack of self-preparation and skipping many English classes, lack of time for preparing.
• I realized that I do not have enough points for a satisfactory mark anyway. Now I understand that it was worth trying to pass IELTS.
Among other reasons a popular one was work: 'I have to work on Saturdays and thus did not have an opportunity to come', 'The time for the exam was inconvenient because of my work'.
Some respondents were quite critical of themselves: 'I have a bad habit of doing everything at the last moment. I have learned a lesson thanks to IYAKTdepartment'.
However, 'lack of knowledge' did not prevent students from feeling confident about an online retake. Thus, 25% assessed their chances as 100%; 59% as 80%, 30% as 50%, and nobody thought that their chances were lower than 50%. Such optimism can be caused by different opportunities for cheating while doing a test online.
The findings presented in this section indicate that 'unsuccessful' students did not have the intrinsic motivation to succeed in IELTS, and this fact can also be the cause of poor results in 2021.
Conclusion and discussion. The article describes unconventional experience in large-scale assessing the English program efficiency. The measuring tool was a standardized external exam IELTS, the participants were all bachelor degree students, and the duration of the observations was eight consecutive years. Conditions for admission, as in any other technical university in Russia, did not involve English proficiency, and applicants ranged from A1 to B2.
The main result of this study is that the target set by MISiS at IELTS Band 5,5 was achieved only by half of graduates at best, and higher proportion of upper-intermediate levels seems unrealistic under the present conditions. Much research on a link between language proficiency and academic success claim that a minimum prerequisite for studying in EMI is Band 6,0 [5, 9, 16, 18]. Consequently, under the existing state of things only 50% of graduates have the potential ability to participate in academic program in EMI. However, even a 50% of B2/C1 graduates can be considered as success, given the abundance of low levels upon admission.
The proportion of B2/C1 levels increased constantly from 28% in 2014 to 49,2% in 2019, in accordance with the rise in the entrance proficiency. There is a strong correlation between IELTS bands and placement test results (p = 0,893) in pre-COVID years. There should be other factors contributing to this relative success, for instance, better teaching, as IYAKT department conducted a series of training with external experts on IELTS teaching and assessment. The teachers from the department also regularly participate in in-house sessions, do team teaching and peer observations to improve their understanding of IELTS, but the effect of teaching is not in the scope of this paper.
In the last two years IELTS results went down, and this trend can be a proof that the dis-
tant format affects the studies and motivation in a negative way.
The trajectories drawn for every level CEFR support the fundamental role of secondary school language education in progressing further at university. Thus, students who arrive with A1, rarely achieve B2 (only 11,6 % on average), and almost never C1; students with A2 with proper teaching and motivation were able to achieve B2 (54%) and even C1 in best years (3%). This category of learners should be the focus of close attention in the future. Students who arrived with B1 or B2 generally did not struggle with IELTS. The defining role of school versus pre-sessional university courses was also reported by Thorpe et al [18].
Surprisingly, students who did well at school and entered with B2 failed to demonstrate 100% of B2/C1, though many did achieve C1 (47% in 2019). One reason for that might be that IELTS format is different from the State Exam in English [2, 12]. IELTS tests academic English, and as such requires the ability to read academic texts, listen to subject lecturers and, what is more, writing academic texts in a limited time-frame. Students who skip classes and home assignments generally demonstrate results lower than their actual score in General English.
Speaking about absenteeism, this is a significant issue at a technical university. Although this research indicates that the proportion of students who did not show up for the exam did not correlate directly with total IELTS results for each year, 15 or even 25% of absentees definitely worsen statistics. This category remains a 'grey zone' for the department due to practical impossibility to trace them accurately.
Our university is not a closed system, where we assess exactly the same population as we started to teach. Some students are transferred from other institutions in the middle of the course, others are expelled or take an academic leave on the way, some continue studying after expulsion. These fluctuations, treated by the university as necessary evil, nonetheless affect the program KPI.
Finally, an important role in a non-English speaking university with limited opportunities for academic mobility is played by learners' motivation to study English profoundly. Unlike the situation in universities with EMI environment [9], in MISiS English is just a subject in a curricular rather than a vital need.
Surveys revealed that while good students
find intrinsic motivational factors to take IELTS, for example, to get an objective measure of their achievements, or to have a nice line in a resume; underachievers do not even appear on the exams, giving various pretexts. IYAKT department constantly makes efforts to increase the intrinsic motivation by making an English class an attractive place to visit with communicative activities, media resources and all-department events. These drivers work to a certain extent, but they are not a universal solution.
A considerable decline in the proportion of B2/C1 graduates in COVID years may indicate
lower efficiency of online studies as compared to the face-to-face format. Students' answers to questionnaires also demonstrated that the overall motivation to study English decreased in these years.
The article gives a snapshot of the current situation of applying IELTS on a large scale at a technical university, and does not offer quick solutions on how to achieve higher than 50% B2/C1 results. However, reflection on this experience may provide teachers and course designers with new ideas on revision of English language programs.
References
1. Almetov M., Zhorabekova A., Sagdullaev I., Abilhairova Z., Tulenova K. Engineering Education: Problems of Modernization in the Context of a Competence Approach. International Journal of Engineering Pedagogy, 2020, vol. 10, no. 6, pp. 7-20. Available at: https://online-journals.org/index.php/ i-jep/article/view/14043 (accessed 10.03.2023).
2. Baideldinova A. [The Impact of Russian National Exam in English while Preparing to Take IELTS]. The World of Science, Culture and Education, 2022, vol. 1, no. 92, pp. 176-177. (in Russ.) Available at: https://cyberleninka.ru/article/n/vliyanie-podgotovki-k-ege-pri-sdache-mezhdunarodnyh-sertifitsirovannyh-ekzamenov-na-primere-ielts (accessed 10.03.2023).
3. Cambridge English Support Site. Guided Learning Hours. Available at: https://support. cambridgeenglish.org/hc/ru/articles/202838506 (accessed 17.03.2023).
4. Chemsi G., Sadiq M., Radid M., Talbi M. Study of the Self-determined Motivation among Students in the Context of Online Pedagogical Activities. International Journal of Emerging Technologies in Learning, 2020, vol. 15, no. 5, pp. 17-28. Available at: https://bpspsychub.onlinelibrary.wiley.com/ doi/10.1111/bjep.12135 (accessed 17.03.2023).
5. Cho Y., Bridgeman B. Relationship of TOEFL iBT Scores to Academic Performance: Some Evidence from American Universities. Language Testing, 2012, vol. 29, iss. 3, pp. 421-442. DOI: 10.1177/0265532211430368
6. Dearden J. English as a Medium of Instruction: A Growing Global Phenomenon. British Council,
2014. Available at: https://www.britishcouncil.org/sites/default/files/e484_emi_cover_option_3_final_
web.pdf (accessed 17.03.2023).
7. Elder C. Language Proficiency as a Predictor of Performance in Teacher Education. Melbourne Papers in Language Testing, 1993, vol. 2, no. 1, pp. 68-85.
8. Evans S., Morrison B. Meeting the Challenges of English-medium Higher Education: The First-year Experience in Hong Kong. English for Specific Purposes, 2011, vol. 30, iss. 3, pp. 198-208. DOI: 10.1016/j.esp.2011.01.001
9. Feast V. The Impact of IELTS Scores on Performance at University. International Education Journal, 2002, no. 3, pp. 70-85. Available at: https://www.researchgate.net/publication/228424721 (accessed 15.03.2023).
10. Financial University. London Educational Project, 2019. Available at: http://www.fa.ru/_ layouts/15/AdvSearchResults.aspx?k=%D0%BB%D0%BE%D0%BD%D0%B4%D0%BE%D0%BD% D1%81%D0%BA%D0%B8%D0%B9%20.cessed (accessed 16.03.2023).
11. Higgins R, Hartley S, Skelton A. The Conscientious Consumer: Reconsidering the Role of Assessment Feedback in Student Learning. Studies in Higher Education, 2002, no. 27 (1), pp. 53-64. DOI: 10.1080/03075070120099368
12. IELTS Guide for Teachers, 2021. Available at: https://www.ielts.org/for-teachers/ielts-for-teachers (accessed 17.03.2023).
13. Kember D. Characterizing the Motivational Orientation of Students in Higher Education: A Naturalistic Study in three Hong Kong Universities .British Journal of Educational Psychology, 2008, no. 78, pp. 313-329. DOI 10.1348/000709907X220581
14. Petrichtcheva T. [Contemporary Issues in IELTS]. Sovremennye tendentsii v prepodavanii ino-strannykh yazykov v nelingvisticheskom vuze [Current Trends in Teaching Foreign Languages at a Non-linguistic University], 2014, no. 8, pp. 80-83 (in Russ).
15. Rossikhina O., Ermakova P., Aleshchenko O. Analysis of the English Language Needs of Students at the Russian Technological University. RUDN Journal of Psychology and Pedagogics, 2019, vol. 16, no. 1, pp. 88-100. DOI: 10.22363/2313-1683-2019-16-1-88-100
16. Schoepp K. Predictive Validity of the IELTS in an English as a Medium of Instruction Environment. Higher Education Quarterly, 2018, no. 72, pp. 271-285. DOI 10.1111/hequ.12163
17. Sherine A, Seshagiri A.V.S., Sastry M. Impact of WhatsApp Interaction on Improving L2 Speaking Skills. International Journal of Emerging Technologies in Learning, 2020, vol. 15, no. 3, pp. 250-259. DOI: 10.3991/ijet.v15i03.11534
18. Thorpe A., Snel M., Davey-Evans S., Talman R. Improving the Academic Performance of Non-native English-Speaking Students: the Contribution of Pre-sessional English Language Programmes. Higher Education Quarterly, 2017, vol. 71, no. 1, pp. 5-32. DOI 10.1111/hequ.12109
19. Vallerand R., Pelletier L., Blais M., Briere N., Senecal C., Vallieres E. The Academic Motivation Scale: A Measure of Intrinsic, Extrinsic, and Amotivation in Education. Educational and Psychological Measurement, 1992, vol. 52, iss. 4, pp. 1003-1017.
20. Zhang Y., Lin C-Y., Zhang D., Choi Y. Motivation, Strategy, and English as a Foreign Language Vocabulary Learning: A Structural Equation Modelling Study. British Journal of Educational Psychology, 2017, no. 87, pp. 57-74. Available at: https://bpspsychub.onlinelibrary.wiley.com/doi/ 10.1111/bjep.12135 (accessed 16.03.2023).
Information about the authors
Olga G. Rossikhina, Associate Professor of the Department of Modern Languages and Communication, Coordinator the Academic English Module, National University of Science and Technology MISiS, Moscow, Russia.
Nina V. Chernushkina, Senior Lecturer of the Department of Modern Languages and Communication, Exam Officer responsible for administering IELTS exam for all engineering undergraduates, National University of Science and Technology MISiS, Moscow, Russia.
Polina V. Ermakova, Associate Professor of the Department of Modern Languages and Communication, the Academic Coordinator of the Bachelor Degree English Program for Engineers, National University of Science and Technology MISiS, Moscow, Russia.
Информация об авторах
Россихина Ольга Глебовна, кандидат биологических наук, доцент кафедры иностранных языков и коммуникативных технологий, координатор модуля «Академический английский язык», Национальный исследовательский технологический университет МИСиС, Москва, Россия.
Чернушкина Нина Васильевна, старший преподаватель кафедры иностранных языков и коммуникативных технологий, координатор пересдач и экзаменационного модуля, Национальный исследовательский технологический университет МИСиС, Москва, Россия.
Ермакова Полина Викторовна, кандидат филологических наук, доцент кафедры иностранных языков и коммуникативных технологий, руководитель направления «Английский для инженеров-бакалавров», Национальный исследовательский технологический университет МИСиС, Москва, Россия.
Contribution of the authors:
Rossikhina O.G. - research concept; data analysis, methodology development; text writing.
Chernushkina N.V. - data collection over an 8-year period, data analysis, research concept, text writing.
Ermakova P.V. - general administration of the exam over an 8-year period, scientific supervision, text writing.
The authors declare no conflicts of interests.
Вклад авторов:
Россихина О.Г. - концепция исследования, анализ данных, развитие методологии, написание исходного текста.
Чернушкина Н.В. - сбор данных на протяжении 8 лет исследования, анализ данных, концепция исследования, написание текста.
Ермакова П.В. - научное руководство, общее администрирование экзамена на протяжении 8 лет исследования, написание текста.
Авторы заявляют об отсутствии конфликта интересов.
The article was submitted 12.04.2023
Статья поступила в редакцию 12.04.2023
