CC BY-NC-ND
55
Russian Schools during the COVID-19 Pandemic: Impact of the First Two Waves on the Quality of Education
T.Yu. Chaban, R.S. Rameeva, I.S. Denisov, Yu.D. Kersha, R.S. Zvyagintsev
The article was submitted to the Editorial Board in October 2021
Tatyana Chaban, head, Department for Monitoring the Quality of Education, "Center for Education Quality Assessment" Krasnoyarsk Territorial State Government-Owned Specialized Establishment. Email: [email protected]. Address: 9 Vysotnaya St., Krasnoyarsk, 660062, Russia
Roza Rameeva, head, Department of Information and Analysis, "Center for Education Quality Assessment" Krasnoyarsk Territorial State Government-Owned Specialized Establishment. Email: [email protected]. Address: 9 Vysotnaya St., Krasnoyarsk, 660062, Russia
Ilya Denisov, research assistant, Center for Psychometrics and Measurements in Education, Institute of Education, National Research University Higher School of Economics. Email: [email protected]. Address: 16 Potapovsky Lane, bldg. 10, Moscow, 101000, Russia
Yulia Kersha, junior research fellow, Pinsky Center of General and Extracurricular Education, Institute of Education, National Research University Higher School of Economics. Email: [email protected]. Address: 16 Potapovsky Lane, bldg. 10, Moscow, 101000, Russia
Roman Zvyagintsev, junior research fellow, Pinsky Center of General and Extracurricular Education, Institute of Education, National Research University Higher School of Economics. Email: [email protected]. Address: 16 Potapovsky Lane, bldg. 10, Moscow, 101000, Russia (Corresponding author)
Abstract International studies about the impact of the COVID-19 pandemic on the quality of general education have yielded contradictory results: educational outcomes have fallen markedly in some countries while remaining more or less constant in others. At the same time, over half of published studies attest to the growth of educational inequality during the pandemic. The present research assesses the impact of the pandemic using the data of a regional monitoring survey of all schoolchildren in grades 4, 6, and 8 in the Krasnoyarsk Region in 2019 and 2021. Tests of reading literacy in grades 4 and 6 as well as of science literacy in grade 8 have shown satisfactory psychometric quality. Multilevel regression analysis was used to show that the level of functional literacy of the "pandemic" cohort of schoolchildren, controlled for contextual characteristics, was significantly lower for students in all grades except grade 4. The biggest loss was found in scientific literacy. No correlation was found between the pandemic effect size and the socioeconomic composition of the class (the gap between children with different SES remains at the same level as it was before the pandemic). The teachers' opportunity to conduct online classes did not serve to improve the educational outcomes of schoolchildren during the pandemic.
Keywords school learning, educational inequality, educational outcomes, reading literacy, science literacy, COVID-19, socioeconomic composition of the class, regional monitoring studies.
For citing Chaban T.Yu., Rameeva R.S., Denisov I.S., Kersha Yu.D., Zvyagintsev R.S. (2022) Rossiyskaya shkola v period pandemii COVID-19: effekty pervykh dvukh voln i kachestvo obrazovaniya [Russian Schools during the COVID-19 Pandemic: Impact of the First Two Waves on the Quality of Education]. Voprosy obrazovaniya / Educational Studies Moscow, no 1, p. 160-188. https://doi.org/10.17323/1814-9545-2022-1-160-188
The COVID-19 pandemic has taken a heavy toll on the economy, as well as health care and education systems. The scientific, as well as social and political literature, has been widely discussing the potential short- and long-term effects of the pandemic on schools, schoolchildren, their families, and there is active research ongoing in this area.1 Some authors have already conducted systematic reviews of the results of these studies [Hammerstein et al., 2021]. The current changes and school closures have been shown to increase educational inequality.2 A wide range of compensatory practices are being discussed.3 The forced transition to distance learning4 in 2020 raised many questions about the education system, with no answers so far. Education policy makers voiced opposing views about whether schools would be able to move to distance education, or whether it might be better to close them altogether.5 Most countries transitioned to distance learning for varying periods, after which children returned to learning in person.6
The most vulnerable groups of students: disadvantaged children, underachievers, learners with special educational needs, and so forth caused the most concern. There were no easy solutions, and some even proposed to let the most vulnerable groups of children attend schools so that they had somewhere to go.7 Some countries did not close schools at the beginning of the pandemic. However, at the peak of the first wave in mid-April, UNESCO re-
1 https://www.rand.org/multimedia/audio/2020/03/31/the-impact-of-covid-19-on-the-us-education-system.html
2 Storey N., Zhang, Q. (2021) A Meta-Analysis of COVID Learning Loss. Preprint.
3 https://www.rand.org/blog/2020/04/is-it-time-to-rethink-the-separation-be-tween-the-high.html
4 In this paper, by distance learning we mean online learning, as it is most relevant at the moment; we do not consider other distance learning practices here.
5 https://www.k12.wa.us/about-ospi/press-releases/novel-coronavirus-covid-19-guidance-resources
6 https://www.rand.org/blog/2020/04/coronavirus-will-require-changes-in-schools-when-they.html
7 https://www.theguardian.com/education/2020/apr/14/the-schools-open-during-lockdown-for-some-kids-its-the-only-safe-place
ported that 192 countries had stopped providing in-person instruction in all their schools and universities, which affected 90% of the world's students, or almost 1.6 billion children and young people. In most of these cases, the educational process continued in a distance mode [Kocape^MM m gp., 2020]. Force majeure interruption of in-person schooling of this scale was an extraordinary situation in education, immediately becoming the focus of research from different perspectives [DeMatthews et al., 2020; Gao, Zhang, 2020; Ghosh et al., 2020].
The pandemic has given researchers a rare chance to study the impact of crises on the education system. The impact of force majeure events, disasters, and terrorist attacks on education quality and student well-being had been studied before. In particular, there had been some research on school closures due to snowstorms [Goodman, 2014; Marcotte, Hemelt, 2008], floods [Tham-tanajit, 2020], earthquakes [Sapkota, Neupane, 2021], and even terrorist attacks [Gershenson, Tekin, 2018]. However, all of these studies examine local events that have not even affected the education system of a country's region, let alone a country or a group of countries. The COVID-19 pandemic has created entirely new research opportunities, mostly in countries that have established systems to monitor student academic achievement at different stages of general education.
This paper aims to estimate the loss in education quality due to the pandemic for Russian primary and secondary school students. We seek to answer the following research questions:
• How have the educational outcomes of the 4th, 6th, and 8th graders in 2021 changed compared to 2019 when controlling for contextual characteristics?
• How are these changes related to the socioeconomic composition of the class?
• Are the learning outcomes different for 4th, 6th, and 8th graders in 2021 who were taught by teachers with different levels of technological readiness to teach online?
The current situation with the pandemic and school closures is a good example of a natural experiment. Most schoolchildren found themselves in an educational environment that was completely new to them. No one could have anticipated the pandemic well in advance and prepared for it. Consequently, the distribution of students into experimental and control groups — those who were exposed to the pandemic in 2021 and their predecessors — was exogenous, that is, without any will of the students involved [Murnane, Willett, 2010]. It is natural to assume that 4th, 6th, and 8th graders in 2019 differ from their peers in 2021 primarily because of pandem-
ic-related circumstances. The observed difference in educational outcomes can therefore be regarded as an impact of the pandemic rather than other factors, which could also have played a role, but their significance is incomparable to the effect of the pandemic.
1. Predicting and estimating the learning loss due to the pandemic
All researchers of crises in the education system share the same opinion: students of schools closed under force majeure circumstances lose in learning. Obviously, the loss may not only be due to school closures per se but also due to general stress and disruption of routines [Goodman, 2014]. At the beginning of the pandemic, some researchers expected schoolchildren to lose 30-50% of their previous year's learning by the fall [Kuhfeld, Tarasawa, 2020]. According to estimates based on PISA 2018 data, losses in reading literacy might range from 9 to 16 points depending on the availability and quality of distance education, and the proportion of functionally illiterate students would increase by 8% if schools stayed closed for four months [Kelmendi, Gresham, Iqbal, 2020]. The proportion of "learning poor" — children at age of ten who still could not read — was expected to rise to 63% [Azevedo et al., 2020]. Economic losses for the generation of students affected by the pandemic were tentatively estimated at 3% of their lifetime earnings [Ha-nushek, Woessmann, 2020], and losses in low- and middle-income countries would be higher than in high-income countries [Psacha-ropoulos et al., 2021].
The authors emphasized that any predictions should consider several factors that were key to determining how much the pandemic would set back students' learning: the duration of school closures, the quality of distance education, the level of family preparedness and resources, and the availability of high-speed Internet for teachers and families [Kocape^MM m gp., 2020]. One more factor to keep in mind is the subjective well-being of students and their families, which was predicted to decline [Ghosh et al., 2020] and has actually declined, causing problems in behavior and interaction within families [Patrick et al., 2020].
The COVID-19 pandemic has significantly exacerbated educational inequalities. Any disasters and social upheavals have a greater impact on the most vulnerable and poorest segments of the population [Fothergill, Peek, 2004]. Studies of education systems show that children from families with low socioeconomic status [Sirin, 2005] and children with special educational needs [Cooc, 2019] experience the greatest loss in critical situations. Moreover, children from affluent families get an opportunity to progress better due to receiving extra time for tutoring, having access to paid online courses, and their generally better digital readiness [Stern, Adams, Elsasser, 2009]. Thus, the real effect of the pandemic is not a
general decline in performance, but rather increased stratification and larger opportunity gaps between different groups of students.
There are several ways to estimate the actual loss in the quality of general education due to the pandemic: testing the same children with equated tests just before school closures and immediately after they return to school; comparing the learning levels of children from past cohorts not affected by the pandemic and those of children in the same schooling stage who had to study during the pandemic; comparing the rate of learning progress in previous years with that observed now; finally, comparing the dynamics of educational outcomes of children who experienced distance learning and of those who did not. The academic community has already made some estimates of the actual loss in the quality of education due to the pandemic. These estimates are available for multiple countries, including:
• Belgium [Maldonado, De Witte, 2021];
• Netherlands [Engzell, Frey, Verhagen, 2021];
• Switzerland [Tomasik, Helbling, Moser, 2021];
• Great Britain;8
• USA;9
• France;10
• several other education systems.11
The general conclusion is that insignificant loss (about 1-2 months of learning) occurred even in systems where distance learning was well organized and schools were closed for a very short period. At the same time, the studies conducted in the USA and France, for example, have shown that on average there has been almost no learning loss due to the pandemic. Yet even with a minimal overall loss, the detrimental effect of the pandemic on children from families with low socioeconomic status is about 1.5 times greater than for children from wealthy families, meaning that the pandemic increases inequality. In contrast to the aforementioned countries, which are among the most successful both economically and in terms of education, India, for instance, has faced a substantially greater loss.12
8 https://assets.publishing.service.gov.uk/government/uploads/system/up-loads/attachment data/file/962330/Learning Loss Report 1A - FINAL.pdf
9 https://www.nwea.org/content/uploads/2020/11/Collaborative-brief-Learn-ing-during-COVID-19.NOV2020.pdf
10 https://www.education.gouv.fr/evaluations-de-debut-de-sixieme-2020-pre-miers-resultats-307125
11 https://educationendowmentfoundation.org.uk/eef-support-for-schools/ covid-19-resources/best-evidence-on-impact-of-school-closures-on-the-at-tainment-gap/
12 https://www.orfonline.org/research/regression-in-learning/7amp
The Russian education system has not yet undertaken research to assess the learning loss experienced by schoolchildren. In this paper, using data from the regional monitoring surveys of reading and science literacy conducted in the Krasnoyarsk Region among all 4th, 6th, and 8th graders, we investigate what educational loss Russian schools have faced as a result of the first and second waves of the COVID-19 pandemic.
2. Learning Conditions During the Pandemic: National and Regional Context
The current state of Russia's education system is such that in case of a pandemic every fourth pupil and every fifth school may have to interrupt the educational process, as the Internet speed will not be enough for proper distance learning in about 40% of Russian schools. Furthermore, every fifth teacher in Russia does not have the skills necessary to organize distance learning [Заир-Бек, Мерцалова, Анчиков, 2020]. Even during the second wave of COVID-19, in 12 constituent territories of the Russian Federation, the degree of school preparedness for distance learning was such that the experts of the Institute of Education of the National Research University Higher School of Economics classified these territories as a risk zone based on a combination of school preparedness for distance education and the level of viral threat.13
Families with high and low socioeconomic status differ from one another not only in overall technology readiness but also in whether parents and children have the skills needed to organize distance learning. Poor families are more likely than affluent families to report that they lack computer literacy and skills for organizing learning at home (31% against 10%); that their children are not ready for studying online (60% against 45%); that when studying in a distance mode their children have an insufficient study load and receive lower quality education (76% against 56%).14 For more details on both learning conditions and education policy during the pandemic at the national level, see [Мерцалова и др., 2021]. In this study, we focus on the regional specifics.
In the schools of the Krasnoyarsk Region, in-person learning was interrupted on March 17, 2020. On that day, following the governor's decree, all schools in the region went on an early spring break, after which distance learning began, continuing until the end of the school year. The new school year started with face-to-face learning, but on October 28, 2020, given the complicated epidemiological situation, the Ministry of Education of the Krasnoyarsk Region recommended schools in Krasnoyarsk and ten other towns — Achinsk, Kansk, Yeniseysk, Divnogorsk, Lesosibirsk, Minusinsk,
https://www.hse.ru/news/expertise/416606518. html
http://pltf.ru/2020/04/24/vypusk-8-detskoe-onlajn-obrazovanie/?fbclid=IwAR-30lhle9bnwKHkgtW-L5M9KqOcW13cbmm2dbM5cqMlomngiM7MS05C4r7Y
Sosnovoborsk, Sharypovo, Zheleznogorsk, and Zelenogorsk — to transfer students of grades 5-11 to distance learning right after the fall break. Primary school students continued to study in person.
Municipalities were entitled to decide whether to transfer schools to distance learning based on the COVID-19 rate in their administrative territories. A few weeks later, four more towns — Borodino, Bogotol, Nazarovo, and Norilsk — transferred their primary and secondary school students to distance education. On December 7, 2020, ninth- and eleventh-graders were back to their classes, as well as fifth graders in all towns, except Krasnoyarsk. Only on January 11, with the start of the second half of the year, schools were able to return to the traditional mode of instruction.
Thus, for about two months schoolchildren in grades 6-8 in almost all urban schools of the Krasnoyarsk Region studied remotely. Their peers in non-urban areas and five small towns of the region with a population of fewer than 25,000 people — Artemovsk, Dudinka, Zaozyorny, Uzhur, and Uyar — continued in-person learning. While the ratio of schools providing in-person and distance education in November-December 2020 was 70 to 30, the ratio of students in grades 6-8 was almost the opposite: 65% of them studied from home and only 35% in physical classrooms.
In both the first and second waves of the pandemic, equipping participants in online learning remained a challenge. Only a handful of schools were able to provide laptops, netbooks, or tablets to schoolchildren who did not have computer hardware at home. In
2020, not all teachers had access to computers from which to teach online classes. In the region stretching from the Sayan Mountains to the Arctic coast, settlements often had an Internet connection available only in the school building or had no Internet connection at all, and a mobile phone signal could be received only at a few locations on high ground. For this reason, in many rural schools, schoolchildren handed in their completed assignments to teachers in paper form.
Due to the majority of 6th-8th graders moving to distance learning the regional diagnostic tests (RDTs) in these cohorts which had been scheduled for late fall 2020 were postponed until early 2021. The RDT in reading literacy in 6th grades was held on January 26,
2021, just 15 days after 6th graders returned to school following a period of distance learning and vacation. The RDT in science literacy for 8th graders took place on February 16. Only the RDT in reading literacy in 4th grade was held as planned on March 17, just as before the pandemic.
By the beginning of the study, all sixth- and eighth-graders had spent more than half of the school year in a distance mode of learning (about 4.5 months in the spring and fall of 2020), and fourth-graders just over two months (in the spring of 2020). Further-
more, there were almost no schools where a class, or even the entire school, would not switch to extra periods of distance learning during quarantines, thereby increasing the interruption of in-per-son education.
3. Methodology
3.1. Characteristics of the Monitoring Survey
The diagnostic testing in reading literacy is conducted in all schools of the Krasnoyarsk Region annually in grades 4 and 6. Since 2018, the 8th graders have been taking an RDT in science literacy. Each regional diagnostic test includes two test versions. We have used the results of the RDT administered in 2018, 2019, and 2021. Since each of the RDT has two purposes — to assess individual student achievement and the overall situation in the regional education system — they are designed as a "dual-use" procedure. Municipal observers from other schools arrive during the test to monitor the process. Completed assignments are assessed by municipal or school commissions. The type of assessment is chosen by the municipal education authority.
All RDTs are modeled on the international PISA study15. In measuring mathematics and science literacy, the same groups of competencies are assessed as in PISA. The reading literacy tests are based on the reading literacy assessment model developed by the Institute for Strategy of Education Development of the Russian Academy of Education under the direction of G. S. Kovaleva and also take into account the PISA experience. The tests contain both dichotomously and polytomously scored items.
Not only do the diagnostic tests vary from grade to grade, but also from cohort to cohort. The 2019 tests for the grades studied in this paper differ from tests for the same grades in 2021. To assess the impact of the pandemic and to compare the results of the 2019 and 2021 RDT, we must ensure that the test results are comparable.
3.2. Ensuring In order to equate tests from multiple assessments [KapgaHOBa, Comparability of HeMMaH, 2003] , the method of common, or anchor, item equating the Test Results is usually used. Different diagnostic instruments include several common items, so that test scores can be calculated on a common scale. The number of common items in the two tests should be sufficient to consider the equating results reliable. Since the data available to us originally had no common items, we applied the equation by pseudo-common items.16
To obtain pseudo-common items, we selected items similar in topic, format, and scoring principle in the test versions from differ-
15 https://www.oecd.org/pisa
16 https://www.winsteps.com/winman/equating.htm
ent years. Then all test results were arranged in such a way that pseudo-common items could be used to "anchor" the tests from different years and place them on a single scoring scale. As a result, all parameters of test items and test takers were on the same scale. The test results from different years could then be compared to each other, albeit with some limitations, within the framework of item response theory [Нейман, Хлебников, 2000].
Thus, equating by pseudo-common items allows us to directly compare the performance of 4th, 6th, and 8th graders in 2019 and 2021 since their scores are now comparable. To increase the reliability of the procedure, we used all available test results — those from 2018, 2019, and 2021. For each cohort, the reference test version was the first version in one year out of three.
Among the models of the latent trait theory, the Rasch one-parameter logistic model [Wright, Stone, 1979] is most commonly used for equating. The study data were processed using specialized Winsteps software17 developed for psychometric analysis using the Rasch model. The pseudo-common-item equating was therefore also conducted using the Rasch model. The procedure included the following steps:
• difficulty estimation for all items in the reference group, including common ones (pseudo-common in this case), setting the average trait score to zero;
• difficulty estimation for all items of all other test versions in the three years, for example, for the 8th graders it is Version 2 of 2019 (Version 1 is the reference one) and Versions 1 and 2 of 2018 and 2021. For all of them, the average latent trait score is also set to zero;
• calculation of the difference in the difficulty of the pseudo-common items between the reference version and each other test version. After that, the arithmetic mean difference is calculated, which is the equating constant;
• latent trait estimation for each group that took a particular test version. The measurements are carried out on a test version's own scale;
• conversion of scores on the other test versions to scores on the reference test version by adding the equating constant. Since eventually each test version will be scored on the scale of the first 2019 version (in the case of 8th graders), all these scores can be considered comparable and can be actually compared with each other.
17 https://www.winsteps.com
After the input data are pre-processed, Winsteps autonomously runs all the following steps, which allows for simultaneous pseudo-equation of all six versions of the measurement instrument and their presentation in the framework of a partial credit model.18
3.3. Contextual To assess the impact of the pandemic, we used comparable test reData and Analysis sults of 4th, 6th, and 8th graders in 2019 (before the pandemic) and Strategy 2021 — a total of 165,740 schoolchildren from 1,047 schools in the Krasnoyarsk Region. The data have a three-level structure: individual student scores are grouped first by grade and then by school. The schools surveyed were the same in all years when the RDTs were conducted, while the composition of classes varied. In other words, the ability level of, for example, fourth graders in 2019 was compared with the ability level of new fourth graders in 2021.
The main dependent variable in the analysis is the standardized test results of a student, indicating his or her ability level. Other individual student characteristics considered included gender, test version, and whether the student belonged to the 2019 or 2021 cohort. At the grade level, we used a variable indicating the number of schoolchildren whose at least one parent had a university degree, to control for students' socioeconomic composition [Kepwa, 2020]. In each year, based on the distribution of this variable, three equally populated groups of classes — with low, middle, and high socioeconomic composition — were identified. Drawing on the information from teachers, at a specific stage of the analysis 4th and 8th grades were divided into classes where the teacher was able to give online lessons and those in which the teacher was not. In 8th grade, tested in science literacy, we considered this ability in three types of teachers — physics, biology, and chemistry. At the school level, the type of settlement where the school was located was controlled for. We distinguished groups of schools located in small towns and rural areas, in larger cities, and separately in Krasnoyarsk. The main groups to be compared were the 2019 and 2021 cohorts, but to control for students' previous performance, the average score of the respective class in the 2018 test was also calculated. Descriptive statistics for the variables used are presented in Appendix.19
Since the tests in grades 4, 6, and 8 differ from each other, calculations for different cohorts were performed separately. To compare student test scores in 2019 and 2021, three-level regression models were constructed that accounted for contextual characteristics at the student, grade, and school levels [Hox, 2010]. Since one of the research goals was to see whether the change in student ability lev-
18 https://www.rasch.org/rmt/rmt143k.htm
19 Access the anonymized data, calculation procedures, and appendix here: doi: 10.17632/k7gv2c43bd.1
els in 2021 compared to 2019 differed by socioeconomic class composition, the analysis was conducted using a random slope model with cross-level interaction [Hox, 2010]. In addition, sub-sample analysis was conducted for groups of classes with different technological readiness, as well as for students with the highest ability level (25% of the highest scores in each wave) and the lowest (25% of the lowest scores). Models were selected and compared using goodness-of-fit measures (AIC, BIC, Log-Likelihood) and a likelihood ratio test. Models were built step-by-step, starting with an intercept-only model with fixed coefficients (Appendix). There were five models for each grade level — intercept-only model (1); one with added covariates and fixed coefficients (2); one with varying coefficients at grade level (3); one with varying coefficients at school level (4); one with cross-level interaction (5). For the subsample analysis, model (4) was used without the cross-level interaction variable. The most comprehensive model including cross-level interaction was as follows:
abili% = Y000 + Yiik INDijk + Yjk CLj + Yk SCHk + ^ INDijk x
x CLSjk + u0jk INDjk + u0jk + V
where ability is a dependent variable for an individual ability level; i — first-level index (students), j — second-level index (grades), k — third-level index (schools); y000 — model intercept; IND, CLS, and SCH — vectors of covariates at individual, grade and school levels respectively; u — dispersion of residuals; e — model residuals.
The analysis was performed using the software package for data analytics Stata/SE 16. Since the sample size is quite large, even the smallest effects can be found statistically significant in case of high accuracy of parameter estimation. Therefore, instead of the p-value, we provide the values and confidence intervals of the average marginal effects of the study year for the examined parameters [Lin, Lucas, Shmueli, 2013], expressed in standard deviations of the difference between the ability levels in 2021 and 2019. In addition, the bootstrapping procedure is also used to estimate parameters on the large sample [Yu, 2002]. To calculate each model, a sub-sample of 1000 observations was randomly drawn 100 times. The effect size was also calculated for groups of classes distinguished by their socioeconomic composition (low, middle, or high) and the teacher's technical ability to conduct online lessons. The effects of learning during the pandemic were also calculated for groups of schoolchildren with the highest and the lowest academic performance.
3.4. Limitations Our study has several limitations that must be taken into account when reporting and discussing the results.
• We estimate the impact of the pandemic only in one constituent territory of the Russian Federation, as only there we were able to find the data necessary to answer the main research question — what the learning loss of Russian schoolchildren is due to the pandemic. This limitation could have been avoided if Russia had conducted regular nationwide monitoring of students' functional literacy levels using equated measurement instruments.
• The data obtained in the Krasnoyarsk Region's monitoring study cannot be considered completely reliable. Students' test scores may have been affected by the traditions and attitudes of individual schools and municipal education systems. An external observer, who is sent to monitor the RDT from another school or another municipal institution, is not always able to notice when schoolchildren are being helped or identify bias in grading. In many schools, internal incentive payments depend on the results of external evaluations. Teachers may be trapped by the reputation of their gymnasium or lyceum in which brilliant subject results are the norm. In small towns and settlements, personal motives may come to the fore: "People will say that I am the worst teacher". At the same time, there are school and municipal staff for whom "unfair play" is unacceptable at all, or municipal regulations set a very strict procedure for the RDT when students' works are taken to the municipal commission immediately after they are checked.
• The tests differ from year to year, even if only slightly. The regional authorities have never set out to compare test results of the same age cohorts from different years.
• The method of pseudo-common-item equating has some limitations compared to more traditional methods of ensuring comparability of test results. They include, among other things, the absence of identical items common for different test versions that could be used as anchors in equating, differences in scoring scales, and incomplete correspondence of item assessment procedures.
• In our multilevel regression models, we are forced to use grade-and school-level variables as control ones. For example, we have no information about the socioeconomic status of individual students, but we do have information about the socioeconomic composition of the class. Variables aggregated at the class level are significant predictors of academic achievement, but without individual characteristics, we still lose much of the information.
• There is no regional data for 2020 when the regional monitoring study did not take place. Consequently, we are comparing the results of children in 2019, when there was no pandemic, with the results in 2021 when the entire education system had
already experienced it for about a year. Moreover, children of different ages had studied remotely for different periods, which must also be taken into account when interpreting the results.
• The tests measured science literacy in grades 8 and reading literacy in grades 4 and 6. For this reason, we cannot directly compare effect sizes for different cohorts and can only capture particular losses for individual cohorts. Only the 4th and 6th grades can be validly compared with each other.
4. Results
4.1. Psychometric Analysis of Instrument Quality
In order to confirm the validity of using the monitoring results in the study of pandemic effects, we assessed the quality of the measurement instruments used. The results of this assessment show satisfactory psychometric properties of the scales used (reliability of individual tests is from 0.75 to 0.82), as well as a sufficient number of items in different test versions from different years that can be selected as pseudo-common. The tests have a quite high level of reliability and can be considered essentially unidimensional. Detailed calculations concerning psychometric properties of individual tests, selection of pseudo-common items, and pseudo-equation of test results are openly available in the Appendix.20
4.2. Effects of the COVID-19 Pandemic on Students' Functional Literacy
The results of the regression analysis show that 4th, 6th, and 8th graders in 2021 generally performed lower than their peers in 2019. The calculation results are presented in Figure 1 as the size of the learning effect in 2021 compared to that in 2019. The negative effect size indicates that the results in 2021 are lower than those in 2019. The largest decrease in scores after the pandemic can be observed in 8th graders (by an average of 0.87 SD of the ability level). In 6th graders, the decrease is not as strong (by 0.24 SD on average), and in 4th graders, there is no decrease at all. We have found no statistically significant differences between the impact of the pandemic on groups within the same cohort, differing in terms of the socioeconomic composition of the class.
When comparing the learning loss experienced by schoolchildren with different levels of achievement, mixed results were obtained (Figure 2). As in the previous case, 4th graders suffered no loss. For the sixth graders with the highest reading test scores in 2019 and 2021, the loss was more than twice as high (0.38 SD) as for those with the lowest scores (0.14 sd). In the science test for eighth graders, in contrast, test results of high-scoring students declined not as severely (0.72 SD) as those of children with the lowest scores (1.04 sd).
doi: 10.17632/k7gv2c43bd.1
20
Figure 1. The Average Marginal Effects of Learning in 2021 Compared to 2019 for 4th, 6th, and 8th Graders From Classes with Different Socioeconomic Composition
Grade 4 (reading)
--- ---- —4-------4— ----£-----
Grade 6 (reading)
--- 5 —'}------- §
Grade 8 (science)
--- 5 § § §
All Low SES Middle SES High SES
Note. Marginal effects are calculated using a multilevel regression model with cross-level interaction. The 95% confidence interval was used for the coefficient. Effect size coefficients are presented as standard deviations of test results in 2021 compared to 2019 when accounted for covariates in the model.
Figure 2. The Average Marginal Effects of Learning in 2021 Compared to 2019 for 4th, 6th, and 8th Graders With Different Achievement Levels
0.5 0.0 -0.5 -1.0
0.5 —f
CD
3 0.0 2 -0.5
LU
-1.0
0.5 0.0 -0.5-1.0
Grade 4 (readin
—5—
Grade 6 (readin
Grade 8 (science)
Low achievement (25%)
High achievement (25%)
Note. Marginal effects are calculated using a multilevel regression model with cross-level interaction. The 95% confidence interval was used for the coefficient. Effect size coefficients are presented as standard deviations of test results in 2021 compared to 2019 when accounted for covariates in the model.
4.3. Comparison of Test Results in Classes With Different Levels of Technological Readiness
The data from the survey of class teachers in grades 4 and 8 give an idea of how well teachers and students were provided with technical means for online learning. The majority of teachers in the Krasnoyarsk Region had the technical ability to conduct online lessons: almost 70% of 4th-grade teachers and 72% to 76% of 8th-grade science teachers (Figure 3).
Figure 3. The Proportion of Teachers Who Had the Technical Ability to Teach Online Lessons When the Class Switched to Distance Learning in the Spring of 2020
69.46
75.97
71.37
76.17
4th-grade 8th-grade 8th-grade 8th-grade
teachers physics teachers chemistry teachers biology teachers
Figure 4. Average Marginal Effects of Learning in 2021 for Groups of Classes With Different Levels of Technological Readiness
I Grade 4 (reading)
1---------1- ----------- -
1 Grade 8 (science)
Ï 3 E ----------- 5
Without ICT in 2021 With ICT in 2021
Note. Marginal effects are calculated using a multilevel regression model with cross-level interaction. The 95% confidence interval was used for the coefficient. Effect size coefficients are presented as standard deviations of test results in 2021 compared to 2019 when accounted for cova-riates in the model for different subsamples of 2021 students.
The relatively small decline in student test scores in 2021 relative to those of their peers in 2019 appeared to be unrelated to the teacher's technical ability to conduct online classes in 2021 (Figure 4). In classes that were not technically ready for distance learning, the impact of the pandemic did not differ from that in classes
where the teacher had the technical ability to deliver online lessons. The functional literacy of fourth graders in 2021 did not differ from that of their peers in 2019, regardless of the teacher's level of technological readiness. Among 8th graders, there was a decrease in results for both schoolchildren whose teacher was technologically ready to conduct online classes and those whose teacher was not.
5. Discussion The results of this study indicate that the first two waves of the COVID-19 pandemic have resulted in a certain learning loss for Russian schoolchildren. While taking into account the limitations described above, we can, nevertheless, confidently make the following conclusions:
• Fourth-grade students have suffered virtually no loss in reading literacy. Schoolchildren from the pandemic cohort — those belonging to different groups by socioeconomic class composition, as well as those differing in the level of individual achievement — score at the same level as their predecessors before the pandemic.
• Sixth graders have experienced losses in reading literacy, but these are not very large: an average of about 0.24 sd, which is roughly equivalent to four months of schooling [Hattie, 2015]. We observe no worsening inequality in this case, rather the opposite: scores decline more strongly in groups with high socioeconomic class composition and among high-achieving students.
• Eighth graders have suffered a serious loss in science literacy. Inequality between groups of students differing in family wealth has not increased: children from groups by the socioeconomic composition of the classes have experienced equal learning losses. At the same time, the magnitude of loss varies greatly among students with different levels of achievement: low-achieving students have lost more in learning. The average effect is 0.87 SD, which is more than two years of schooling.
• In all grades, the level of literacy within the same cohort is strongly related to the socioeconomic composition of the class: the higher the proportion of children whose parents have a university degree, the better the results of children in that class.
• No significant relationship has been found between whether teachers can properly use ICT and the severity of the pandemic effect.
We will first discuss the results for grades 4 and 6 due to the common subject matter of the tests they have taken. The most obvious explanation for the difference in pandemic effects on 4th and 6th graders is that 4th graders have spent much less time in distance mode, which is characteristic not only of the Krasnoyarsk Re-
gion but of the whole country [Kosaretsky et al., 2022]. Moreover, 6th graders returned to in-person learning just before the monitoring study was held, which might have made their situation less favorable as they had fewer opportunities to catch up. As for 4th graders, even if they were first falling behind, they managed to catch up in a full six months of in-person learning. The loss in reading literacy among Russian 6th graders was substantial but did not exceed the average loss found in studies of schoolchildren in other countries [Storey, Zhang, 2021].21 In fact, the loss of the Russian sixth-graders, when expressed in years of study, was slightly less than the total time spent in distance learning, but we should take into account that there was also a summer vacation between the two waves of the pandemic.
A less obvious explanation for the difference in pandemic effects between 4th and 6th graders may have to do with how the reading instruction of primary school students is organized in the Russian education system. Russian 4th graders consistently score very high in reading literacy in the PIRLS study:22 they always crowd the top of the list, and in 2016, Russia headed the list of leaders. By contrast, the results of the Russian 15-year-old schoolchildren on reading literacy in PISA are much worse; the best result, which was the 19th to 30th positions in the international ranking, was achieved in 2015, while in the other years the scores were even lower [Адамович и др., 2019]. In primary school, children learn to read for further learning, as well as for literary and aesthetic experience and to learn the world through literary texts (these are the main purposes of reading as stated in the PIRLS study). In doing this, children are greatly helped by their families (only 1% of parents say that they did not teach their child to read before school)23 and the experience of autonomous reading, which still remains a cultural value and an integral part of a normal childhood in the worldview of Russian parents. In contrast, the basic school should teach reading "for life", that is, how to compare authors' attitudes and theses, identify contradictions and authors' implicit goals, assess the reliability of information, work with texts from different domains with their own specifics, and use information from the text in situations not directly related to those described in the text. These reading skills are less discussed in society, they are not typically addressed in the family, and they are spontaneously developed in a relatively small number of students. The school's role in developing these skills is therefore substantial.
21 https://www.mckinsey.com/industries/public-and-social-sector/our-insights/ covid-19-and-education-the-lingering-effects-of-unfinished-learning?cid=-soc-twi-mip-mck-oth-2107--&sid=5229869053&linkId=125555357#
22 https://ioe.hse.ru/lepa/news/212696860.html
23 Ibid.
The results for science literacy of 8th-grade students show a completely different picture. Even taking into account the limitations we have described, a loss of 0.87 SD is very high. None of the studies reviewed above, and none of the other reports24 describe effects of this magnitude. Comparable loss (up to 0.82 SD) has been reported only for mathematics literacy of schoolchildren in Mexico [Hevia et al., 2022]. There are very few studies devoted specifically to science literacy. For example, there is one conducted in Great Britain, but the learning loss reported there is much less significant.25 Assuming that the data we obtained are valid, their interpretation is problematic. Most likely, we observe a cumulative effect of all factors at once: Russian schoolchildren have never shown high results in science literacy in international comparative studies (a characteristic of the education system); eighth-graders have spent a very long time in distance learning and returned to in-person learning just before the test; it is much more difficult to study science at home than it is to maintain reading skills — primarily because of the lack of access to laboratory equipment and the necessary tools to complete practical assignments.
The worldwide trend of increasing educational inequality due to the pandemic is not supported by our data. The results of students from classes with different socioeconomic compositions vary widely, but this gap has not widened during the pandemic. First, the functional literacy of sixth- and eighth-graders from classes with different socioeconomic compositions declines by about the same value in our study, possibly because the study participants come from only one region of Russia and they are much more homogeneous in socioeconomic status than schoolchildren from different regions of Russia would be. Second, the effect of socioeconomic composition can be not very significant due to the overall poor preparedness of schools for using digital tools in the education process, that is, all schools proved to be equally unprepared. Third, the availability of in-person learning by itself may not have significantly reduced inequality among students, that is, the school was not successful in overcoming social inequality. As a result, the interruption in the students' interaction with their educational institution did not increase inequality among groups of schoolchildren with different wealth levels.
The results regarding the impact of the pandemic on groups of low- and high-achieving schoolchildren are inconsistent. The greatest loss in science literacy has been found among the lowest-performing eighth graders — over 2.5 years of study, while the
24 https://educationendowmentfoundation.org.uk/guidance-for-teachers/ covid-19-resources/best-evidence-on-impact-of-covid-19-on-pupil-attainment
25 https://gl-assessment.co.uk/media/352700/30443-gl-assessment-analysis-re-port-final.pdf
best-performing students have experienced 1.7 years of loss in this competency. Studies in other education systems have also found more significant declines in scores of low achievers [Kim et al., 2021; Storey, Zhang, 2021]. Some authors believe that low-achieving students experienced the greatest reduction in organized learning time and, in the absence of adequate school and parental support, replaced it least effectively with less useful activities [Grewenig et al., 2021]. However, among the sixth graders considered in our study, the best-performing students, conversely, showed the greatest decrease in reading literacy scores. The divergent trends found in the responses to the pandemic among sixth- and eighth-graders can be related both to the specifics of the subject areas tested and age differences among students. Yet the idea that the pandemic might hit successful students particularly hard already comes forward in some academic papers.26
The decrease in the heterogeneity of results — the gap between the scores of high and low achievers — in the 6th grades due to the pandemic most likely indicates that for reading literacy the effect of in-person learning is much more pronounced for motivated and high-performing students, and when schools move to distance learning these students, therefore, lose more than their lower-achieving peers. Conversely, the heterogeneity of science literacy scores in grades 8 increases, meaning that the gap between the high and low achievers becomes wider. Apparently, in this subject area, in-person education is most needed by the lowest-achieving children. Otherwise, in distance education, they lose much more in learning than their better-performing peers.
The conducted study did not reveal a correlation between the teacher's technical ability to conduct online lessons and smaller losses in test results in 2021. It must be acknowledged that the overall readiness of Russian teachers for the rapid transition to instruction using digital tools is insufficient: according to one survey, 80% of teachers faced problems when transitioning to the distance mode of instruction.27 Even when they had the necessary technology to conduct online classes, teachers still encountered many obstacles: not all children having the necessary devices, the teacher's lack of online instruction experience, and lack of methodological assistance in working with digital tools. It is probably the reason why the technical ability to teach remotely in a crisis situation was not as important a factor in reducing literacy loss. It may have been the skill of teaching remotely, which few people had time to master, that has played a key role.
26 https://www.smh.com.au/national/for-high-achievers-the-pandeiTiic-could-be-a-tipping-point-20200505-p54q2o.html
27 https://maximumeducation.com/news/survey_teachers%20
Taking into account all the above, is there any basis for the alarmist sentiment expressed by education researchers28 and practitioners at the beginning of the pandemic? Based on the results of our study, fourth graders suffered no learning loss because of the pandemic, and for sixth graders, the loss in reading was slightly greater than the average loss found in other studies. At the same time, the level of science literacy of 8th graders in 2021 turned out to be much lower than in 2019. However, the quality of science education in Russia had been short of satisfactory even in the past. This should have been a cause for concern well before the pandemic began.
In this study, we have found no evidence of the previously expected dramatic decline in the quality of education among the most disadvantaged children. Does this mean that we can ignore the pre-pandemic manifestations of educational inequality, such as the limited access to the basic resources necessary for learning, that apply to a large proportion of Russian schoolchildren? Definitely no; even regardless of the pandemic, new support measures for students are needed to compensate for inequality, such as dedicated vacation and summer schools, enhanced programs for at-risk schoolchildren, and extensive government tutoring programs for the poorest families.
Perhaps the main lesson of the pandemic for the Russian education system and policy makers should be the impossibility to assess the effects of the pandemic on the vast majority of the country. The reason is simple: in Russia, there are no national monitoring surveys that would ensure a stable assessment of the progress of students' basic skills at different stages of schooling. Such studies would not only allow us to estimate the effects of the pandemic validly and reliably but also build education policy based on transparent and interpretable data.
The study was implemented in the framework of the Basic Research Program at the National Research University Higher School of Economics (HSE University) in 2021.
The authors thank S. V. Semenov and I. V. Antipkina for their research assistance.
References Adamovich K., Kapuza A., Zakharov A., Froumin I. (2019) Osnovnye rezultaty rossi-yskikh uchashchikhsya v mezhdunarodnom issledovanii chitatel'skoy, matemat-icheskoy i estestvennonauchnoy gramotnosti PISA-2018 i ikh interpretatsiya [Russia Results in Math, Reading, and Science in PISA 2018 and What They Say about Education in the Country]. Moscow: HSE. Available at: https://ioe.hse. ru/data/2019/12/23/1525056145/$0 2(25) 2019 эпектроннblM.pdf (accessed 22 January 2022).
28 https://publications.hse.ru/view/384257919
Azevedo J.P., Hasan A., Goldemberg D., Iqbal S.A., Geven K. (2020) Simulating the Potential Impacts of COVID-19 School Closures on Schooling and Learning Outcomes. World Bank Group Policy Research Working Paper no 9284. Available at: https://openknowledge.worldbank.org/bitstream/handle/10986/33945/ Simulating-the-Potential-Impacts-of-COVID-19-School-Closures-on-School-ing-and-Learning-Outcomes-A-Set-of-Global-Estimates.pdf?sequence=1 (accessed 22 January 2022). Cooc N. (2019) Teaching Students with Special Needs: International Trends in School Capacity and the Need for Teacher Professional Development. Teaching and Teacher Education, vol. 83, no 1, pp. 27-41. doi:10.1016/j.tate.2019.03.021 DeMatthews D., Knight D., Reyes P., Benedict A., Callahan R. (2020) From the Field: Education Research During a Pandemic. Educational Researcher, vol. 49, no 6, Article no 0013189X2093876. doi:10.3102/0013189X20938761 Engzell P., Frey A., Verhagen M.D. (2021) Learning Loss due to School Closures during the COVID-19 Pandemic. Proceedings of the National Academy of Sciences, vol. 118, iss. 17, Article no e2022376118. doi:10.1073/pnas.2022376118 Fothergill A., Peek L.A. (2004) Poverty and Disasters in the United States: A Review of Recent Sociological Findings. Natural Hazards, vol. 32, no 1, pp. 89110. doi:10.1023/B:NHAZ.0000026792.76181.d9 Gao L.X., Zhang L.J. (2020) Teacher Learning in Difficult Times: Examining Foreign Language Teachers' Cognitions about Online Teaching to Tide over COVID-19. Frontiers in Psychology, vol. 11, September, Article no 549653. doi:10.3389/ fpsyg.2020.549653
Gershenson S., Tekin E. (2018) The Effect of Community Traumatic Events on Student Achievement: Evidence from the Beltway Sniper Attacks. Education Finance and Policy, vol. 13, no 4, pp. 513-544. doi:10.1162/EDFP_a_00234 Ghosh R., Dubey M.J., Chatterjee S., Dubey S. (2020) Impact of COVID-19 on Children: Special Focus on the Psychosocial Aspect. Minerva Pediatrica, vol. 72, iss. 3, pp. 226-235. doi:10.23736/S0026-4946.20.05887-9 Goodman J. (2014) Flaking Out: Student Absences and Snow Days as Disruptions of Instructional Time. NBER Working Paper no 20221. Cambridge, MA: National Bureau of Economic Research. doi:10.3386/w20221 Grewenig E., Lergetporer P., Werner K., Woessmann L., Zierow L. (2021) COVID-19 and Educational Inequality: How School Closures Affect Low-and High-Achieving Students. European Economic Review, vol. 140, no 3, Article no 103920. doi:10.1016/j.euroecorev.2021.103920 Hammerstein S., König C., Dreisörner T., Frey A. (2021) Effects of COVID-19-Related School Closures on Student Achievement-A Systematic Review. Frontiers in Psychology, vol. 12, September, Article no 746289. doi:10.3389/fpsyg.2021.746289 Hanushek E.A., Woessmann L. (2020) The Economic Impacts of Learning Losses. OECD Education Working Papers no 225. Available at: https://www.oecd.org/ed-ucation/The-economic-impacts-of-coronavirus-covid-19-learning-losses.pdf (accessed 22 January 2022). Hattie J. (2015) What Doesn't Work in Education: The Politics of Distraction. London: Pearson.
Hevia F.J., Vergara-Lope S., Velásquez-Durán A., Calderón D. (2022) Estimation of the Fundamental Learning Loss and Learning Poverty Related to COVID-19 Pandemic in Mexico. International Journal of Educational Development, vol. 88, Article no 102515. doi:10.1016/j.ijedudev.2021.102515 Hox J.J. (2010) Multilevel Analysis: Techniques and Applications. New York. NY: Rou-tledge.
Kardanova E.Yu., Neiman Yu.M. (2003) Problema vyravnivaniya v sovremennoy te-orii testirovaniya [The Problem of Alignment in Modern Testing Theory]. Vo-prosy testirovaniya v obrazovanii, no 8, pp. 21-40. Kelmendi F., Gresham J., Iqbal S.A. (2020) The Economic and Social Impact of COV-ID-19: Education. Western Balkans Regular Economic Report no 17. Available
at: https://documents1.worldbank.org/curated/en/590751590682058272/ pdf/The-Economic-and-Social-Impact-of-COVID-19-Education.pdf (accessed 22 January 2022).
Kersha Y. (2020) Sotsial'no-ekonomicheskaya kompozitsiya shkoly kak faktor vosproizvodstva neravenstva v obrazovanii [School Socioeconomic Composition as a Factor of Educational Inequality Reproduction]. Voprosy obra-zovaniya / Educational Studies Moscow, no 4, pp. 85-112. doi: 10.17323/18149545-2020-4-85-112
Kim D.H., Lee H.J., Lin Y., Kang Y.J. (2021) Changes in Academic Performance in the Online, Integrated System-Based Curriculum Implemented due to the COV-ID-19 Pandemic in a Medical School in Korea. Journal of Educational Evaluation for Health Professions, vol. 18, Article no 24. doi: 10.3352/jeehp.2021.18.24
Kosaretsky S., Zair-Bek S., Kersha Y., Zvyagintse, R. (2022) General Education in Russia during COVID-19: Readiness, Policy Response, and Lessons Learned. Primary and Secondary Education during COVID-19. Disruptions to Educational Opportunity during a Pandemic (ed. F.M. Reimers), Cham, Switzerland: Springer, pp. 227-261.
Kosaretsky S.G., Zvyagintsev R.S., Kersha Yu.D., Froumin I.D. et al. (2020) Prog-noziruemye poteri dlya shkol'nogo obrazovaniya iz-za pandemii COVID-19: otsenki i poisk sposobov kompensatsii [The COVID-19 Pandemic Losses for Education: Forecasts and Search for Ways of Compensation]. Moscow: HSE.
Kuhfeld M., Tarasawa B. (2020) The COVID-19 Slide: What Summer Learning Loss Can Tell Us about the Potential Impact of School Closures on Student Academic Achievement. Available at: https://www.nwea.org/content/uploads/2020/05/ Collaborative-Brief Covid19-Slide-APR20.pdf (accessed 22 January 2022).
Lin M., Lucas H., Shmueli G. (2013) Too Big to Fail: Large Samples and the p-Value Problem. Information Systems Research, vol. 24, no 4, pp. 906-917. doi:10.1287/ isre.2013.0480
Maldonado J.E., De Witte K. (2021) The Effect of School Closures on Standardised Student Test Outcomes. Leuven: KU Leuven, Department of Economics.
Marcotte D.E., Hemelt S.W. (2008) Unscheduled School Closings and Student Performance. Education Finance and Policy, vol. 3, no 3, pp. 316-338. doi:10.1162/ edfp.2008.3.3.316
Mertsalova T., Zair-Bek S., Anchikov K., Senina N. (2021) Shkoly protiv pandemii: ef-fekty shkol'nogo obrazovaniya [Schools against the Pandemic: The Effects of School Education]. Otsenka kachestva obrazovaniya v usloviyakh distantsionno-go obucheniya: opyt prozhivaniya pandemii sistemami shkol'nogo obrazovaniya stran postsovetskogo prostranstva [Assessment of the Quality of Education in the Conditions of Distance Learning: The Experience of Living with the Pandemic by the School Education Systems of the Post-Soviet Countries] (eds V.A. Bolotov, T.A. Mertsalova), Moscow: Aleks, pp. 111-125.
Murnane R.J., Willett J.B. (2010) Methods Matter: Improving Causal Inference in Educational and Social Science Research. Oxford: Oxford University.
Neiman Yu.M., Khlebnikov V.A. (2000) Vvedenie v teoriyu modelirovaniya iparametri-zatsii pedagogicheskikh testov [Introduction to the Theory of Modeling and Parametrization of Pedagogical Tests]. Moscow: Testing Center of the Ministry of Education of the Russian Federation.
Patrick S.W., Henkhaus L.E., Zickafoose J.S., Lovell K. et al. (2020) Well-Being of Parents and Children during the COVID-19 Pandemic: A National Survey. Pediatrics, vol. 146, no 4, Article no e2020016824. doi:10.1542/peds.2020-016824
Psacharopoulos G., Collis V., Patrinos H.A., Vegas E. (2021) The COVID-19 Cost of School Closures in Earnings and Income across the World. Comparative Education Review, vol. 65, no 2, pp. 271-287. doi:10.1086/713540
Sapkota J.B., Neupane P. (2021) The Academic Impacts of 2015 Nepal Earthquake: Evidence from Two Secondary Schools in Sindhupalchok District. Education Sciences, vol. 11, Article no 371. doi:10.3390/educsci11080371
Sirin S.R. (2005) Socioeconomic Status and Academic Achievement: A Meta-Ana-lytic Review of Research. Review of Educational Research, vol. 75, no 3, pp. 417453. doi:10.3102/00346543075003417 Stern M.J., Adams A.E., Elsasser S. (2009) Digital Inequality and Place: The Effects of Technological Diffusion on Internet Proficiency and Usage across Rural, Suburban, and Urban Counties. Sociological Inquiry, vol. 79, no 4, pp. 391417. doi:10.1111/j.1475-682X.2009.00302.x Storey N., Zhang Q. (2021) A Meta-Analysis of COVID Learning Loss. doi:10.35542/ osf.io/qekw2
Thamtanajit K. (2020) The Impacts of Natural Disaster on Student Achievement: Evidence from Severe Floods in Thailand. The Journal of Developing Areas, vol. 54, no 4, pp. 129-143. doi:10.1353/jda.2020.0042 Tomasik M.J., Helbling L.A., Moser U. (2021) Educational Gains of In-Person vs. Distance Learning in Primary and Secondary Schools: A Natural Experiment during the COVID-19 Pandemic School Closures in Switzerland. International Journal of Psychology, vol. 56, no 4, pp. 566-576. doi:10.1002/ijop.12728 Wright B.D., Stone M.N. (1979) Best Test Design. Rasch Measurement. Chicago: Mesa. Yu C.H. (2002) Resampling Methods: Concepts, Applications, and Justification. Practical Assessment, Research, and Evaluation, vol. 8, no 1. doi:10.7275/9CMS-MY97
Zair-Bek S., Mertsalova T., Anchikov K. (2020) Gotovnost' rossiyskikh shkol i semey k obucheniyu v usloviyakh karantina: otsenka bazovykh pokazateley [Readiness of Russian Schools and Families for Teaching and Learning under Quarantine: Evaluation of Basic Indicators]. Moscow: HSE.
