CC BY-NC-ND
90
Digital Transformation of Schools and Student's Information and Communication Literacy
S.M. Avdeeva, A.Yu. Uvarov, K.V. Tarasova
This article was submitted to the Editorial Board in December 2021
Svetlana Avdeeva — Candidate of Technical Sciences, head of the Laboratory for Measuring New Constructs and Test Design, Centre for Psychometrics and Measurement in Education, Institute of Education, National Research University Higher School of Economics; deputy executive director, National Training Foundation. E-mail: [email protected]. (Corresponding author)
Alexander Uvarov — Doctor of Sciences in Education, leading researcher at the Institute of Cybernetics and Educational Computing, The Federal Research Centre "Computer Science and Control" of the Russian Academy of Sciences; Professor at the Department of Educational Programs, Institute of Education, National Research University Higher School of Economics. E-mail: [email protected] Ksenia Tarasova — Candidate of Sciences in Education, deputy head of the Laboratory for Measuring New Constructs and Test Design, Centre for Psychometrics and Measurement in Education, National Research University Higher School of Economics. E-mail: [email protected]
Address: 16 Potapovsky Lane, bldg. 10, Moscow, 101000, Russia
Abstract Information and communication literacy is one of the main meta-subject competencies that graduates of the secondary school should possess. The full-fledged formation of this competence is considered as one of the tasks of preparing students for life in the information society and the digital economy. The article discusses the results of a monitoring study of the information and communication literacy of 9th grade students, which was conducted in 21 regions of the Russian Federation in 2020. About half (45.4%) demonstrated a level of competence that corresponds to the readiness for life in the digital economy. The influence of the equipment of schools and the peculiarities of the organization of educational activities on the students' level of information and communication literacy (ICL) was considered. The study results indicate a strong connection between the students' competence and their out-of-school environment and a weak — with their work in the school. Research findings allow to determine the reasons for the insufficient level of ICL in a significant part of graduates and used to propose recommendations for its increase.
Keywords information and communication competence; mail school; digital transformation of education; scenario assessment methods; computer testing; assessment of competencies.
For citing Avdeeva S.M., Uvarov A.Yu., Tarasova R.V. (2022) Tsifrovye tekhnologii v shkole i informatsionno-kommunikatsionnaya kompetentnost' uchashchikhsya [Digital Transformation of Schools and Student's Information and Communication Literacy]. Voprosy obrazovaniya / Educational Studies Moscow. No 1. P. 218-243. https:// doi.org/10.17323/1814-9545-2022-1-218-243
Forty years ago, Academician A. Ershov had validated the notion of a second ('computer') literacy as an important meta-subject competency of an information society member [Ershov, 1981]. Nowadays this notion is universally accepted and underpins the concept of digital literacy as well as information and communication literacy (ICL). Ten years ago, the cultivation and development of students' ability to use information and communication technologies became part of the obligatory requirements for the results of the basic educational programme in secondary schools1.
The Organisation for Economic Co-operation and Development recommends that all member countries regularly assess digital literacy in secondary school as a mandatory component of the development of digital learning environments [OECD, 2021]. Such an environment not only provides the prerequisite for the development of ICL in students, but also makes it possible to use previously inaccessible intellectual tools to assess the effectiveness of its development. Since 2013, the International Computing and Information Literacy Study (ICILS) for 8th grade students has been held, coordinated by the International Association for the Evaluation of Educational Achievement (IEA). Two cycles of the ICILS study have been conducted. The third cycle will take place in 2023.
The implementation of the Federal Project Digital Learning Environment, for its part,2 also requires monitoring of the digital transformation of general education and regular assessment of the ICL level of students. Under this project, schools purchase computers and other digital equipment, increase digital bandwidth, develop digital courseware, and organise additional professional development programmes for teachers. In order to assess how these efforts affect students' ability to handle information in a digital environment, solve learning tasks and problems outside of school, and their commitment to lifelong learning, in February-October 2020 a study was commissioned by the Ministry of Education of the Russian Federation as part of the project Monitoring the level of information and communication literacy of 9th grade students (graduates of basic schools). The monitoring also included an online survey of teachers and school administrators about the use of digital technology and the organisation of the educational process at school.
1 Order of the Ministry of Science and Education of the Russian Federation No. 1897 of 17 December 2010 On approval of the Federal State Educational Standard for Basic General Education.
2 The Federal Project Digital Learning Environment is part of the National Project Education. It aims to create and implement a digital learning environment in all educational institutions in the country, which would ensure a digital transformation of education. The project seeks to provide organisations with modern equipment and develop digital services and content for educational activities: https://edu.gov.ru/national-project/
The information and communication literacy of 9th grade students was assessed using the ICL test (Information and Communication Literacy Test). It is a state-of-the-art instrument whose validity has been confirmed in the ICL assessment of several thousand Russian and foreign schoolchildren, also within the framework of World Bank projects. It provides fast, real-time automatic processing of the measurement results, displays the test results to each student on the computer screen in an interactive mode and suggests personalised guidance for improvement of their ICL. The ICL test has gained international recognition: the University of Helsinki is localising it for use in Finland, while the Inter-American Development Bank is adapting it for Latin American countries. The purpose of this study is:
• to document, using a reliable up-to-date tool, the level of pupils' ICL at the graduation from basic school in the context of the roll-out of the Federal Project Digital Learning Environment; to evaluate the extent to which the availability of digital equipment, digital services and digital learning content in an educational institution affects the level of ICL of basic school leavers;
• to identify the specific characteristics in the work of educational institutions that influence the formation of ICL in the basic school.
Current approaches to assessing ICL and related constructs
The concept of computer literacy (competency) has not yet been fully established3. The terms 'computer literacy', 'digital literacy', 'information and communication technology competency' ('ICT competency'), 'digital competence' etc. are often used as equivalent terms4. An academic dictionary has retained a late last century interpretation of computer literacy as 'the mastery of skills in the use of computer hardware; an understanding of the fundamentals of computer science and the importance of information technology in society'5. This interpretation has informed the course in computing, which became a compulsory part of the school curriculum in Russia more than 35 years ago. The framework curriculum for ba-
3 For instance, the website of the Agency for Social Initiatives defines digital competences as an element of digital literacy: 'Digital literacy is the body of knowledge and skills that are necessary for the safe and efficient use of digital technology and Internet resources. It includes digital consumption, digital competences, and digital security'. https://old.asi.ru/future skills/
4 In the English-language sources, a number of terms with similar meanings are used: computer literacy, digital literacy, digital competence, ICT literacy, information literacy, internet literacy, digital culture.
5 https://dic.academic.ru/dic.nsf/enc3p/161138
sic general education6 defines ICT competency as 'the ability to use ICT for information management in learning tasks, individual cognitive activities, life and work in the present-day high-tech society'.
In the mid-1990s, the European Commission supported the development of a working standard and certification tool for assessing computer literacy in Europe7. The new framework and associated measurement tool came into use in August 1996 in Sweden under the name 'European Computer Driving License' (ECDL). By the end of the 1990s, the ECDL had become popular in Europe and expanded beyond its borders to become known as International Computer Driving License (ICDL). By the early 2010s, with the support from UNESCO, it had spread around the world. ICDL testing services were provided by more than 20,000 certification centres in 148 countries. Today, the updated ICDL tools8 enable individuals to assess their computer skills and knowledge of the most common computer applications. While constantly evolving, the ICDL framework, just like other such tools, does not test the ability to work with information in a digital environment. It is limited to technological skills, and the associated training materials and testing tools are primarily aimed at bridging the digital divide in technology.
The key importance of computer literacy in a digital environment was pointed out as early as the 1980s by the American Library Association [Association of College and Research Libraries, 1989]. By the beginning of the 21st century, insights had emerged on how the spread of information technology affects the requirements for information literacy in the workforce [ITAA, 2000]. An international panel of experts convened by the Educational Testing Service (ETS) in 2002 pointed out that the digital divide is not only created by a lack of access to computers, software, and the Internet. Equally important in this respect is the lack of general literacy and the inability of users to work with information. Experts stated that mastering technology skills without developing general and information literacy would not help to bridge the growing digital divide [International ICT Literacy Panel, 2002. P. 6]. The international panel defined ICT literacy as 'using digital technology, communications tools, and/or networks to access, manage, integrate, evaluate, and create information in order to function in a knowledge society' [Ibid. P. 2].
The proposed ICL framework provided the basis for most of ICL assessment tools [Virkus, 2003; Webber, Johnston, 2017; Kim, Kil, Shin, 2014; Eisenberg, Lowe, Spitzer, 2004; Sparks, Katz, Beile, 2016]. It underpins The European Digital Competence Framework for Cit-
6 Approved by the Federal Methodological Association for General Education (Record 1/15 of 8 April 2015) http://fgosreestr.ru/registry/primernaya-os-novnayaobrazovatelnaya-programma-osnovnogo-obshhego-obrazovaniya-3/
7 http://icdlcenter.com/about/our-history.html
8 http://www.ecdl.com
izens — DigComp9. The DigComp framework includes the ability to identify information needs, find and retrieve the right digital material (content), assess the relevance of the material and its source, organise, store and use the information [Carretero, Vuorikari, Punie, 2017]. The DigComp framework, in turn, was used as the basis for developing a framework for information and data literacy by UNESCO [UNESCO, 2018]. The latter included the identification of information needs, data search and retrieval, assessment of the credibility of source and content, storage, management and organisation of the information.
The ICT Competency Framework that has been developed by the international panel of experts assembled by ETS, was used to create the NAP ICTL tool in Australia. This tool is used to monitor ICL at the national level and to identify factors influencing its development [Ainley et al, 2005]. It represents a collection of multiple-choice questions. Based on the ICTL NAP, a tool has been developed and has been used since 2013 for assessing schoolchildren's computer and information literacy in International Computer and Information Literacy Study (ICILS). Russian schools were also involved in this study [Avdeeva, 2015; Gvozdev, Nikulin, Rodnevska-ya, 2017]. The assessment framework of ICILS [Frailon et al., 2019] bundles technical (computer) literacy, information skills and digital communication used to achieve various educational and communication goals in problem solving.
Tool of the study The ICL test, as well as the ICILS tools, relies on a definition and framework of information and communication literacy that have been developed by the panel of experts assembled by the ETS. During the ICL test, students are immersed in the digital environment of the tool.
The test uses 16 scenario-type tasks of varying difficulty and is based on evidence-centred design10. Recognising that ICL is not solely acquired at school, the test contains tasks that have academic, out-of-school and personal contexts, with the academic (school) context making up 40% of the set. All problem scenarios (tasks) presented to the respondents are as close as possible to everyday life. For further details on the ICL test, see [Avdeeva et al., 2017].
During test development, five achievement level descriptors (ALDs) were established for ICL as a measured construct and for its seven components, with their contribution to the overall con-
9 See https://ec.europa.eu/jrc/en/digcomp
10 Evidence-centred design involves a set of interrelated procedures that help to answer the two questions: what in the respondents' behaviour indicates that they do or do not possess the competencies of interest; how can we create a situation that allows us to determine this? [Mislevy, Almond, Lukas, 2003].
struct of ICL, as well as sets of observable activities whose accomplishment indicates the level of respondent's mastery of each of those components. These indicators facilitate the use of test results for students and their teachers as well as for school administrators and other decision-makers in education. The units of measurement of the ICL are testimonies (observable variables) and not the test tasks themselves. The chosen procedure for developing the test made it possible to relate the latent variables (ICL and its components) to the observed indicators (testimonies) in a transparent way. Moreover, the tasks themselves may contain varying numbers of such testimonies, and each element of the task serves to record the manifestation of one of the components of the ICL. This design makes it possible to create numerous versions of the test, to cover all components of the ICL and to compile the test from items of varying difficulty.
Table 1 presents descriptions of five levels of ICL reflecting the abilities of 9th grade students to work with information in a digital environment.
Table 1. Abilities Corresponding to Different ICL Levels of 9th Grade Students
ICL level
Advanced The student performs the tasks at a high level, related to abilities to: • formulate the problem correctly; • And information from different sources; • organise information according to certain criteria; • assess the quality of information and the reliability of its sources; • compare and synthesise information from different sources; • draw the right conclusions from existing information; • present information to other people.
Above basic The student performs the tasks at a high level, related to abilities to: • formulate the problem correctly; • And information from different sources; • organise information according to certain criteria; • draw the right conclusions from existing information; • share information with other people. The student performs the tasks at a satisfactory level, related to abilities to: • assess the quality of information and the reliability of its sources; • compare and synthesise information from different sources.
Basic The student performs the tasks at a satisfactory level, related to abilities to: formulate the problem correctly; • ??? • And information from different sources; • organise information according to certain criteria; • assess the quality of information and the reliability of its sources; • compare and synthesise information from different sources; • draw the right conclusions from existing information; • share information with other people.
ICL level
Below basic The student performs the tasks at a satisfactory level, related to abilities to: • formulate the problem correctly; • And information from different sources; • organise information according to certain criteria; • draw the right conclusions from existing information; • share information with other people. The student fails to perform the tasks related to abilities to: • assess the quality of information and the reliability of its sources; • compare and synthesise information from different sources.
Developing The student fails to perform the tasks related to abilities to: • formulate the problem correctly; • And information from different sources; • organise information according to certain criteria; • draw the right conclusions from existing information; • share information with other people.
The ICL test has a number of features that make it unique. It assesses a basic school leaver's ability to use computers and digital technology in the acquisition of new knowledge, communication, research, lifelong learning, and self-realisation in professional life. The automatic processing of the test results makes it possible to inform the students of their ICL level immediately after the test is completed and to offer recommendations for its improvement. Thus, the ICL test can be used for both resulting (monitoring studies) and formative (in the learning process) assessment of ICL levels. The ICL test is the subject of a patent for an invention11.
Design The Centre of Education Quality Monitoring at the National Re-of the study search University Higher School of Economics, with active involvement of education authorities, school administrators and teachers, conducted a monitoring study of the ICL level among basic school leavers in 21 regions of the Russian Federation in 2020. The aim was to assess the ability of 9th grade students to operate with information, solve practical problems using ICL, think and work in a 'digital world'. The administrative regions for the study were selected on the following criteria:
• participation in the federal project to introduce a targeted model for a digital learning environment in general education institutions;
• experience of monitoring studies to assess the quality of education, including international comparative studies;
l • ong-term successful experience of introducing ICT into the educational process.
http'//vo hse ru " Patent for an invention No. 2656699: ICL test — a tool for measuring informa-
tion and communication literacy in the digital environment. Authors: Avdeeva S.M., Tarasova K.V. Et al. Date of state registration in the State Register of Inventions of the Russian Federation: 6 June 2018.
From the 21 regions that met these criteria, data were collected on their general education institutions, such as: type of school, location area, the number of 9th grade classes, the number of students enrolled, etc. Based on these data, a two-step stratified cluster sample was created, in which the strata corresponded to the location of schools (urban or rural), and the clusters corresponded to study groups (classes) of specific schools. All 9th grade students in the selected regions were treated as the general population. Classes were randomly added to the generated sample (classes smaller than 6 or larger than 30 students were discarded), until the total number of students in the sample exceeded 36,000. The selection of classes was made separately among urban and rural schools in proportion to the total sample size and their representation in the general population. A total of 30,011 ninth graders took part in the study. The criteria for selecting regions and a description of the sample are presented in more detail on the website of the study12.
Alongside the ICL assessment, a questionnaire was completed by students, school administrators and teachers who taught in the classes being tested. Teachers and administrators obtained individual accounts and filled in the questionnaire electronically at ictlit.ru at their convenience. The students filled in their questionnaires during online testing in computer classrooms, where each 9th grade student was assigned an individual workstation with Internet access. Before the testing, school coordinators briefed the students. They then made sure that the test was carried out by the schoolchildren themselves, recording breaches of procedure in the protocol. At the end of the test, the results were automatically sent to the server and each student received a message about their ICL level and personalised advice on how to improve it.
The results of the ICL study of 9th grade students in 2020 are shown in Figure 1. Just under a third of the study participants (29.6%) have a basic ICL level. 12.4% of schoolchildren are above the basic level and 3.4% are at the advanced level, i.e., about half of 9th grade students (45.4%) are prepared for life in the digital economy.
More than a third (35.3%) of the tested ninth graders are below the basic level. In order to reach the basic and higher levels, their ICL needs to be improved. For those at the 'developing' level (19.3%), additional training sessions are not enough: systematic work must be purposefully organised.
In rural schools, more than a quarter of students (25.6%) were at the developing level. At the same time, there are almost half as many rural ninth graders at the advanced level as in urban schools
12 https://ioe.hse.ru/monitoring/monitonng_icl/materials
Results ofthe study and discussion
(1.9% versus 4.1%). In contrast to cities, in rural areas the secondary school is practically the only institution capable of providing all learners with the digital hardware, software and resources (individual training assignments, collaborative projects, online courses, etc.) required to develop their ICL.
Figure 1. Results of ICL assessment in 2020 (all regions)
403530252015105
0
25.6
19.3
16.3
ICL level (%), all regions
35.334.9
36.4
29.6
30.7
27.2
12.4
14.0
8.9
3.2
4.1
on
1.9
Developing Below basic Basic Above basic
□ Total □ Urban □ Rural
Advanced
To sum up, a significant gap has been identified in the ICL level between urban and rural school leavers. They have the same type of distribution of their test results, that is left-skewed, but the number of those who showed a developing level of ICL among rural school leavers is half as large again as among urban school leavers (25.6 vs. 16.3%). At the 'above basic' level there are one and a half times fewer rural school leavers than urban ones (8.9 vs. 14%), and at the advanced level there are half as many (1.9 vs. 4.1%). However, the gap in the percentages of urban and rural school leavers at 'below basic' and 'basic' levels is not as large (1.5 and 3.5%). These data suggest that there is a systemic problem preventing the development of ICL in the mainstream school environment.
The percentages of students who showed the developing ICL level varied considerably across the regions of the Russian Federation (Figure 2). Thus, in Astrakhan, Voronezh and Nizhny Novgorod Oblasts this rate is more than twice as high as in Yamalo-Nenets Autonomous Okrug. Tyumen Oblast (5.6%) and Perm Krai (5.3%) lead at the advanced level, while the results of Altai Krai, Astrakhan, Kaluga and Novgorod Oblasts are almost half as high (< 3%). The leaders in the development of the ICL of basic school leavers are Tyumen, Kaliningrad and Chelyabinsk Oblasts, Krasnoyarsk and Perm Krais, and Yamalo-Nenets Autonomous Okrug.
Meanwhile, in the regions, as well as in the total sample, significant differences in ICL levels were found between urban and ru-
Figure 2. Results of ICL assessment of 9th grade students in 21 regions of the Russian Federation
ICL level by regions In 2020
27.1 I 40.9 I 23.8 |5.2 2H
19.5 | 36.4 I 31.2 I 8.7 EU 27.4 I 40.9 I 23.8 I 5.2 129
24.7 I 39.3 I 26.5 I 7.2"25
21.9 I 36.5 I 29.0 I 8.4 141
28.9 I 39.3 I 22.2 I 5,7 I3j
25.4 I 41.7 I 22.2 I 8.3 2J
21.4 I 38.1 I 27.0 I 8.2 I p]
20.7 I 39.0 I 26.7 I 8.3 5*^
22.8 I 39.1 I 26.4 I 7.5^*2 23.3 I 37.4 I 27.2 I 10.1^*0
12.5 I 34.6 I 32.7 I 15.4 14*Sl
17.5 I 31.2 I 31.3 I 15.2 14.7
18.3 I 36.6 I 31.4 I 12.6^1 19.8 I 34.9 I 31.9 I 11.1 2.3|
24.9 I 33.9 I 28.5 I 10.4 2.3
20.0 I 36.5 I 28.5 I 11.2 l|7
16.4 1 31.8 I 32.2 I 14.0 15^
17.6 I 35.2 I 30.4 I 12.7 142 11.1 I 40.2 I 27.8 I 16.4 4^4 15.5 I 36.9 I 27.5 I 15.5 I 1
-1-1-1-1-1-1-1-1-1-1
0 10 20 30 40 50 60 70 80 90 100
□ developing □ below basic □ basic □ above basic □ advanced
ral students. In Kaluga Oblast, more than a third of students in rural schools (36.8%) and only a fifth in urban schools (20.9%) were at the developing level. In Perm Krai, there were almost twice as many students in rural schools (28.6%) as in urban schools (14.6%) at the developing level. In Tyumen Oblast, urban schools had twice as many students at the 'above basic' and 'advanced' level as rural schools (23.9 vs. 11.6%). The same pattern can be observed in Yamalo-Nenets Autonomous Okrug (23.8 vs. 9.9%).
For further analysis, the five levels of ICL have been numbered from 1 (developing) to 5 (advanced), making it possible to calculate the average ICL level for a school or group of schools. All the educational institutions taking part in the study have been ranked according to the average ICL level of all respondents in the school and then divided into 10 groups with an equal number of schools in each group. Figure 3 shows the percentage of ninth graders at each ICL level in each of the ten groups of schools and the average ICL level per group (in parentheses next to the group number).
The first group consisted of the educational institutions whose 9th grade students had the lowest level of ICL. On average, 60% of their students are at the developing level, 30% are at the 'below basic' level and just under 10% are at the basic level. In the tenth group, which had the highest average ICL level, only 3% of ninth graders were at the developing level, 20% were at the 'below basic'
Astrakhan Oblast Bryansk Oblast Voronezh Oblast Kaluga Oblast Kemerovo Oblast Nizhny Novgorod Oblast Novgorod Oblast Perm Krai Sakhalin Oblast Stavropol Krai Altai Krai Kaliningrad Oblast Krasnoyarsk Krai Novosibirsk Oblast Pskov Oblast Republic of Bashkortostan Rostov Oblast Tyumen Oblast Chelyabinsk Oblast Yamalo-Nenets Autonomous Okrug Yaroslavl Oblast
Figure 3. Percentage of 9th grade students representing each ICL level in each of the ten school groups
10 (3.33) 3 20 34 29 14
I 9 (2.98) 5 27 41 21 6
o 8 (2.76) 8 33 1 39 16 4
§ 7 (2.58) 12 I 36 1 35 1 16 1
§ 6 (2.43) 15 40 33 1 11 1|
1 5 (2.31) 20 41 29 1 8 2
CO _
t 4 (2.17)_ 25 42 25 1 7 1|
-5 3 (2.02) 29 45 20 M
S 2 (1.83)" 39 41 17 |3|<
1 (1.51) r 60 30 9 1
□ Developii ng □ Below basic □ Basic □ Above basic □ Advanced
level, while the majority of students reached the basic level (34%), the 'above basic' level (29%) and the advanced level (14%). In the 1st group the predominance of rural schools was observed (70%), and in the 10th group urban schools were prevailing (74%).
The increase in the average score between lower groups of schools (1st to 3rd) is mainly due to an increase in the proportion of students at the 'below basic' and basic levels and a sharp decrease in the proportion of ninth graders at the developing level. Further on, the increase in the average score is due to both a decrease in the percentage of students at the developing level and an increase in the number of students at the basic and 'above basic' levels.
The impact ofschool and out-of-school environment
on the
information and communication literacy of 9th grade students
One of the aims of the study was to find out how ninth graders' ICL levels were related to the use of digital technology in educational institutions, that is, to what extent the formation of ICL is the result of school work (the use of digital technology in teaching, the pedagogical practices implemented in schools, etc.), and to what extent it depends on the out-of-school environment (cultural, educational and material resources of the family, the accessibility of digital technology at home, its use outside school, etc.).
A questionnaire survey of ninth graders showed that the majority (79%) use computers at school only in computing classes. Less than a tenth (9%) of students use computers in classes in various subjects, and 12% use them mostly in the library or in the media centre. However, these differences do not have a significant impact on the level of ICL (Figure 4).
This observation is broadly consistent with the conclusion drawn from PISA studies that there is little correlation between the ICT ac-
Figure 4. The relationship between the ICL levels of students and the accessibility of digital technology in school
35-1
30-
oa 25-
»a oa
»a
3 10-1
20 15
5
0
jmQ
Developing Below basic Basic Above basic Advanced
□ In the school I can only use a computer in computing classes
□ In the school I can use a computer in classes in various subjects
□ In the school I can only use a computer in a dedicated place: the library or the media centre
cess in schools and the educational outcomes of students in traditional school systems [OECD, 2015; Schleicher, 2019].
As shown in Table 2, the regression coefficients between the level of ICL and the amount of time spent by students using digital technology for learning and entertainment do not differ across the sample as a whole. However, for rural school students, the regression coefficient between the level of ICL and the use of digital technology for learning is half as low as the respective regression coefficient for the use of digital technology for entertainment (R = 0.06 vs. R = 0.12). These data can be seen as evidence that in rural areas the influence of school work on the formation of ICL is noticeably weaker in comparison to the out-of-school environment.
Table 2. The Relationship between the ICL of 9th Grade Students and the Use of Digital Technology for Learning and Entertainment
Variable Schools [ß, p, (SE)]
All Rural Urban
How much time do you use your computer, tablet, smartphone and other gadgets for:
learning (doing schoolwork, etc.) 0.08*** 0.06*** 0.09***
(0.01) (0.02) (0.02)
entertainment (playing games, watching videos, etc.) 0.08*** 0.12*** 0.07***
(0.01) (0.02) (0.02)
communication (video and/or audio calls, chat messaging and social media) -0.04*** -0.06*** -0.03
(0.01) (0.02) (0.02)
Significance level: ***p < .01, **p < .05, *p < .10. Standard error (SE) is given in parentheses.
A number of questions in the questionnaire aim to clarify the nature of the relationship between 9th graders' ICL and the use of digital technology in school. This relationship was found to be positive with regard to two questions (Table 3): (a) 'How often do you and/ or your classmates work with information in school lessons, alone or in a group: confront facts and/or concepts, compare, classify and analyse information (making tables, charts, etc.)?' and (b) 'How often are you assigned homework from a textbook or book of problems at school?'. All other indicators showed a negative relationship.
Table 3. The Relationship between the ICL of 9th Grade Students and the Use of Digital Technology in School
Variable Schools [ß, p, (SE)]
All ! Rural Urban
How often do you use a computer, gadgets or an interactive whiteboard in class (except for computing classes) to:
carry out experiments and/or laboratory work -0.07*** -0.04 -0.08***
(0.02) (0.04) (0.03)
take quizzes -0.06*** -0.05 -0.07***
(0.02) (0.03) (0.02)
How often in school lessons do you and/or your classmates:
play educational games / business games, practise training, role-playing, etc. -0.11*** -0.06 -0.14***
(0.02) (0.04) (0.03)
draw diagrams, construct concept maps (e.g., mind maps) -0.09*** -0.07* -0.09***
(0.02) (0.04) (0.03)
work with information on your own or in a group: confronting facts and/or concepts, comparing, classifying and analysing information (making tables, charts, etc.) 0.10*** (0.02) 0.06* (0.03) 0.12*** (0.02)
At school, how often are you assigned homework:
from a textbook or a book of problems 0.21*** 0.26*** 0.18***
(0.02) (0.04) (0.03)
requiring the preparation of slides and/or a text for a presentation on a computer -0.07*** -0.06* -0.07***
(0.02) (0.03) (0.02)
requiring work in team with classmates -0.10*** -0.10*** -0.10***
(0.02) (0.04) (0.03)
Significance level: ***p < .01, **p < .05, *p < .10. Standard error (SE) is given in parentheses.
The connection between the ICL level and the frequency with which students practise information analysis is obvious: over 45% of students replied 'often' or 'every day' to question (a). It is also possible to explain the correlation between the ICL level and traditional
approach to homework based on a textbook or book of problems. Students are regularly required to do this work, which has a positive impact on their ability to read texts meaningfully and to work with information. However, the mechanisms of this connection need to be further explored.
The questionnaire survey of ninth graders did not show a connection between the ICL level of schoolchildren and the use of innovative, digitally supported learning methods (search for information on the Internet, carrying out experiments, laboratory work and quizzes, business games, simulations, role-playing, etc.). Moreover, the use of game methods, concept maps and group tasks is negatively related to the formation of ICL (Table 3). An explanation could be the rare use of such practices in the learning process: when asked 'How often do you or your classmates draw diagrams, create associative maps?' 80.7% of the respondents answered 'rarely' or 'never'.
These data suggest that the accessibility of digital technology at school and existing learning practices are weakly related to the ICL level of 9th graders.
Among the factors of the out-of-school environment, the ICL level of 9th grade students is related to their plans to continue their studies, the availability of books and smartphones at home, and their mothers' educational level (Table 4).
Students with high educational ambitions, that is those who plan to continue their education, have a higher level of ICL (R = 0.23 for rural students and R = 0.13 for urban students). A strong correlation has been revealed between the ICL of the ninth graders and their mothers' educational level: among pupils with developing ICL scores, those whose mothers have higher education are 2.3 times fewer than those whose mothers have only completed secondary school. At the 'above basic' and 'advanced' levels, there are 2.5 times more students whose mothers have higher education than those whose mothers have only completed secondary school. The correlation of the average ICL level of students in the ten groups of schools (see Figure 4) with their plans for further education and with their mothers' education is shown in Figure 5.
The availability of a smartphone was found to be a significant factor contributing to higher ICL levels of the ninth graders: this device is used most often to access the Internet (81%). For urban students, there is also a strong correlation between the ICL level and the availability of books at home, as well as with the amount of time spent by parents on computers: among students with developing ICL, there are 3.7 times more those whose parents do not use computers at all (7%) than students whose parents spend a lot of time on a computer (26%). According to the data collected, more than 90% of basic school leavers are active users of modern digital devices. 93% of students think they can freely use the Internet for
Table 4. The Relationship between the ICL of 9th Grade Students and their Plans for Future Education as well as Home Setting
Variable Schools [ß, p, (SE)]
All i Rural Urban
Plans for future education:
to continue education at the comprehensive school 0.17*** 0.23*** 0.13***
(0.03) (0.05) (0.04)
to go to a vocational school -0.09*** 0.03 -0.16***
(0.03) (0.05) (0.04)
to go to work -0.21*** -0.16 -0.25***
(0.07) (0.12) (0.09)
Home setting:
availability of a smartphone 0.22*** 0.26*** 0.19***
(0.05) (0.07) (0.07)
mother's higher education 0.11*** 0.10*** 0.11***
(0.02) (0.03) (0.02)
availability of more than 200 books at home 0.09*** 0.02 0.11***
(0.02) (0.04) (0.02)
the amount of time spent by parents on the computer 0.06*** 0.03 0.06***
(0.01) (0.02) (0.02)
Significance level: ***p < .01, **p < .05, *p < .10. Standard error (SE) is given in parentheses.
their own purposes, while no more than 15% of the time they spend on the Internet is devoted to educational purposes13.
The technological digital divide is narrowing; it can be expected to become insignificant in the coming years. However, data on the patterns of ninth graders' use of digital technology (Table 5) show that the amount of time they spend watching videos, listening to music, or using various services is weakly related to their ICL levels. The use of digital technology for traditional routine operations (telephony, video, text exchange, etc.) does not lead to a reduction of the new digital divide, that is of the disparity between those who are able to use digital technology to handle information productively and those who use it as a substitution tool. This is also evidenced by the correlation found between the ICL levels of 9th grade students and their activity in office applications and reviewing content of specialised websites (Table 5).
13 These data are consistent with the results of the 2019 HSE study. See Indika-tory informatsionnogo obshchestva, 2019 [Indicators of the information society, 2019]: https://www.hse.ru/primarydata/ice2019
Figure 5. The relationship between average ICL levels in the ten school groups and students' plans for further education as well as their mothers' educational level
Proportion of mothers with higher education (%)
Table 5. The Relationship Between the ICL of 9th Grade Students and the Patterns of their Use of Digital Technology
Variable Schools
All Rural Urban
How much time do you spend on each of the listed activities while using a computer and/or other gadgets:
Reviewing content on specialised portals/sites or in social media feeds 0.09*** 0.09*** 0.09***
(0.01) (0.02) (0.02)
Playing games 0.03** 0.03 0.03*
(0.01) (0.02) (0.02)
Watching videos, listening to music 0.04** 0.07** 0.02
(0.02) (0.03) (0.02)
Working in office applications (notepad, squibs, calendar, Word, Excel, Power Point, Notes, etc.) 0.10*** 0.09*** 0.11***
(0.01) (0.02) (0.01)
How much time do you spend on each of the listed activities while using social media (VK, Facebook, Instagram, Twitter, etc.):
Writing texts, uploading photos, making posts -0.08*** -0.09*** -0.07***
(0.01) (0.02) (0.02)
Reading feeds, blogs, posts 0.04*** 0.08*** 0.03
(0.01) (0.02) (0.02)
Playing games, using social network apps -0.07*** -0.10*** -0.06***
(0.01) (0.02) (0.01)
Significance level: ***p < .01, **p < .05, *p < .10. Standard error (SE) is given in parentheses.
Thus, the results of the study show a strong connection of the ICL of 9th grade students with their out-of-school environment and a weak connection with their work in the school.
Conclusion A monitoring study of the ICL level of 9th grade students, which involved about 30,000 schoolchildren from 21 regions of Russia, showed that more than half of them (54.6%) had insufficient ICL. School leavers whose ICL is 'below basic' (35.3% of those surveyed) may be able to overcome their deficit with appropriate correction and additional training in the next stages of education. Almost a fifth of the ninth graders (19.3%), who are at the 'developing' level, have not actually formed their ICL. They are unable to correctly formulate the problems they face, to search for and organise the necessary information, and to draw reasoned conclusions from it. They are not good at presenting information to other people. Most of those at the 'developing' level are students in rural schools (70%). All ninth graders with a 'developing' ICL need help in developing their ability to work with information in a digital environment. Politicians, education authorities, educators and parents should find ways to offer them additional training and opportunities to master the use of information, digital devices, and educational resources.
In regions where targeted work on ICT in education had been carried out in recent years and the ICL level of schoolchildren has been assessed, the proportion of ninth graders with ICL at the 'developing' level is almost twice as low as the average for the regions covered by the project (21.74%). In Kaliningrad Oblast, for example, it is 12.5% and in Yamalo-Nenets Autonomous Okrug it is 11.5%.
The regions selected for participation in the study were those that were implementing the target model of a digital learning environment in general education institutions, and many of them had been actively working on ICT-use in school for the previous 15 years. It is unlikely that in regions where such work was less intensive, the school impact on the formation of the ICL of 9th grade students would be stronger. Therefore, extrapolation of the study findings to all regions of the country can hardly be made without further research.
In recent years, an increasing variety of digital technologies have become available to schools. 59% of teachers surveyed said that they were incentivised to use digital technology in the classroom. More than 68% of teachers believed that their school had enough digital equipment, like computers and video projectors. However, there are regions where the situation is not so good, for example, in Bashkortostan 49% of teachers do not consider the availability of digital equipment in schools to be satisfactory. Across all regions of the project, a notable proportion of teachers report-
ed a weak digital infrastructure in schools. More than 40% said that they did not have the opportunity to use the computer classroom for their lessons at least once a week. 43% believed that the technical support available to them was not sufficient to maintain the available equipment and make full use of the digital learning environment.
About half of the teachers (52%) responded that their school had stable Internet access with a connection speed of at least 2 Mbps. In some regions, however, the majority of teachers disagreed with that: 66% in Pskov Oblast, 57% in Rostov Oblast, 56% in Krasnoyarsk Oblast, and 71% in the Republic of Bashkortostan. These were primarily teachers in rural schools. These results diverge from some sources14, which report that more than 90% of schools in the country are connected to high-speed Internet.
The real demand from school administrators for the development of digital infrastructure is low. For example, increasing the speed of Internet access in schools has not been given the highest priority: 85% of administrators rate its priority as medium, 10% rate it as low, and 4% do not recognise it as a priority.
A weak connection was found between the level of ICL of students and the availability of digital technology in the school, i.e. the development of its digital learning environment. Thus, increasing the accessibility of digital devices and reducing the technological digital divide alone does not ensure bridging the second digital divide and creating a sound ICL in students. The reason seems to be that schools today focus mainly on reproductive forms of learning practices. Teachers use digital technology primarily as a substitute and/or enhancement for presentations in frontal teaching. Although more than 90% of school administrators declared the importance of the digital learning environment in forming the necessary educational outcomes for students and 78% of teachers noted that they have access to digital technology at school, more than a third of all respondents (39%) indicated that the use of digital technology was not among their priorities.
The results of the monitoring study showed that the ICL level of the ninth graders was strongly related to the characteristics of their out-of-school environment: their mother's educational level, their plans for further education, the availability of a smartphone, and the number of books at home. At the same time, the correlation between the ICL of 9th grade students and their ability to work in the school digital environment is weak.
14 Coordinating Centre of the National Domain of the Internet, National Research University Higher School of Economics. Analytical report Tendentsii razvitiya Interneta [Trends in Internet Development]: https://issek.hse.ru/ data/2018/04/19/1150466651/Tendencii razvitiya interneta v Rossii.pdf
The analysis of the study results makes it possible to set out a number of recommendations aimed at improving the ICL of basic school leavers.
1. The technological digital divide is narrowing, but even today it has not been fully bridged, especially in rural schools. The monitoring study took place in what the local education authorities consider to be reasonably well-equipped schools, but even here sometimes there is no reliable broadband Internet connection. It is necessary to simultaneously:
• fast-track the connection of schools, especially in rural areas, to broadband Internet and provide them with modern equipment, including peripherals;
• offer schools evidence-based, digitally supported, active learning methods that would be accepted by both pupils and their teachers.
2. Formative assessment of students' ICL should be made an integral part of the school' everyday activities. ICL was added to the list of meta-subject outcomes of the Federal State Educational Standards for general education more than a decade ago. Even today, however, there is no development or broad implementation of tools and procedures for assessing the ICL of students at different levels of the mainstream school. Monitoring studies involving low-stakes assessment procedures, which have no negative consequences for students, their teachers and schools, allow students to know their level of ICL and receive targeted advice on its development. Such surveys provide parents, teachers, and educational managers with objective information about students' readiness for further education. The findings can serve as a basis for the elaboration of targeted support measures for schools operating in difficult circumstances and for students from socially disadvantaged families.
3. To ensure that all basic school leavers successfully bridge the second digital divide, more work on digital upgrade of general education is required [Uvarov, Frumin, 2019]. It is necessary to introduce evidence-based organisational forms and methods of teaching and learning related to the use of digital technology in the learning process, which provide, among other things, the formation of students' ICL during individual and group work in class, homework, research and project work.
4. Today, a digital tool is available in our country that allows for a fairly simple and reliable automated assessment of the ICL level of students. The development of the digital learning environment and the connection of schools to the Internet provide
favourable conditions for the broad use of this tool and its derivatives in the mainstream school. Using methods of artificial intelligence and big data, further research and development is needed that will help to provide in real time detailed, individualised, targeted recommendations for each pupil, their parents and teachers on how to improve the ICL levels, prepare briefs for school administrators, education managers and methodo-logists on how to improve the work in schools, and introduce continuous formative assessment of students' ICL into everyday educational work.
Thus, considerable effort will be required to ensure that every basic school leaver possesses the ability to handle information in a digital environment as a prerequisite for successful personal and professional development throughout their life.
The research was supported by a grant provided by the Ministry of Science and Higher Education of the Russian Federation (agreement No. 075-15-2020-928).
References Ainley J., Fraillon J., Freeman C. (2005) National Assessment Program: ICT Literacy Years 6 & 10 Report. Available at: https://eric.ed.gov/?id=ED530360 (accessed 20 January 2022).
Association of College and Research Libraries (1989) A Progress Report on Information Literacy: An Update on the American Library Association Presidential Committee on Information Literacy: Final Report. Available at: https:// www.ala.org/acrl/publications/whitepapers/progressreport (accessed 20 January 2022).
Avdeeva S.M. (2015) Mezhdunarodnoe issledovanie komp'yuternoi i informatsion-noi gramotnosti (ICILS-2010-2015): pervye rezul'taty [International Study of Computer and Information Literacy (ICILS-2010-2015): First Results]. Available at: https://docplayer.ru/32137495-Mezhdunarodnoe-issledovanie-kompy-uternoy-i-informacionnoy-gramotnosti-icils-pervye-rezultaty.html (accessed 20 January 2022).
Avdeeva S.M., Rudnev M.G., Vasin G.M., Tarasova K.V., Panova D.M. (2017) Otsenka informatsionno-kommunikatsionnoy kompetentnosti uchashchikhsya: pod-khody, instrument, validnost' i nadezhnost' rezul'tatov [Assessing Information and Communication Technology Competence of Students: Approaches, Tools, Validity and Reliability of Results]. Voprosy obrazovaniya / Educational Studies Moscow, no 4, pp. 104-132. doi: 10.17323/1814-9545-2017-4-104-132
Carretero S., Vuorikari R., Punie Y. (2017) DigComp 2.1: The Digital Competence Framework for Citizens with Eight Proficiency Levels and Examples of Use. Luxembourg: Publications Office of the EU. doi:10.2760/38842
Eisenberg M.B., Lowe C.A., Spitzer K.L. (2004) Information Literacy: Essential Skills for the Information Age. Westport, CT: Libraries Unlimited.
Ershov A. (1981) Programming, the Second Literacy. Proceedings of the IFIP TC-3 3rd World Conference on Computer in Education (WCCE 81) (Lousanne, Amsterdam, 1981), pt. I, pp. 1-17.
Fraillon J., Ainley J., Schulz W., Duckworth D., Friedman T. (2019) IEA International Computer and Information Literacy Study 2018. Assessment Framework. Cham, Switzerland: Springer Open. doi: 10.1007/978-3-030-19389-8
Gvozdev E.N., Nikulin N.N., Rodnevskaya M.A. (2017) Osobennosti mezhdunar-odnogo sravnitel'nogo issledovaniya ICILS. Pedagogicheskie izmereniya, no 2, pp. 81-85.
International ICT Literacy Panel (2002) Digital Transformation for ICT Literacy. NJ, Princeton: ETS. Available at: https://www.ets.org/research/policy research reports/publications/report/2002/cjik (accessed 20 January 2022).
ITAA (2000) Bridging the Gap: Information Technology Skills for a New Millennium. Arlington, VA: ITAA.
Kim H.-S., Kil H.-J., Shin A. (2014) An Analysis of Variables Affecting the ICT Literacy Level of Korean Elementary School Students. Computers & Education, vol. 77, August, pp. 29-38. doi:10.1016/j.compedu.2014.04.009
Mislevy R., Almond R., Lukas J. (2003) A Brief Introduction to Evidence-Centered Design. CSE Report 632. ETS Research Report Series, no1. doi:10.1002/j.2333-8504.2003.tb01908.x
OECD (2021) Recommendation of the Council on Children in the Digital Environment. Paris: OECD. Available at: https://legalinstruments.oecd.org/en/instru-ments/0ECD-LEGAL-0389%20 (accessed 20 January 2022).
OECD (2015) PISA 2015 Results. Vol. II. Policies and Practices for Successful Schools. Paris: OECD. Available at: https://www.oecd-ilibrary.org/doc-server/9789264267510-en.pdf?expires=1638046143&id=id&accname=guest&-checksum=B4E21 DAE070512D5025132A8FD592B38 (accessed 20 January 2022).
Schleicher A. (2019) PISA 2018. Insights and Interpretations. Paris: OECD. Available at: https://www.oecd.org/pisa/PISA%202018%20Insights%20and%20In-terpretations%20FINAL%20PDF.pdf (accessed 20 January 2022).
Sparks J.R., Katz I.R., Beile P.M. (2016) Assessing Digital Information Literacy in Higher Education: A Review of Existing Frameworks and Assessments with Recommendations for Next-Generation Assessment. ETS Research Report Series, vol. 2016, iss. 2, pp. 1-33. doi: 10.1002/ets2.12118
UNESCO (2018) A Global Framework of Reference on Digital Literacy Skills for Indicator 4.4.2 Information Paper no 51. June 2018. UIS/2018/ICT/IP/51. Available at: http://uis.unesco.org/sites/default/files/documents/ip51-global-frame-work-reference-digital-literacy-skills-2018-en.pdf (accessed 20 January 2022).
Uvarov A.U., Frumin I.D. (eds) (2019) Trudnosti i perspektivy tsifrovoy transformatsiii obrazovaniya [Difficulties and Prospects of Digital Transformation of Education]. M.: HSE.
Virkus S. (2003) Information Literacy in Europe: A Literature Review. Information Research, vol. 8, no 4, Article no 159. Available at: http://informationr.net/ir/8-4/paper159.html (accessed 20 January 2022).
Webber S., Johnston B. (2017) Information Literacy: Conceptions, Context and the Formation of a Discipline. Journal of Information Literacy, vol. 11, no 1, pp. 156183. doi:10.11645/11.1.2205
