CC BY
25Journal of Siberian Federal University. Humanities & Social Sciences 2 (2016 9) 491-499
УДК 159.99 +159.9.07+37.014.544+37.014.542
International Educational Surveys: The Review of School Assessment Surveys
Elena V. Sabelnikovaa and Natalia L. Khmelevab*
aOECD-HSE Partnership Centre, Institute for Statistical Studies and Economics of Knowledge, National Research University "Higher School of Economics" 20 Myasnitskaya Str., Moscow, 101000, Russia bSenior Lecturer, Department of Psychology and Pedagogy, Faculty of Psychology and Pedagogy, Institute of the State Administration 80 Leninskiy prospect, Moscow, 119261, Russia
Received 28.09.2015, received in revised form 06.10.2015, accepted 28.01.2016
This paper is dedicated to the initiation and implementation of large-scale surveys in education field at different educational and age levels. In order to determine the feasibility of such surveys and their importance for the subsequent analysis of the obtained data, the authors reviewed the concept of "evaluation" and "assessment." The analysis of existing researches allowed to submit a classification of large-scale surveys on the basis of which different school surveys and their implementation in the Russian Federation were reviewed (PISA, PIRLS, TIMMS). The design features of such surveys are not only the data collection of the test results but also the collection of the context information. Moreover such results give an oppurtunity to look at the data on the international level, the successfulness of the country education policy, the population capabilities, the gaps in education process and activities and the competencies distribution among the population. The competencies here mean a person's ability to apply his/her knowledge in a variety of (new, unknown or familiar) contexts and real-life situations. The implementation of competency-based approach in forming the national education policy, the necessity of competencies forming and development are very important components for full inclusion in education sector at the national and international levels.
Keywords: educational policy, education, competencies, international research projects, evaluation, assessment, PISA, PIRLS, TIMMS, TALIS, AHELO, PIAAC.
DOI: 10.17516/1997-1370-2016-9-2-491-499.
Research area: pedagogy.
Introduction growth (Pliskevich, 2012; Korchain). This goal We can see today that there is a trend can be reached with the help of education sector
in many countries to focus on human capital quality development because education is the
evolution in order to accommodate economic main field of a country future development. The
© Siberian Federal University. All rights reserved
* Corresponding author E-mail address: esabelnikova@hse.ru, nlh2009@yandex.ru
Russian Federation follows these global trends of education quality improving. The best way to collect multidimensional system information about different educational systems all over the world is the implementation of international large-scale surveys of education quality assessment. The analysis of such data gives an opportunity to researchers, professionals and politicians of assessing strengths and weaknesses of national education systems, looking through the best practices in educational policy and systems of different countries which play the leading role according to the surveys results. This means that they can use surveys data for improving their national education systems.
The aim of this article is to present an overview of the most important large-scale international surveys in the sphere of school education and to show the importance of their implementation for education quality improvement.
We found out that a problem exists in Russian language connecting with differentiating of two concepts which should be distinguished: assessment surveys and evaluation. Russian researchers use the concepts "assessment" and "evaluation" as a calque from English. This means that these meanings are very close to each other or equivocal. Besides there are many other words for describing estimation procedures, for example, appraisal and measurement. Another aim of the article is to present only the assessment surveys based on the terminology analysis.
Evaluation is an integrative category of evaluative and analytical activities in different surveys of education quality management (Zvonnikov, 2012). This term first appeared in Russian scientific literature in 2011 in the book "Pedagogical Education in University: Context-Biographical Approach". The authors analysed the interpretation of the concept "evaluation" on the basis of German education researchers' understanding (Gavrikov, at al., 2001). The
concept of evaluation becomes very popular in Europe in the late 60's - early 70-ies. According to V. Zvonnikov, the modern interpretation of the concept includes a wide range of theoretical, methodological and practical results of systematic researchers' analysis of learning outcomes and educational process quality (Zvonnikov, 2009). Analysis is rooted in consistent methodology, combines quantitative and qualitative methods, investigates character and dynamics of changes.
The concept "assessment" is often used in the field of human resources in Russia. The meaning includes multidimensional comprehensive standardized assessment (Ksenofontova, 2007; Sergienko at al., 2006). In this case the company human resources assessments are conducted, during which different mutually complementary methods are used in order to assess the object from different perspectives. The assessment result is multidimensional picture of assessed object including different contextual characteristics and factors. Direct measurement of participants' behaviour and activities is the obligatory part of assessment. It can include different kinds of measurement: testing, interviewing, questionnaire, professional simulation. Assessment results are presented in quantitative form using different metric and non-metric scales. Mathematical and statistical methods and probabilistic forecasting can be used on the basis of these scales. The principles of assessment (criteria unambiguity, equal opportunities, independence and complexity) make the results objective. Validity of each method, reliability of the results and representativeness of a sample are the main points of analyzing and comparing reliable results.
In this paper the authors represent leading international large-scale surveys which were implemented as the assessment surveys and were developed on the basis of comprehensive methodological approach including respondents'
testing and contextual information collecting. These parts give an opportunity to make deeper analysis.
The research question is how the school assessment research surveys contribute to development and improvement of education quality?
Surveys in the sphere of education
The leading international surveys can be divided into two categories according to the criterion "level of education":
1. The first level covers age period of preschool and school education and includes: PIRLS (TIMMS and PIRLS Home, 2014), TIMMS (TIMMS and PIRLS Home, 2014; IEA: Home, 2014), PISA (PISA, 2014; OECD Data, 2014).
2. The second level covers the whole period after graduation from the school including so-called lifelong learning: AHELO (Testing students and university globally, 2014) and PIAAC (PIAAC, 2014).
All these surveys are developed and implemented by large international organisations. The Organization for Economic Cooperation and Development (OECD) initiated and organized such surveys as PISA, AHELO, PIAAC. PIRLS and TIMMS are conducted by the International Association for the Evaluation of Educational Achievements (IEA). A part of PIAAC (Education and Skills Online) was implemented in cooperation of these two organizations. Surveys databases are publicly available on the websites of the concerned organizations.
The OECD provides "competency" or "skills" policy in the sphere of education. In the first decade of the XXI century the OECD has developed a set of documents under «OECD Skills Strategy» (Towards an OECD Skills Strategy, 2011). These documents indicate the goals and priorities of modern education, substantiate
the transition from knowledge paradigm to competency.
In this article we will discuss first level surveys.
One of the key areas of modern educational policy is necessary and sufficient skills because they guarantee the successfulness and competitiveness of a modern man. Knowledge and skills enable a person to act successfully or perform specific or global tasks in variety of contexts (new or familiar). These competencies are the prerequisites of improving social involvement of every citizen and the economic growth of the country (Towards an OECD Skills Strategy, 2011). The difference between the new competency approach and previous paradigm oriented to knowledge lies in the postulate that a modern man needs not only isolated knowledge but the ability to apply it. The fact that modern world is full of information means that actuality of knowledge becomes out-of-date very fast and is no longer of an absolute value. Different devices and technologies can help you to find the needful information very easily but a man who has knowledge but not competencies (the ability to apply the necessary knowledge in a certain context or in certain situations) can become noncompetitive.
The above-mentioned surveys implement competency policy.
Progress in International Reading Literacy Study, PIRLS
International survey "Progress in International Reading Literacy Study" (PIRLS) (TIMMS and PIRLS Home, 2014) is a monitoring survey. The main goal is to assess the level and the quality of students' reading literacy at the end of primary school in different countries (4th grade in Russia).
Two types of reading were assessed according to the purpose of use. Students use these types
during classes and in out-of-school activities more often than other activities. They are:
- Reading helps to acquire reader's literary experience;
- Reading helps to acquire and use information.
These types also include "reading to acquire reader's literary experience", "reading to acquire and use of information". In this survey such reading skills as finding information given in an explicit form, drawing conclusions, interpretation and synthesis of information, analysis of content, language features and text structure are assessed.
The texts with text tasks for students, the questionnaires for four groups (students, teachers, parents, administration) were used to collect the needful information. These four groups are very important for the process of teaching reading. The test consists of literary and information texts with 12-16 items which should be completed in 80 minutes.
The students' questionnaire includes questions about school, family, family resources, parents-children communication, reading lessons. The teacher's questionnaire includes questions about teacher (demographic characteristics), school, aspects of teacher's work including questions about the process and methods of teaching reading. The parents' questionnaire includes questions about child's educational path and educational experience, parents-children communication, the role of reading in family, some parents' characteristics. The administration questionnaire includes questions about school characteristics and school environment, parents' participation in school life and school activities, peculiarities of teaching reading.
The first cycle of the research was implemented in 2001. It is held now every five years. In 2011 48 countries took part in the research including Russia. The Russian Federation takes part in
every cycle since the research has begun. Besides in 2011 OECD conducted an additional research (prePIRLS) to measure the level and the quality of reading and text comprehension in countries of low literacy. In 2016 PIRLS will also include PIRLS Literacy and ePIRLS. PIRLS literacy is easier than PIRLS because it has been designed for assessing foundational reading skills. The new part of PIRLS is ePIRLS which simplifies the participation procedure and analysis for other countries because it assesses reading online.
The Centre of Education Quality Assessment of the Russian Academy of Education implements this survey in the Russian Federation (Centre of Education Quality Assessment, 2014). According to G.S. Kovaleva at al., 4461 primary school graduates from 220 educational institutions from 42 regions took part in PIRLS-2011 (Kovaleva, 2013). The sample is general population of Russian primary school graduates.
Such survey results allow to compare reading literacy of primary school graduates from different countries, to see national dynamics and distribution of reading literacy levels, to find differences in national education systems. We can find a link between reading literacy and other different factors (reading interest, family support, family reading attitudes, teaching reading, methods of teaching reading) with the help of data analysis.
Trends in International Mathematics and Science Study, TIMMS
Trends in International Mathematics and Science Study (TIMMS) (TIMMS and PIRLS Home, 2014) is the longest survey. The first four-year cycle was conducted in 1995. The aim of this survey is to assess students' educational achievements in 4th and 8th grades in mathematics and science.
The tools include students' test and questionnaire. The test consists of four blocks of
items: two blocks of items in mathematics and two blocks of items in science with 44-50 items for 4th grade students and with 55-60 items for 8th grade students.
The questionnaire collects the contextual information about different factors influencing on education process. The students' questionnaire includes questions about home life, school life, basic demographic information, home environment characteristics, school environment, self-perception, attitudes towards learning mathematics and science. The parents' questionnaire (4th grade students) includes questions about primary school preparation, reading or mathematical activity before entering a school (reading books, listening songs, playing with toys) as well as information of household resources, parents' educational level and employment status. The teachers' questionnaire includes questions about their professional education and development, teaching practice, mathematics and science curriculum, methods of teaching and study guides. The administration questionnaire includes questions about students' demographic characteristics, availability of resources, types of educational programmes and school learning environment. In addition the national coordinator fills in the questionnaire about curriculum content and teaching mathematics and science.
63 countries took part in the research in 2011 including the Russian Federation which has been taking part in this project since its development. Besides the additional part of survey TIMMS Advanced was implemented. It assessed high-level achievements of final-year students in mathematics and physics. By this time two cycles have been implemented (1995 and 2008). The next round is scheduled for 2015.
Centre of Education Quality Assessment of the Russian Academy of Education carries out
the project in the Russian Federation (Centre of Education Quality Assessment, 2014). According to the group of authors under the leadership of G.S. Kovaleva 412 schools from fifty Russian regions took part in TIMSS-2011 (Demidova, 2013).
The results of the survey give a wide range of findings in different spheres, for example, educational context, students' background and attitudes toward mathematics and science, pedagogical education, teachers' experience, learning environment characteristics, learning activities characteristics, school environment. All these components facilitate students' educational progress in mathematics and science. The survey implementation allows to assess the state of education in mathematics and science through the international educational standards, to compare the achievements of different education systems, to define the factors of students' high level performance, to formulate educational prospects in research areas.
Programme for International Student Assessment, PISA
Programme for International Student Assessment, PISA (PISA, 2014) is a long-term assessment survey aimed to assess compulsory education quality and its development trends. This survey focuses on competencies assessment such as reading, mathematics and science.
The main idea is to assess competencies which is understood as the abilities to apply knowledge in real-life situations. Test items for 15-years-old students do not directly connect with the school curriculum. They were developed to evaluate students' abilities to apply their own knowledge in unknown and real-life situations at the end of compulsory education in order to provide students' readiness for participation in social life. Besides the questionnaires help
to collect context information which is very important for data analysis.
This survey consists of 3-years cycles for 15-year-old students in three fields: reading, mathematics and science, focused on one of these subjects during the test (two thirds of testing time). The main research area in 2000 and 2009 was reading, 2003 and 2012 - science. Besides there was an additional computer testing in 2012 which included new type of interactive tasks on reading and mathematics so as complex problem solving and financial literacy. In 2015 the survey focuses on science.
The procedure of testing takes two hours and consists of open questions (brief or full answer) and multiple choice questions (with one or several right answers). All tasks are united by the field. The total number of tasks is from 52 to 60. Each of the tasks consists of one to four subitems.
The participants (students and management) fill in background questionnaire which collects the information about educational and cultural levels of participants' families, participants' education background, schools, teaching process, school system and learning environment.
79 countries take part in this research including the Russian Federation (from 2000). In 2012 the fifth cycle has been realized in 64 countries. According to the group of authors under the leadership of Galina Kovaleva 5219 students from 227 educational institutions from 42 Russian regions took part in PISA-2012 (PISA-2012 Main Results, 2013). The sample consisted of 15-year-old students from different types of schools (secondary schools, vocational education schools, secondary vocational education schools).
This survey gives an opportunity to collect systematic data of education quality of a country, to assess the administrative decisions contribution to educational policy, to improve organizational and economic mechanisms of
education management. The results of this survey allow to analyse the effectiveness of appliance of knowledge and skills studied at the school by 15-year-old students and to compare students' learning outcomes.
To improve the Russian Federation education policy politicians and researchers can use other countries experience which have the highest positions by the results of this survey. It makes possible the development of the national system of education quality assessment, different monitoring studies in the field of education.
Conclusion
Looking through all above-mentioned surveys it can be seen that all of them assess the competencies in different fields, i.e. person's opportunity of knowledge applying in different context and situations (new or familiar).
Nowadays there is a trend in international community to diversify country economy and education with the help of human capital development. In comparison with economy based on natural resources this trend leads to use population potential and reduces the importance of natural resources. Some countries implement skills development strategies along with the development of science and technology fields. Competencies development becomes the main trend of education policy in many countries which is implemented in all levels of education (pre-school education, primary and secondary school, higher education, adult education, lifelong learning). In these surveys not only the competencies have been assessed but also different factors of educational environment. Such factors facilitate or obstruct building competencies. The results of such surveys highlight the importance of literacy in the modern world as well as the importance of knowledge-based competencies for economic development, individuals and society as a whole (Brown at al., 2005).
International surveys attract international community attention to comparative competencies assessment. The outcomes allow to compare education systems and results of different countries which lead to identification of the problem areas in education field and make policy decisions more relevant and effective for improving outcomes on the national and international levels. The most important thing is understanding of the necessary skills and demands for country economic development. Many factors such as the level of country industrialization, promising sectors, country openness for the migration flows and methods of labour market control influence on this demands very much. This fact is very important for the national labour markets.
For the present moment the focus on skills and competencies development is one of the
main global trends which governs the goals of countries policy of developing necessary competencies.
To sum it up the international assessment surveys contribute to improve quality of education. The focus of the surveys is skills and competencies, i.e. a person ability to apply knowledge in different familiar or new contexts. The formation of such competencies is the main goal of international education policy on all education stages (primary school, secondary school, higher education, adult education, lifelong learning). Assessment surveys give the opportunity to analyse context data as well as detailed information of competencies distribution of different groups of population and different factors which influence on competencies formation. International surveys collect multidimensional information for political decisions based on evidence information.
References
Brown, G., Micklewright, J., Schnepf, S.V., Waldmann, R. (2005). Cross-National Surveys of Learning Achievement: How Robust are the Findings? : Discussion Paper Series. Bonn: The Institute for the Study of Labo. 37 p.
Demidova, M.Iu. (2013). Osnovnye resultaty mezhdunarodnogo issledovaniia kachestva matematicheskogo i estestvennonauchnogo obrazovaniia TIMMS-2011: analiticheskii otchet [Main Results of International Mathematics and Science Study TIMMS-2011: Analytical Report]. Moscow: MAKS Press, 2013. 154 p.
Gavrikov, A.L., Pevzner, M.N. (2001). Pedagogicheskoe obrazovanie v universitete: kontekstno-biograficheskii podkhod: monografiia [Pedagogical Education in University: Context-Biographical Approach: Monograph]. Veliky Novgorod: NovGU named after Iaroslav the Wise. 30 p.
IEA: Home (2014), Available at: http://www.iea.nl/ (accessed 15 October 2014).
Korchagin, Iu.A. Effektivnost' i kachestvo natsional'nykh chelovecheskikh kapitalov stran mira [Efficiency and Quality of National Human Capital in Different Countries], «Regional Economics' problems», V.38, Available at: http://www.lerc.ru/?part=bulletin&art=38&page=1 (accessed 15 October 2014).
Kovaleva, G.S. (2013). Osnovnye resultaty mezhdunarodnogo issledovaniia chitatel'skoi gramotnosti PIRLS-2011: analiticheskii otchet [Main Results of International Reading Literacy Study PIRLS-2011: Analytical Report]. Moscow: MAKS Press. 132 p.
Ksenofontova, E.G. (2007) Assessment-sentr chto eto? [Assessment-centre. What Does It Mean?]. Available at: http://istina.msu.ru/publications/article/2981651/ (accessed 15 October 2014).
OECD Data (2014), Available at: http://data.oecd.org/ (accessed 17 December 2014).
PIAAC (2014), Available at: http://www.oecd.org/site/piaac/ (accessed 05 November 2014).
PISA (2014), Available at: http://www.oecd.org/pisa/ (accessed 22 October 2014).
PISA-2012 Osnovnye rezultaty (2013) [PISA-2012 Main Results], Available at: http://www. centeroko.ru/public.htm#pisa_pub (accessed 13 October 2014).
Pliskevich, N.M. (2012). Chelovecheskii kapital v transformiruyushcheisia Rossii [Human capital in transforming Russia]. Moscow: Institute of Economics RAS. 231 p.
Sergienko, S.K. (2006). Rol' assessment-tsentrov v razvitii kapitala organizatsii [The Role of Assessment-Centre in the Development of Organization Staff], HR Manager. HR Management. № 12. P. 50-54.
Testing student and university performance globally (2014), Available at: http://www.oecd.org/ edu/skills-beyond-school/testingstudentanduniversityperformancegloballyoecdsahelo.htm (accessed 02 November 2014).
TIMSS and PIRLS Home (2014), Available at: http://timssandpirls.bc.edu/ (accessed 22 October 2014).
Towards an OECD Skills Strategy. OECD, 2011. 44 p.
Tsenter otsenki kachestva obrazovaniia [Centre of Education Quality Assessment] (2014), Available at: http://centeroko.ru/ (accessed 12 October 2014).
Zvonnikov, V.I. (2009). Kontrol' kachestva obucheniiapri attestatsii: kompetentnostnyi podkhod [Control of the Quality of Teaching During the Appraisal: Competence Approach]. Moscow: University Book, Logos. 272 p.
Zvonnikov, V.I. (2012). Otsenka kachestva rezul 'tatov obucheniia pri attestatsii (kompetentnostnyi podkhod): ucheb. posobie [Assessment of the Quality of Learning Outcomes During the Appraisal (Competence Approach): learning guide]. Moscow: Logos. 280 p.
Международные образовательные проекты: обзор школьных ассессмент-проектов
Е.В. Сабельниковаа, Н.Л. Хмелева6
аНациональный исследовательский университет «Высшая школа экономики» Россия, 101000, Москва, ул. Мясницкая, 20 бИнститут государственного администрирования Россия, 119261, Москва, Ленинский пр., 80
Данная статья посвящена вопросу инициации и реализации крупномасштабных проектов в области образования на разных образовательных и возрастных ступенях. С целью определения возможности реализации оценки и ее важности для последующего анализа полученных данных авторами рассматриваются понятия «эвалюация» и «ассессмент». В результате проведенного анализа существующих исследований авторы предлагают свою классификацию крупномасштабных проектов, на основе которой рассматриваются проекты в области школьного образования (PISA, PIRLS, TIMMS) и их имплементация в России. Особенность таких проектов - сбор не только информации об успешности прохождения испытаний, но и сбор контекстной информации. Кроме того, результаты позволяют рассмотреть на международном уровне успешность реализации образовательной политики в стране, возможности ее населения, определенные лакуны в осуществлении образования и распределение компетенций населения. Компетенции понимаются как способность человека применять полученные или имеющиеся знания в различных (новых или уже известных) контекстах и жизненных ситуациях. Реализация компетентностного подхода в формировании национальной образовательной политики, необходимость формирования и развития компетенций являются важными компонентами для полноценного включения в образовательный сектор на национальном и международном уровнях.
Ключевые слова: образовательная политика, образование, компетенции, международные исследовательские проекты, эвалюация, ассессмент, PISA, PIRLS, TIMMS, TALIS, AHELO, PIAAC.
Научная специальность: 13.00.00 - педагогические науки.
