CC BY-NC-ND
52
The First Assessment of Adult Competencies in Russia
O. Podolskiy, D. Popov
Oleg Podolskiy
Ph.D., Candidate of Sciences, PIAAC National Coordinator in the Russian Federation; Research Fellow, Institute of Education, National Research University— Higher School of Economics. E-mail: opodolskiy@hse.ru Dmitry Popov
Candidate of Sciences, Senior Researcher, Institute of Education, National Research University—Higher School of Economics. E-mail: dpopov@hse.ru
Address: 20, Myasnitskaya str., Moscow, 101000, Russian Federation
Abstract. This paper opens a series of publications based on the Program for the International Assessment of Adult Competencies (PIAAC) data. Among the many tasks PIAAC is aimed on the identification and measurement of differences between citizens and countries in the area of key competencies, evaluation of the impact of skills on individual economic and social achievement, the effectiveness of various national educational and training systems in providing necessary skills as well as the creation of the conditions for lifelong learning. PIAAC draws the first “profile” and pathbreaking results of Russian adults' key competencies on literacy, numeracy and problem solving in technology-rich environment. The central part of the paper is dedicated to a general analysis of the role of competencies and competence-based approach, a description of the research method, sampling and tools used (including test samples) in
the PIAAC, and particularly in Russian Federation. Finally, the paper deals with the first results of the adult competencies research in Russian Federation. The results on identified key competencies are presented in accordance with various age groups, and include gender and qualification level differences. PIAAC is a long-term research project. Russia participated in the programme in 20092013. Over 5000 adults aged between 16 and 25 from 25 regions and 94 localities participated in the programme in Russia. The total sample included 3800 respondents; it is representative for the whole country except Moscow and Moscow Oblast. The average points Russia obtained in reading and mathematical literacy are comparable to those obtained in OECD countries. Nearly half of working-age adults in Russia either don't have any computer experience or are incompetent users. As compared to OECD citizens, computer skills of Russian adults are more often restricted to doing only simple tasks (like logging in and out of their email boxes). As the level of education grows, so does the level of competencies of adult population in Russia. Similarly, competencies seem to accumulate with ageing, until the peak age of 45-49 years is reached. The discussion of the first PIAAC results and brief statement of research questions for future studies are presented. Keywords: Programme for the International Assessment of Adult Competencies, Russia, adult competencies, literacy, numeracy, problem-solving, lifelong learning.
Received in December 2013
http://vo.hse.ru
1
THEORETICAL AND APPLIED RESEARCH
The onrush of information and communication technologies started in the past century is responsible for the increasing role of knowledge and skills at work and in everyday life. A modern-day human needs a solid set of basic skills to get integrated in the society and deal with social institutions at an appropriate level. Good command of language, proficiency in numeracy, and ICT skills comprise the indispensable minimum that one needs to know. The basic skills specified above also work as efficient indicators of the level of society development allowing to evaluate its potential.
This simple truth came to people’s minds as early as in the 1960s (see, for example, [Machlup, 1962]), but it was only in the 1990s when the first major projects designed to assess competencies and knowledge were launched. The Programme for International Student Assessment (PISA) has become one of the most well-known and recognized of these projects and, still maintaining its position today. Competency approach to assessing knowledge and skills allowing to measure contextualized cognitive abilities became widespread largely due to this Programme. However, the PISA is designed to test knowledge of 15-year-old school students and to assess performance of national education systems. The globalized labor market developing, inter alia, through mass migration, stays out of the focus. Besides, the PISA specifics is that the test does not provide answers to how knowledge and skills change throughout life, how labor and family affect development of competencies, or how unequal skills and opportunities for development promote and aggravate social inequality.
The first international projects initiated to explore adult compe-tencies—International Adult Literacy Survey (IALS) [OECD and Statistics Canada, 1995; OECD, 2000]) and the Adult Literacy and Life skills survey (ALL) [Murray, Clermont, Binkley, 2005]) —had limited coverage and toolset. Nevertheless, their results were blatantly alarming: from 1/3 to 2/3 of respondents, depending on the country, didn’t even get to the minimal acceptable level 3 (out of 5) when doing reading tasks. Besides, the surveys demonstrated that people with insufficient reading skills couldn’t use Internet and computer effectively. Most countries revealed a wide gap between the most educated and the rest of the population [Podolskiy, Popov, 2012] .
The IALS and ALL experience and results provided the foundation for another research project, the Programme for the International Assessment of Adult Competencies (PIAAC). This project went truly global, with 24 countries participating in the first round: Australia, Austria, Belgium (Flanders), Canada, the Czech Republic, Cyprus, Denmark, Estonia, Finland, France, Germany, Ireland, Italy, Japan, the Netherlands, Norway, Poland, the Russian Federation, the Slovak Republic, South Korea, Spain, Sweden, the United Kingdom, and the United States. Over 157,000 respondents were surveyed in total. This was the first mass-scale survey of adult skills that involved the Russian Federation, where the project was carried out with the as-
2
Educational Studies. 2014. No 2
O. Podolskiy, D. Popov Г ■ The First Assessment of Adult Competencies in Russia
sistance of the National Research University-Higher School of Economics. First results of the survey were made available in 2013.
Nine additional countries are currently collecting data: Chile, Greece, Indonesia, Israel, Lithuania, New Zealand, Singapore, Slovenia, and Turkey. The data on these countries will be available in 2016.
The PIAAC is a long-term survey project, in which Russia participated in 2009-2013. The data collected by the survey allow an effective evaluation of human capital and social potential on the national and international scales.
Most generally, the Programme for the International Assessment of Adult Competencies is designed to give insights into the key adult competencies and in how they affect success and self-fulfillment at work and at home in information-rich and technology-rich environments.
Among objectives of PIAAC, the most important are as follows:
• Identify and measure differences in the key competencies among people and countries;
• See how competencies affect individual socioeconomic attainments;
• Evaluate performance of different national education systems in developing skills that are needed;
• Determine factors that promote improvement of market positions for different categories of adult population and providing conditions for lifelong learning;
• Promote development of mechanisms to ensure improvement of competencies at workplace or as part of the existing education system modernization, as gaps in the key skills are detected.
Competencies are generally understood as certain results of applying human knowledge, skills and experiences in practice. European, American and Australian researchers have conducted a number of surveys since the 1970s to assess basic knowledge and skills, as well as professional competencies, and to collect information on participation of adult population in working and daily activities.
Competency-based approach to assessing knowledge and skills has been most demanded in modern business and industry. Research built on the competency-based model of human activity suggests that performance at work depends on proficiency level in the key competencies [Davis, Fisher, Ford, 2009]. The key, or basic, competencies are indispensable for taking part in the technical advancement, global computerization, and internetization of each professional field [Saavedra, Saavedra, 2011] .
The role the key competencies play in today’s world excited attention of fundamental researchers and practitioners in economics and education. Ample empirical data points to an ever lower level of basic knowledge and skills among young people, their key graduate com-
1. Literacy as the key PIAAC competency
http://vo.hse.ru
3
THEORETICAL AND APPLIED RESEARCH
petencies being unable to meet the existing requirements. In fact, a number of studies revealed a decrease in adult literacy, as well lower quality and quantity of reading, and poor writing quality [Fallahi et al., 2006]. Employers point out that university graduates are often not prepared to work, as their key competencies are inadequate despite possession of essential skills required for challenging professional activities [Brown et al., 2008; Davis, Fisher, Ford, 2009; Saavedra, Saavedra, 2011].
As Moser states, millions of young people with poor reading and writing skills graduate from schools every year. 20 percent of British adult population have difficulties using basic skills and handling information correctly, i. e. they lack functional literacy required for working and daily activities [Moser, 1999] .
The OECD has provided a new approach towards understanding of the key competencies. Today, what one needs is not just knowledge, because knowledge is extremely diverse, depends on the context, and becomes obsolete very quickly. In the 21st century, one needs to be able to deal with various data, searching for relevant information, structuring it, and using search results to solve real-life problems [OECD, 2005]. Knowledge as such will not guarantee successful performance in today’s world; that’s why assessing one’s background knowledge is not enough to estimate their preparedness for working and daily activities. This statement was used as a theoretical basis for conception of the Programme for the International Assessment of Adult Competencies. PIAAC extends the notion of “literacy” beyond reading, writing and number skills only, adding also a set of knowledge and skills that enable people to succeed in activities which become ever more technological. The Programme is focused on activity as ability to apply one’s knowledge, skills and competencies. This is the key difference of the PIAAC from the “knowledge testing” projects.
Literary sources often discriminate between “competency” and “skill”. Competency is regarded as an ability that may manifest itself in a rather wide range of real contexts, while skill is considered to be a basic unit of competency, i. e. a unit of ability (not infrequently technical) associated with a specific context. The OECD methodology defines competency as a “coherent set of skills, attitudes and knowledge that are specific for the context” [OECD, 2007]. PIAAC uses the terms “competency” and “skill” synonymously to refer to ability to act in accordance with a given situation, to use information properly and appropriately. Both competency and skill suggest use of expertise, tools, and cognitive knowledge, and involve certain values, interests, and implications. Skills (or competencies) may be broken into more specific and measurable skills (or competencies) or merge into larger, more generic ones.
The PIAAC competency assessment method aims not to measure skills precisely (e. g. vocabulary or learned mathematical operations), but to evaluate ability to react quickly in a given context and to
4
Educational Studies. 2014. No 2
O. Podolskiy, D. Popov Г ■ The First Assessment of Adult Competencies in Russia
process acquired information correctly in changing conditions. In order to achieve loads of everyday goals, one should be skilled in responding to various real-life situations.
Competencies assessed in PIAAC are required to understand, analyze and use information in real-life situations. These skills are referred to as key information-processing competencies, as they are:
• indispensable for fully integrating and participating in the labor market, education and training, and social and civic life;
• relevant to adults in many social contexts and work situations;
• developed under the influence of socioeconomic factors;
• typical of all adults.
Literacy and numeracy form the basis for development of higher-level cognitive skills, such as analytical or critical thinking, and constitute an essential condition for access to specific fields of knowledge and understanding of complex information. These two skills are associated with a wide variety of real life contexts, from working activities to situations everyone encounters daily.
Ability to read and process information is vital, whether you deal with prescribing information that contains contraindications, or an email from your colleague, or a special website with school admission requirements. Today, problem solving in technology-rich environments provides access to the global information system, allowing to assess, analyze and share information, particularly with the help of high-tech electronic devices or applications.
Information literacy in today’s society is the key competency which is explored in the PIAAC project.
PIAAC main research instruments include a specifically designed questionnaire and test tasks assessing skills in three domains: literacy, numeracy, and problem solving in technology-rich environments.
Using a questionnaire along with tests allows for integrating a number of contextual variables into analysis and for identifying the relationship between competencies and such fundamental characteristics as respondent’s social background, economic status, educational attainments, and social status. Questionnaire data is used to describe individual and institutional mechanisms affecting changes in respondent’s competencies and to assess respondent’s performance (efficiency) in the context of their social status and success in the labor market. The questionnaire covers aspects of respondent’s life, like adult learning opportunities, issues associated with transition from study to work, success (or bad luck) in the labor market, specifics of social involvement, and health.
PIAAC tests assess skills in three domains: numeracy, literacy, and problem solving in technology-rich environments. These com-
2. PIAAC
instruments
http://vo.hse.ru
5
THEORETICAL AND APPLIED RESEARCH
2.1. Assessment of proficiency in literacy and numeracy
petencies were analyzed using a specifically developed computer system. Respondents who were incompetent PC users received the tests in special printed leaflets.
Part of PIAAC literacy and numeracy tasks was taken from the IALS and ALL which had been carried out earlier. The few countries that had participated in the preceding surveys of adult competencies had a chance to analyze changes the society had undergone since then. The Russian Federation will have a chance like that when the next round of PIAAC is completed.
Reading and interpretation skills are assessed at several levels, from the basic level (vocabulary) to the advanced level evaluating integrated knowledge, ability to interpret, summarize and compare information. The survey used texts of various difficulty, which allowed to assess skills of respondents with any level of literacy proficiency with equal efficiency [OECD, 2009a]. The tests were provided in the form of conventional and interactive texts (Figure 1) .
Searching for information, its understanding, summarizing and interpretation are the key elements of literacy assessment on which all test tasks are built.
Searching and understanding. Reader’s main purpose is to reconstruct meaning, whether explicit, expressed verbally, or implicit. Understanding of text is assessed at different levels. Thus, basic literacy skills are assessed through tasks related to understanding of individual words or fragments or searching for required information. For instance, one reads through a recipe and has to tell how many tablespoons flour is needed for a cake. Tasks of higher levels test ability to extract required information from expatiative descriptions, to collect information from different parts of a given text. The tasks assess how respondents understand the content of the text and the specific subject which is often conveyed through extensive narration or detailed argumentation. Readers should also understand the context in each task and how this context affects structure and content of a given text.
Summarizing and interpreting. These skills are assessed through, for example, asking respondents to determine whether a given text is suitable for solving a specific problem. Interpretation tasks suggest that respondent should find relationship between different parts or fragments of the text and decide what is similar or different between them. Overall understanding is assessed through asking respondents to identify the purpose or the main idea of the text. In a number of tasks, respondents are supposed to rely upon knowledge that is external to the text itself. The reader should evaluate relevance of suggested information to the problem specified, as well as soundness of information and arguments contained in the text.
All PIAAC tasks are designed in such a way that ability to use acquired information is tested in nearly real-life conditions, and the information is applied to solve most diverse everyday tasks. Sometimes
6
Educational Studies. 2014. No 2
O. Podolskiy, D. Popov Г ■ The First Assessment of Adult Competencies in Russia
Figure 1. Sample literacy skills assessment task
Physical Exercise Equipment
Howto cfioose?
A Decide what effect you want the exercise to v have on your body
E
Assess the space you have available at huine.
Cheese the equipment that suits your nhjertivfta If пепяяяягу ask я spariaist fnr
OBJECTIVE Bum off calories STfUTEGV Cardiovasaifer exercises EQUIPMENT Reeving machete. Bicycie. Skarachine. TrsadmS, Stars,...
Strengthen your muscles Endurance exercises Bench for Press-ups. Weights and Dumbbells, Etastc Tubes
Effects Cardlo-Tralnlng Muscle Building
Exercise Injyuh: Kowing ■ішінс Stepper 1 read-nd Ah Lhimtb Ub, Elastic Gym tnaiuh Wuscte- LruOifiiiy Multi- fib ab AB іивеї
@ Ы @1 Ш Й Ш S@ 1 №1 № №
lufIT- •OuuJ Ara-aye ІікГГ- OuuJ "As у good May Зомі Ouud Ouud OuuJ Mil у good Ouud Ouud
Lee strength Good Vfery good Average Vay good Good Ineff- ective Good Average Good Good Ineff- ective Good Good
AUdu- ІГііКГІКі May OuuJ Ouul IlKff- C Drive Ouud '■Any good Ouud Avw-^e Мяу good "Am у good Мяу good
Over-аї muscle binding Ineff- ective Very good Ineff- ective Average Ineff- ective Average Good Good Good Average Good Good Good
Heart1 artenea tey goa Good Very good Very good Good Ineff- ective Average Average Average Good Average Average Aver- age
t*ty ІнеГГ- OuuJ ІІкГГ- ІікГГ- Aveiaye A If Til dye Ouul IpkET- Ineff- A If Ы dye Ouud OuuJ
.feints fVrti \Ary good fVwt СгґЛ lVrt-1 Avfengp fewl iVrtH iv™°" age
Sfrn- mhg Goad Average Vtery good Good Good Ineff- ective Average Good Average Average Good Good Good
Dangers None Back None Legs Ills beat то learn to use these types of apparatus property before you make a m$or effort
Note. Task of difficulty level 2, serving to measure ability to search for and understand information. In this case, respondent should tell which equipment from the list has the “No effect” value most often. Normally, each task goes with a few questions.
understanding alone is enough, but there are times when information cannot be extracted without analytical skills or structural processing of subjects and tasks.
Numeracy in PIAAC is defined as “the ability to access, use, interpret, and communicate mathematical information in a range of situations in adult life [OECD, 2009b]. Just as with literacy, the question here is not about simply having some mathematical knowledge, but about the ability to apply it in different contexts, with different purposes.
Mathematical information in PIAAC is delivered to respondents in various forms (Figure 2). On the basic level, there may be a set of concrete objects to be counted (people, buildings, machines). Mathematical data may be presented graphically, as diagrams or graphs. Besides, respondents are given materials, like city map or project plan, which have to be measured or evaluated in the task. Mathe-
2.2. Assessment of proficiency in numeracy
http://vo.hse.ru
7
THEORETICAL AND APPLIED RESEARCH
Figure 2. Numeracy assessment sample task
If temperature drops by 30°C, what temperature in °C will be indicated on the thermometer?
Note. Numeracy sample task of difficulty level 1.
matical information may be communicated through formulas, but text containing mathematical information or affecting its understanding is of no less importance. Text may include numbers written in words (“five” instead of “5”), mention basic mathematical notions (“fraction”, “multiplication”, “percent”, “mean value”, “ratio”), or use statements that require mathematical interpretation (for instance, “the crime rate has grown by a half”). Mathematical information expressed in standard form may be surrounded with text which is also to be considered for successful task performance.
The key skills measured by PIAAC numeracy tasks include dealing with numerical information, ability to determine form and size of objects (spatial perception), ability to understand mathematical formulas (regularities and dependences) and to work with them, performing operations with data (probability) .
Dealing with numerical information. Ability to interpret the world numerically is tested, for example, through asking respondents to count objects or estimate personal expenses for products and services, determine size or dimensions (length, surface area, volume), temperature, humidity, atmospheric pressure, income, etc. Respondents are asked to work with integer numbers, simple and decimal fractions, percents. The tasks involve data people encounter in everyday life, like telephone numbers or postal codes. Respondents are supposed to identify information of these types in various contexts. Skills in basic mathematical operations (addition and subtraction, multiplication and division, squaring) are required to perform numeracy tasks. A sample task: find the cost of one can if four cans cost RUR 200. A more complicated task may include fractions, as when respondents are asked to find the cost of 0.283 kg cheese at the price of 120.95 RUR/kg. A sample task of medium difficulty is provided in Figure 3.
8
Educational Studies. 2014. No 2
O. Podolskiy, D. Popov Г ■ The First Assessment of Adult Competencies in Russia
Figure 3. Numeracy assessment sample task
Look at the graph about the numbers Pf births. Click to ansver the question below.
Du rung which period (s) was there a decline in the numbers of birth? Click ail that apply.
□ 1957-1967
□ 1957-1977
□ 1977-1997
□ 1937-1997
□ 1997-2007
the following graph shews the number of births in the United States from 1957 to 2007. Date are presented every 10 years
2. Spatial perception, ability to determine form and size. Graphic images, drawings, road signs, etc. are used in PIAAC to assess ability to measure objects in two- or three-dimensional space, deal with projections, calculate surface area and length, represent objects in different planes. One needs to understand both informal descriptions and standard systems of measurement, such as the metric system or the imperial system. The basic task is to determine form and size of a given object; higher-level tasks ask respondent to describe how the object’s volume will change if one of dimensions changes.
3. Regularities and dependences, ability to understand mathematical formulas and to work with them. Ability to spot and analyze regularities and dependences in the world around is largely based on mathematical thinking. People with relevant skills understand correlations and development patterns, like in growth of living organisms, price fluctuations, acceleration or deceleration. Ability to evaluate changes and their pace is based on the skills of constructing and applying mathematical formulas and using proportions. A simple sample task from this section is to use a formula to calculate the area of a square or rectangle. More complicated tasks are associated with using formulas to calculate compound interest or body mass index. Respondents may also be asked to assess budget planning options with different interest rates or to compare efficacy of possible weight loss or gain strategies.
4. Operations with data, probability. Working with data files (sheets) implies that one is familiar with basic statistical notions, such as variable, probability, or sample, is able to understand and interpret data presented in graphs or sheets, and is skilled in sorting and selecting required information out of bulks of data. The ability to define probability of this or that event comes in handy often in everyday life, whether it is about weather, stock exchange, or travel planning. A
http://vo.hse.ru
9
THEORETICAL AND APPLIED RESEARCH
2.3. Assessment of proficiency in problem solving in technology-rich environments
3. Field sampling procedures
simple sample task is to interpret a diagram or a sheet; a more complicated task is to define probability of a given event based on the information provided.
Information technology develops at a tremendous pace, expanding the scope of its application. PIAAC was the first to attempt to assess skills of problem solving in technology-rich environments across populations of different countries. We can hardly say at this point that PIAAC tools simulate a truly technology-rich environment; the project rather assesses computer and Internet using skills [OECD, 2009c]. An adequate and comprehensive evaluation of the ability to solve problems in technology-rich environments will require a good number of means to imitate the diversity and changeability of modern information technology.
Information technology” are also represented in PIAAC problem tasks. One of the key everyday missions of digital technology is to provide people with efficient communication means, that’s why we have email, chats, SMS, AOIP, and video communications.
Literacy, numeracy, and problem solving in technology-rich environments are all based on the same key cognitive processes. Computer environment commonly uses symbol information, which is part of computer interface (icons, commands) and interface of most desktop applications (text processor, spreadsheets, browser, email client). Tasks on problem solving in technology-rich environments also evaluate a number of specific competencies that go beyond processes promoting reading literacy and numerical thinking.
When solving tasks in this section, respondents have to find a suitable software application, choose the best from several possible solution strategies, utilize appropriate functions, implement skills of interpreting poorly structured hypertexts, and use online resources.
Problems are solved in an environment with numerous heterogeneous information sources. Tasks of this type assess ability to find solutions depending on the information source used (for example, specific website); user’s choice becomes crucially important for task performance. These problems are focused on the ability to integrate information from different, sometimes conflicting sources.
A simple sample task in this section is to search for information in electronic environments (e. g. on a specific webpage); more complicated tasks imply effective use of functions in some of the above-mentioned applications.
PIAAC is designed to assess competencies and behavioral practices of adult population: working and non-working men and women in cities and in the countryside. That is why the sample was formed in multiple stages and the statistical population was stratified by a num-
10
Educational Studies. 2014. No 2
O. Podolskiy, D. Popov Г ■ The First Assessment of Adult Competencies in Russia
ber of parameters: type of region, type of locality, type of district, and type of household.
At the first stage in Russia, regions were selected by population size and geographical location. Next, the following types of populated localities were selected within the regions: city, town, rural settlement. Relevant address bases were developed for each sampled locality, with addresses additionally broken into groups (to maintain proportional distribution by types of districts: central, mid-central, and peripheral). The target number of households to be surveyed was determined for each type of district.
The survey covered the following federal districts and macroregions: Central Federal District, Northwestern Federal District, Volga Federal District, Southern Federal District, Ural Federal District, Siberian Federal District, Far Eastern Federal District, Moscow, Moscow Oblast, and Saint Petersburg. Moscow, Moscow Oblast and Saint Petersburg are self-representing strata, while the rest of the surveyed regions are distributed across Federal Districts proportionally to the size of population. All in all, over 5,000 adult respondents aged between 16 and 65 from 25 krais and oblasts and 94 settlements participated in the project in the Russian Federation. Data collection quality tests revealed incorrect function of five tasks in Moscow and Moscow Oblast samples, so the total numbers didn’t include data for these regions. The statistical population included 3,800 people and is representative of the whole country except Moscow and Moscow Oblast.
The survey included screening-procedure of selecting respondents from households in accordance with a screening questionnaire. The interviewer makes a list of all household members of suitable age (from 16 to 65), sorts them by decreasing age and chooses a respondent in compliance with the screening questionnaire rating scale. If a household consists of less than four people aged between 16 and 65, only one respondent is selected for the survey. However, if a household comprises four or more members of appropriate age, the interviewer selects two respondents (the selection scale provides both options) .
Thus, stratified, proportional and random sampling was used depending on the structure of selection parameters. Type of sampling: multistage stratified area random sampling. The interviewing procedure was tested during a 2010 pilot survey.
The survey used computer-assisted personal interviewing technique (CAPI). Respondents were interviewed using computers, but incompetent users or people unfamiliar with electronic environments were offered paper versions of the test.
Each respondent filled out a formalized questionnaire including questions on their age, gender, education, work, and professional qualification, and performed independently a series of tests. 3045 minutes were given to fill out the questionnaire, and test performance took from 60 to 180 minutes. Prior to testing, respondents
http://vo.hse.ru
11
THEORETICAL AND APPLIED RESEARCH
were told that there were no correct answers and any answer would be accepted.
Computer-based version of the test represented a multistage process distributing respondents between sectors to assess literacy, numeracy and problem solving in electronic environments (working with email and using internet resources, systematization of data, etc.). Respondents first had to do basic (the easiest) tasks. If they failed, the interview was stopped and more complicated tasks were never given to them. Respondents who performed basic tasks at the minimal acceptable (or higher) level were given tasks on two randomly selected domains out of three: adaptive assessment of literacy, adaptive assessment of numeracy, or problem solving in technology-rich environments.
As PIAAC was developed, it was necessary to provide possibility to compare the newly obtained results with those of previous international assessments of adult competencies. For this reason, 60 percent of literacy and numeracy tasks came from item pools used in ALL and IALS. In addition, some new items were developed for PIAAC that can fit the computer-based adaptive testing requirements and constraints. Task performance is assessed in PIAAC using a scale similar to those in the ALL and IALS. Proficiency scores are reported on a scale of 0 to 500, with five ability levels in each of the three tested domains. These levels are described in terms of problems respondents can solve.
• Level 1: raw score of 0 to 225.
• Level 2: raw score of 226 to 275.
• Level 3: raw score of 276 to 325.
• Level 4: raw score of 326 to 375.
• Level 5: raw score of 376 to 500.
Each respondent was not offered all of the tasks, but the special calculation procedure allows to obtain approximate predicted results for each item.
The survey determined six levels of literacy proficiency (Levels 1 through 5 and below Level 1), six levels of numeracy proficiency (distributed similarly), and four levels on the problem solving in technology-rich environments scale (Level 1 through 3 plus below Level 1). Detailed description of the levels in terms of problems is provided in [National Report on the International Assessment..., 2014].
4. PIAAC Results obtained in the assessment of adult competencies in the Rus-results sian Federation provide answers to the two key PIAAC questions: What are differences in the key competencies between people and countries? and How do the identified proficiency levels correlate with socio-demographic characteristics of respondents?
12
Educational Studies. 2014. No 2
O. Podolskiy, D. Popov Г ■ The First Assessment of Adult Competencies in Russia
Figure 4. Average literacy and numeracy scores among PIAAC participating countries
Japan І і і
Finland І і ^
Flanders (Belgium) | i------,
Netherlands | , '
Sweden І Ц
Norway I
Denmark | i —,
Slovak Republic | i ,
Czech Republic І і і
Austria Estonia Germany
Russia І і і
Average | , i
Australia
Canada
Cyprus
Korea
England/N. Ireland (UK) | , i
Poland Ireland France United States Italy Spain
240 250 260 270 280 290 300
This section provides results of assessment of key adult competencies in the Russian Federation and in other countries involved in PIAAC (Figure 4) .
The average Russian1 score in adult literacy is 275, as compared to the OECD average of 273. The diagram shows groups of countries with average literacy scores much higher (Japan, Finland, Norway, etc.) and much lower (Germany, the U.S., France, etc.) than the average Russian score.
1 Russian results referred to hereinafter do not include Moscow and Moscow Oblast (see section Field Sampling Procedures) .
Proficiency
□ in literacy
□ in numeracy
4.1. International and Russian PIAAC results
http://vo.hse.ru
13
THEORETICAL AND APPLIED RESEARCH
Figure 5. Problem solving in technology-rich environments: proficiency levels of Russian adults compared to the average OECD levels (percent of the total respondent population)
Below Level 1 Level 2 Level 3 No Preferred Failed the ICT
Level 1 computer P&P testing literacy entry
experience test
The average Russian score in adult numeracy is 270, as compared to the OECD average of 273. The diagram shows groups of countries with average numeracy scores much higher (Japan, Finland, Flanders (Belgium)) and much lower (Canada, the U.S., France, etc.) than the average Russian score
Adults in most countries, except for Germany, Austria, the Czech Republic, the Slovak Republic, Denmark and Belgium, scored better in literacy than in numeracy.
Assessment of proficiency in problem solving in technology-rich environments revealed that a large part of population in many PIAAC participating countries had poorly developed basic skills, no computer experience or no confidence in their ability to utilize information and communication technologies. On the average, every seventh OECD respondent claimed to have no computer experience or basic skills like using a mouse or scrolling through webpages. 10 percent of respondents in the OECD countries preferred traditional P&P tests over computerized versions.
About half of working-age adults in Russia are incompetent computer users or have no computer experience at all. As compared to the OECD countries, Russia has more adults with proficiency below Level 1, i. e. being only able to solve the simplest tasks, such as signin in/out their e-mail accounts.
Figure 5 shows distribution of ICT skills levels in the Russian sample as compared to the average OECD results. Among respondents with skills required for computer-based testing, about 40 percent of
14
Educational Studies. 2014. No 2
O. Podolskiy, D. Popov Г ■ The First Assessment of Adult Competencies in Russia
Figure 6. Identified competencies and educational attainments of adults in Russia
300
290
280
270
260
250
240
230
Russian adults scored at the lowest levels (Level 1 and below) on the problem solving in technology-rich environments scale. At Level 1, respondents are supposed to use confidently various desktop applications to access information and solve complex multistage problems. About one third of Russian respondents demonstrated proficiency of Level 2 or 3 in problem solving in technology-rich environments. Tasks at these levels require use of various applications and programs in new conditions and independent derivation of integrated multistage solutions. A considerable proportion of respondents with no computer experience or no confidence in their ability to work in electronic environments refused from computer-assisted testing.
IALS and ALL had revealed that competencies and literacy of adults correlated with their educational attainments [OECD, 2000]: as the level of educational attainment grew, so did proficiency in all domains of skills.
Figure 6 shows average scores in three surveyed domains, by educational attainments2. As level of educational attainment increases, so does the average score in the three skill domains. One of the peaks of literacy proficiency corresponds to high school education in all domains. We can suggest that this peak indicates young people 2
4.2. Competencies and educational attainments of adults in Russia
2 The classification of levels of education used in the survey assimilated specialist degrees obtained in the USSR to the modern Master's degrees.
http://vo.hse.ru
15
THEORETICAL AND APPLIED RESEARCH
Figure 7. Skills proficiency distribution across different age groups
who just graduated from high school. The second peak of proficiency corresponds to the Bachelor’s degree and indicates people who continue learning and developing actively.
Level of educational attainment is strongly correlated with level of proficiency. Russian adults with Bachelor’s degrees (or incomplete higher education diplomas obtained in the USSR) have an average advantage of 33 points on the literacy scale and 40 points on the numeracy scale over adults who only completed 9 years of high school. Adults with intermediate vocational education or lower scored at Level 2 in both literacy and numeracy.
Average scores on the literacy and numeracy scales in Russia do not fall below Level 2. It means that even adults with the lowest level of educational attainment—no formal qualification—have their proficiency at Level 2.
4.3. Competencies of Russian adults, by age group
Skill levels have a complex dynamic correlation with respondents’ age. Figure 7 presents average scores of Russian respondents in three surveyed domains depending on age intervals. In accordance with the worldwide PIAAC results, respondents of older age groups have a lower skill level than respondents of younger age groups [OECD, 2013]. The age group of 45-49 in the Russian sample demonstrated a rise in skills proficiency.
Russian respondents aged between 30 and 34 provided for an abrupt decline on the Russian and overall PIAAC literacy and numeracy scales. Proficiency in all domains is closely related to age in the OECD countries, reaching a peak at around 30 years of age and then declining steadily. As for Russian respondents, their skills proficiency starts declining after the age of 50.
16
Educational Studies. 2014. No 2
O. Podolskiy, D. Popov Г ■ The First Assessment of Adult Competencies in Russia
Figure 8. Skills proficiency by qualification levels
□ Proficiency in literacy
□ Proficiency in numeracy
□ Proficiency in problem solving in technology-rich environments
4.4. Competencies and qualification levels of Russian adults
gy-rich environments.
High-skilled White collars Blue collars Low-skilled
occupations occupations
The PIAAC survey divides respondents into four groups by their qualification levels: high-qualified workers, white-collar workers, blue-collar workers, and low-qualified workers.
High-qualified workers with special skills and high educational attainments include top managers, company CEOs, and high-level domain experts. High-qualified workers solve complex scientific problems or conceptual problems associated with company strategies.
White-collar workers are professional employees of medium-level qualifications, with intermediate vocational education or Bachelor’s degrees. They mostly perform office, administrative or managerial tasks. This cohort includes office employees, services staff, salespeople, travel agency managers, hairdressers, etc.
Blue-collar workers are working class workers who mostly perform manual labor, like agricultural workers or milking machine operators. This group includes mostly respondents who completed vocational training or intermediate vocational education.
Low-qualified workers include respondents with occupations that require neither education nor skills: laborers, dishwashers, street sweepers, etc.
Decrease in qualification level entails an insignificant decline in the key competencies of literacy and numeracy (Figure 8). However, low-qualified workers showed higher proficiency in literacy and numeracy than blue-collar workers.
Skills proficiency among blue collars is relatively low in Russia. Low-qualified workers leave them behind in literacy and especially in the other two domains: numeracy and problem solving in technolo-
http://vo.hse.ru
17
THEORETICAL AND APPLIED RESEARCH
□ Men
□ Women
4.5. Differences in Russian adult competencies related to gender
5. Results and discussion
Figure 9. Skills proficiency among men and women (average scores)
Proficiency Proficiency Proficiency in
in literacy in numeracy problem solving
in technology-rich
Gender-related differences in the Russian sample are insignificant, but results of women are higher than men ones in all three domains. Average scores for men and women are given in Figure 9.
On the average, women outscored men by 4 score points on the literacy scale, 3 points on the numeracy scale, and 6 points on the problem solving in technology-rich environments scale. Significant difference in the average scores between women and men was revealed on the literacy scale (р = 0.05). Distribution by age also differs a lot between genders (р = 0.05). Men aged between 45 and 54 scored higher than women of the same age cohort. However, women outscored men in the rest of the age groups.
One of the most important objectives of OECD’s international surveys is to assess potential of modern humans, opportunities for their development and factors influencing their attainments. The data obtained in the PIAAC survey in Russia enable us to compare national scores to those earned by countries with high levels of adult competencies. Such analytical work is necessary to develop the national systems of lifelong learning and adult education.
Russia’s first participation in a large-scale survey of adult competencies produced high-level results. This is probably explained by a larger proportion of people with tertiary education in Russia than in many other survey participating countries [OECD, 2012]. Important role is assigned to subsequent data analysis and comparison with results in other countries, especially those that are close to Russia by economic, socio-demographic or national and cultural criteria. Cer-
18
Educational Studies. 2014. No 2
O. Podolskiy, D. Popov Г ■ The First Assessment of Adult Competencies in Russia
tain language and cultural parallels allow to regard Russian-speaking population of Estonia as a control group for comparison and verification of the results obtained.
Analysis of results appears to be useful for the national education policy planning. PIAAC obtained crucially important information on relationships between individual respondents’ characteristics-so-cio-economic status, level of education, age, gender, mother tongue, country of origin, occupation, and professional qualification—with levels of skills proficiency, as well as on how they influence self-fulfillment. This information is indispensable to analyze sources of competencies, to determine perspectives of the education system, and to develop target policies and programs designed to overcome inequality in access to education, increase proficiency in the key competencies, and provide more adult learning and lifelong learning opportunities.
PIAAC results in Russia are largely consistent with the worldwide correlations between levels of skills proficiency and socio-demographic characteristics, particularly with gender- and age-related correlations. At the same time, some unique data was produced by the Russian sample. Thus, skills proficiency in the combined PIAAC sample reaches a peak at around 30 years of age, followed by a gradual decline [OECD, 2013]. In the Russian sample, proficiency peaks at 40-45 years of age and declines steadily, while the age cohort of 3034 has revealed abrupt declines on both literacy and numeracy scales.
Education system and labor market in the Russian Federation have undergone considerable changes for the past 20-25 years. In particular, education has been affected significantly by economic and political reforms. Respondents who are aged 30-34 today studied in general secondary schools in the years of Perestroika and emergence of the new Russia. Perhaps, that was what predicted their relatively low scores on the literacy and numeracy scales. However, skills proficiency in problem solving in technology-rich environments was acquired later and on a rather high level. This age cohort mastered computer literacy and digital skills quite efficiently because active development of technology and computers concurred with their adolescence and youth, when they were sensible, enthusiastic and reaching for everything new. The new came in form of computers and large-scale use of Internet in Russia.
Skills proficiency declines abruptly at the age of 60-65. In Russia, this is the retirement age for men and the end of the first five years of women’s retirement. This data permits to predict how the labor and social policy providing opportunities for further studies and advanced training correlates with maintaining and developing skill levels throughout life. Virtually no generation gap was revealed in Russia, which may indicate stability of competencies and a social and education system that allows maintaining and developing such competencies throughout life.
http://vo.hse.ru
19
THEORETICAL AND APPLIED RESEARCH
Educational attainments predict the level of skills proficiency. Correlation between these factors exists both within OECD countries and within PIAAC participating countries [OECD, 2013], and it’s also typical of the Russian sample. Russian adults with vocational training or intermediate vocational education tend to score lower in competencies than their counterparts with completed secondary education or Bachelor’s degrees. These results mean that the level of skills proficiency among adult population is influenced by educational trajectories, and that some levels of education have a clearly insufficient potential.
Proficiency in literacy and numeracy is obviously related to occupation and qualification level. In all countries, correlation between the literacy scale and the level of professional qualification weakens if other socio-demographic characteristics are considered. First of all, this is explained by predictably higher level of education among adults with high-skilled occupations. Nevertheless, differences in proficiency in literacy and numeracy between respondents groups with different qualification levels persist even when other characteristics are considered. It suggests that qualification level required at work and specific operations people perform at work may play a certain role in maintaining and developing information-processing skills. However, the abovesaid is contradicted by the higher level of competencies among non-qualified workers as compared to low-qualified workers in Russia. This may probably be explained by culturally-conditioned behavioral patterns of adults in Russia who tend to stay in shadow economic sectors, distorting or concealing their true qualifications and incomes.
To summarize, adults in the Russian Federation earned PIAAC literacy and numeracy scores comparable to the average OECD scores. As the level of educational attainment grows, so does the level of skills proficiency among working-age population of Russia, which reaches a peak at 45-49 years of age. Competencies associated with problem solving in technology-rich environments and with using information and computer technologies turned out to be the most important skills domain for adult population in Russia as well as in the rest of PIAAC participating countries.
This paper only gives a primary analysis of data collected by the survey. Secondary analysis will be aimed, in particular, at exploring characteristics of categories of adult respondents who lack the minimal required skills and thus comprise the risk group, as well as of respondents with limited ability to perceive information in their mother tongue or to use numeracy skills not only for work but also for a productive life. Cross-country comparison of results also appears essential to identify qualitative changes education brings in development and further application of competencies throughout life.
No one can change the past. However, educational policies are capable of providing opportunities for further studies, so that mature
20
Educational Studies. 2014. No 2
O. Podolskiy, D. Popov Г ■ The First Assessment of Adult Competencies in Russia
adults could maintain and develop their own potential and competencies. Maintaining and developing skills at a mature age will promote active ageing and maximizing the potential of working-age population.
PIAAC is a unique high-profile project that is going to be the starting point for global changes in education systems of many countries. Knowing exactly the potential of adults with specific socio-demographic characteristics at specific stages of life, we can get the idea of how well off the country is and how successfully or timely it makes political decisions. PIAAC results show not only the actual situation with adult literacy, prospects for adult development and impact factors, but they also draw the bottom line in assessing educational tools and mechanisms that led the adult population to these of those attainments in life. Particularly, the survey provides a different perspective on developing the lifelong learning system in the Russian Federation.
1. Brown C. A., Dickson R., Humphreys A. — L., McQuillan V., Smears E. (2008) Promoting Academic Writing/Referencing Skills: Outcome of an Undergraduate E-Learning Pilot Project. British Journal of Educational Technology, vol. 39, no 1, pp. 140-156.
2. Davis L., Fisher D., Ford C. (2009) Teaching with Technology to Decrease Writing Apprehension and Increase Writing Skills in a Business Communication Course. Business Education Digest, vol. 18, no 1, pp. 1-12.
3. Fallahi C. R., Wood R. M., Austad C. S., Fallahi H. (2006) A Program for Improving Undergraduate Psychology Students' Basic Writing Skills. Teaching of Psychology, vol. 33, no 3, pp. 171-175.
4. Machlup F. (1962) The Production and Distribution of Knowledge in the United States. Princeton, NJ: Princeton University.
5. Moser C. (1999) A Fresh Start: Improving Literacy and Numeracy. London: Department for Education and Employment.
6. Murray S., Clermont Y., Binkley M. (eds) (2005) International Adult Literacy Survey. Measuring Adult Literacy and Life Skills: New Frameworks for Assessment. Ottawa: Statistics Canada.
7. HSE (2014) National Report on International Assessment of Adult Competencies in the Russian Federation (in press)
8. OECD and Statistics Canada (1995) Literacy, Economy and Society: Results of the First International Adult Literacy Survey. Paris: OECD; Ottawa: Minister of Industry.
9. OECD (2000) Literacy in the Information Age: Final Report of the International Literacy Study. Paris: OECD.
10. OECD (2005) The Definition and Selection of Key Competencies: Executive Summary. Paris: OECD.
11. OECD (2009a) PIAAC Literacy: A Conceptual Framework. OECD Education Working Paper no 34. Paris: OECD, PIAAC Literacy Expert Group.
12. OECD (2009b) PIAAC Numeracy: A Conceptual Framework. OECD Education Working Paper no 35. Paris: OECD, PIAAC Numeracy Expert Group.
13. OECD (2009c) PIAAC Problem Solving in Technology-Rich Environments: A Conceptual Framework. OECD Education Working Paper no 36. Paris:
References
http://vo.hse.ru
21
THEORETICAL AND APPLIED RESEARCH
OECD, PIAAC Expert Group on Problem Solving in Technology-Rich Environments.
14. OECD (2012) Education at a Glance 2012: OECD Indicators. Paris: OECD.
15. OECD (2013) Skilled for Life? Key Findings from the Survey of Adult Skills. Paris: OECD.
16. Popov D., Podolsky O. (2012) Naskolko kompetentny sovremennye vzroslye: mezhdunarodnaya perspektiva [Competencies of Modern Adults: International Perspective]. Obrazovatelnaya politika, no 2 (58) .
17. Saavedra A. R., Saavedra J. E. (2011) Do Colleges Cultivate Critical Thinking, Problem Solving, Writing and Interpersonal Skills? Economics of Education Review, vol. 30, no 6, pp. 1516-1526.
22
Educational Studies. 2014. No 2
