CC BY
12
Вестн. Моск. ун-та. Сер. 19. Лингвистика и межкультурная коммуникация. 2014. № 3
ПРАКТИКА ПРЕПОДАВАНИЯ ИНОСТРАННЫХ ЯЗЫКОВ В СИСТЕМЕ НЕПРЕРЫВНОГО ОБРАЗОВАНИЯ
A.A. Korenev
HIGH-STAKES LANGUAGE TESTING BETWEEN SCHOOL
AND UNIVERSITY IN RUSSIA, JAPAN AND THE UK
(constructs and formats)
The article compares formats, constructs and criteria of assessment between school and university in Russia, Japan, and the UK. This comparison allows to analyse the experience of nations that adopted standardized testing for selection of potential students much earlier than the Russian Federation and whose examination systems have already experienced reforms and changes, as well as to discover the cultural values behind the three national examination systems.
Key words: assessment, testing, cultural impact, education system.
В статье сравниваются формат, конструкты и критерии оценивания между школой и университетом в России, Японии и Великобритании. Подобное сравнение позволяет проанализировать опыт стран, которые внедрили стандартизированное тестирование для отбора студентов намного раньше, чем Российская Федерация, и чьи экзаменационные системы уже пережили серию реформ и изменений, а также обнаружить культурные ценности, стоящие за тремя национальными экзаменационными системами.
Ключевые слова: оценивание, тестирование, влияние культуры, образовательная система.
Despite the immense influence that national examinations between school and university have on the development of the whole system of secondary and higher education in each country they have been given less attention in international research on the subject than international examinations, such as TOEFL, IELTS etc. In contrast, they remain a popular topic of discussion in the media, which often leads to misunderstanding of the test construct and format and wrong generalizations made by the general public1. Research in the field of national assessment
Коренев Алексей Александрович — канд. пед. наук, ст. преподаватель кафедры теории и методики преподавания иностранных языков факультета иностранных языков и регионоведения; e-mail: alexeykorenev@gmail.com
1 Ter-Minasova S. Learning and Teaching Languages in Russia: Old Traditions and New Problems // Moscow State University Bulletin. Linguistics and Intercultural Communication. 2013. N 2. P. 9—20.
mechanisms was in focus at the beginning of the 1990-s and developed along with Fullan's theory of innovation and change in education2.
When researchers do analyze national examinations, they usually publish in national journals, many of which are not issued in English, or in the media3, with the exception of the Japanese Center Test for University Admissions that has attracted widespread criticism from international researchers who work at Japanese universities4. Moreover, there have been very few attempts to compare examinations between countries5. Although it is generally believed that the world is moving towards a global system of higher education, the examinations that give access to higher education, even within international alliances, such as the EU or the CIS, may vary considerably in terms of their test construct and format. The difference can be even larger, when national examinations are analyzed across such contrasting countries as Russia, Japan and the UK.
In general, the task is to compare three examinations in terms of their structure, format and test construct (sometimes indirectly, through analyzing the test structure, as the specifications for the Japanese examination are kept confidential), find out similarities and differences and explain them through the lens of the national culture, values, traditions and the context of higher education in each country. This shall enable linking the variations in the three language tests with specific social and cultural factors that affect the national examination system in three countries.
The term 'national examinations' will be used in this work to define nation-wide high-stakes tests between school and university. This term has already been used in numerous studies of high-stakes examinations in developing countries, some of which were conducted for the World Bank6. Eisemon, for instance, looked at the intended and the real wash-back after the introduction of high-stakes national language tests in some African countries.
2 Fullan M., Stiegelbauer S. The New Meaning of Educational Change. 2nd ed. N.Y.: Teachers College Press, 1991; Wall D. Introducing New Tests into Traditional Systems: Insights from General Education and from Innovation Theory // Language Testing. 1996. N 13. P. 334-354.
3 Suzuki S. Damning Report on School English // Asahi Evening News. 1998. September 20.
4 Murphey T. The Tale That Wags (English Version). Perceptia Press, 2010.
5 Korenev A. Through Borrowing We Create Our Own (social and cultural peculiarities of the Japanese National Center Test for University Admissions in English and German languages) // Moscow State University Bulletin. Linguistics and Intercultural Communication. 2011. N 1. P. 122—130.
6 Heyneman S.P., Ransom A.W. Using Examinations and Testing to Improve Educational Quality // Educational Policy. 1990. P. 177—192; Eisemon T.O. Examinations Policies to Strengthen Primary Schooling in African Countries // International Journal of Educational Development. 1990. N 10. P. 69—82.
Tests' Structures and Formats
Now we shall consider the structure of the three examinations, the format and the examination tasks that are used to assess students' knowledge and skills to make relevant conclusions about the test construct. As the examinations are relatively different it their structure, first, an overview of the three examinations' formats will be given, then, the structure of the examinations will be compared to define the similar aspects (the invariable core of the three examinations) and structural variations.
The Japanese Center Test — Assessing knowledge and memory
Unlike the other two examinations in foreign languages, where the examination format does not vary according to the language that is being tested, the Japanese examinations in English and German are different. The first difference is the emergence of the Listening test as a separate part of the English language exam in 2006. This listening examination does not exist in case of German and other foreign languages that can be taken within the Center Test. This leads to unequal Center Test scores for students taking the examination in different languages. Moreover, some universities require the English listening test mark for admissions. Both examinations use the format of multiple choices, so that the whole examination can be graded using computers.
Both English and German tests have 6 parts that are divided into sections. Section A of Part I tests the linguistic knowledge of students by asking them to identify the word, where a sound is pronounced differently (homographs and diphthongs are often used in this task), or to choose the word where the stress is on a different syllable (Section B). Two points are given for every correct answer in this task. From 2007 to 2009 Section C that attempted to assess the knowledge of intonation patterns by asking to choose what a sentence would mean if the stress is on a certain word was also included in the examination, and in 2009 it was even extended to Section D, where students had to choose the sentence where the correct words are stressed. This part seems to substitute a speaking test, where pronunciation is tested, by assessing the students' awareness of various pronunciation rules.
Section A of Part II gives ten gapped sentences and asks students to fill in the gaps with one of four words that are given. In 2007 most of the sentences tested only vocabulary, whereas in 2011 questions that tested vocabulary and those testing grammar constituted one half of this section. Sections B and C are different in the German and English language exams. In the English language test students have to complete a dialogue of 3—4 sentences with a suitable sentence (Section B), and put the words into the correct order to form a sentence (Section C).
The rest of the test (Parts III, IV, V and VI) is assessment of various reading strategies and skills using a wide range of conversation, media, advertising, fictional or academic texts. The texts are both monologues and dialogues and vary in length from 60 to 600 words. The value of every correct answer is increasing from Part III to Part VI (can reach up to 6 points for every correct answer) with the rising cognitive difficulty of the texts.
Listening is a separate examination, which can give additional 25% to the overall English language score. Students have an opportunity to listen to the recordings as many times as they wish within a set limit. All texts are read at a rather low speed and their volume is ranging from 20 (Part I) to 250 words (Part IV). As opposed to the first part of the exam, every correct answer in the listening part is worth 2 points. Text types include both monologues and dialogues short everyday conversation, radio programs, and a conversation on an academic subject. Task types include note completion, answering questions and a task where students need to select the best phrase to respond to a statement or a short dialogue that they have heard.
Overall, the most important aspects of the Japanese test are the extensive stress on comprehension, rather than production. The information about the test construct is not open to the general public, however, from the tasks that are published it can be seen that linguistic knowledge, all types of reading and most listening skills are assessed.
A Levels — Research and production
A Levels are a very opposite case to the Japanese Center test, as the emphasis is laid on assessment of productive language skills. A major advantage of competition between different examination boards is that in order to provide a better environment for teachers, students and centers they can afford assessing speaking and writing, and make a large amount of examination materials accessible on their web pages, including specifications, mark schemes and examiner reports.
Another important difference is that this examination has modular structure, and students can take the exam on two levels: Advanced Subsidiary (AS) or Advanced (A2). There is also a substantial amount of choice inside the units (topic areas for speaking, discursive or creative essays, research essay subject). As the units assess language skills on different levels some correlation may be noticed between Unit 1 (Spoken Expression and Response) and Unit 3 (Understanding and Spoken Response), as well as Unit 2 (Understandingand Written Response) and Unit 4 (Research, Understanding and Written Response)1. Grammar and vocabu-
7 Edexcel. 2001. Getting Started September 2001: GCEs in Modern Foreign Languages. Edexcel Publications, 2007.
lary are tested implicitly through assessment of speaking, writing, listening, reading, and written translation skills.
Unit 1 assesses speaking skills and consists of two sections: a conversation around a stimulus (a short text and a picture) and a conversation with an examiner on the topic area chosen by the student. Assessment criteria include understanding of the general topic area (that largely depends on the research and reading that this student has carried out before the examination), understanding of the stimulus, range of vocabulary and grammatical structure, accuracy and response (spontaneity and fluency). This Unit contributes to 15% of the total GCE A-Level Mark and 30% of the mark at AS level. Unit 2 includes listening and reading tasks of various formats (multiple choices, selecting words to complete notes, answering questions in English or in the target language) based on different texts (everyday conversations, interviews, printed media texts). The last part of Unit 2 is a letter or an email (most often), or an article written in response to a short stimulus (short newspaper article or advertising) that has to address four or five bullet points. Unit 2 makes 35% of the total A-Level mark and 70% of the AS-mark.
Unit 3 has a two-section structure. In the first part of the examination students have to produce a one-minute statement on a controversial topic and then engage in a discussion with the examiner, who supports the opposite point of view. After that, the student and the examiner discuss some issues related to other topic areas. This unit tests not only the spoken language of a candidate, but also the evidence of research carried out before the examination and an ability to support arguments with relevant facts. Unit 4 is the culmination of the whole examination, for it combines research and critical thinking with writing skills. It has three sections: translation, a discursive or creative essay, and a research-based essay.
It can be clearly seen on the example of this specification that the stress in the A-Level language assessment system is laid on production, i.e. speaking and writing. Listening and reading as separate tasks are only included in Unit 2 to and make less than 15% of the total examination mark. Vocabulary and grammar are only assessed indirectly as assessment criteria in speaking and writing (accuracy and range of lexis).
RSE — equally weighed assessment of language skills with a missing
component
The Russian State Exam is divided into three levels that are described in the exam specification (RSE FL Specification 2013): Basic (CEFR A2+ — those descriptors of the A2 level that are closer to B1, than to A1), Intermediate8 or Advanced (direct translation) RSE Level
8 Verbitskaya M. International Standards and National Examinations: The Russian Experience. The Language Teacher // JALT 2008. Pre-Conference Special Issue 2008. N 32/7. P. 37—40.
corresponds with B1 of the CEFR, and RSE Higher Level is based on some descriptors of the CEFR B2 level.
The current RSE in foreign languages is a written examination that consists of four sections: Listening, Reading, Grammar and Vocabulary, and Writing. The listening and reading texts are everyday conversations, announcements, media texts, interviews, popular science and even fiction (at the Higher level). Task types in these sections include multiple choices and multiple matching and assess such skills as reading for general understanding and reading for detail, skimming and scanning. The Grammar and vocabulary section tests the ability to transform the words given on the margins to fill in the gaps in two texts. The third part of the section is a gapped text with multiple choices. Writing is presently the only part of the examination that requires constructed response and operates with two genres: a personal letter on the Basic level and an academic essay on the Advanced and Higher Level. Every part of the examination makes 25% of the examination raw mark. This division can be related to the integrated nature of assessment of multiple skills at Russian universities9.
The most important change that happened to the RSE in 2009 is the exclusion of the Speaking part from the examination. The RSE of 2008 included a ten-minute interview. The speaking part included a short icebreaker-conversation which was not assessed, a monologue on the given subject (e.g.: learning a foreign language) was used as an Advanced level task, and a conversation with the examiner on an everyday subject, during which students were expected to initiate and maintain conversation, explain, suggest, give reasons, agree or disagree, find out the partner's attitudes, ask for the partner's opinion, reach an agreement by taking into account the partner's point of view10. This part was dropped in 2009 due to some organizational reasons that are to be discussed later in this paper.
Table 1 compares the structure of the Center Test, A Levels and the RSE in foreign languages. This comparison makes it clear that all three examinations assess vocabulary, grammar, listening and reading, which form the universal core of assessment between school and university. Despite differences in teaching in the three countries, higher education requires attending lectures and reading various sources, therefore, all three examinations assess academic reading and listening skills.
9 Solovova E.N., Basova I.A. The Need for Teaching Students of Specialized Language Schools / Faculties to Prepare Summaries of Academic Audio Texts // Moscow State University Bulletin. Linguistics and Intercultural Communication. 2012. N 2. P. 19—31.
10 Russian Federal Institute of Pedagogical Measurement. 2007—2013. RSE Demo Versions. RSE Specifications. URL: http://www.fipi.ru/view/sections/92/docs/ (last accessed: 28.01.2013).
Table 1
Examination structure (proportion in the total mark)
Parameters Countries
Russia Japan Great Britain
Theoretical knowledge 0,00% 5,60% 0,00%
Lexical and grammatical skills 25,00% 16,40% Integrated into productive skills assessment (25%)
Translation 0,00% Only in the German language test, used for assessment of reading (general understanding) Translation of texts of app.80 words into the foreign language 3,00%
Reading 25% (press, fiction, popular science and pragmatic texts) 58% (press, fiction, popular science and pragmatic texts) 42,5% (integrated into other parts), 10% (reading tasks)
Listening 25,00% global understanding and full understanding in detail of monologues and conversations (everyday conversations, pragmatic texts, radio reports) Additional examination needed for the most prestigious universities (makes 20% of the total mark) global understanding and full understanding in detail of monologues and conversations (everyday conversations, pragmatic texts, radio reports, popular science texts, continuing a conversation — simulation of speaking) 32% (integrated into listening and speaking), 10% listening tasks: global understanding and full understanding in detail of monologues and conversations (everyday conversations, pragmatic texts, radio reports)
Writing 25,00% personal letter and discursive essay 0,00% 45,00% article/email/letter, translation, discursive essay/creative essay, re-search-based essay
Speaking 0,00% 0,00% 32,00% monologue (one minute), answering text-related questions, discussion with the examiner
Task format Multiple choice, gap filling, writing tasks Multiple choice Multiple choice, short answers, speaking and writing tasks
Division into difficulty levels Basic/Advanced/High, in accordance with CEFR A2 +/B1/B2 Task value 1, 2, 4 h 6 points11 Advanced Subsidiary/ Advanced
Division into units and their autonomy Units cannot be taken separately Listening is an independent unit Four independent units
11 Difficulty levels are not determined by the designers of the test.
Conclusion
This study looked at the construction and development of national testing system as cultural and social phenomena. It compared the test constructs and formats of the three national examinations in foreign languages in very different cultural patterns and tried to find the roots of their difference.
It is noticeable that there has been little evidence of attempts to compare different national examination systems. However, we would argue that this is the point when different concepts and theories, cultures and traditions, as well as the real life will challenge the domain of modern language testing, and link it to other areas of applied linguistics and humanities. Comparing national high-stakes tests may also help to find invariable components (such as the presence of different types of reading and listening in all three tests) and make links between different examination systems on the way to recognition of national examinations as international admissions mechanisms. Furthermore, it is an important step in the development of national examinations systems through adaptation of some mechanisms that are used in different systems (even though social and cultural factors may continue to play a natural threshold role by the implementation of new methods and standards).
References
Edexcel. 2007. Getting Started September 2007: GCEs in Modern Foreign
Languages. Edexcel Publications, 2007. Eisemon T.O. Examinations Policies to Strengthen Primary Schooling in African Countries // International Journal of Educational Development. 1990. N 10.
Fullan M., Stiegelbauer S. The New Meaning of Educational Change. 2nd
ed. N.Y.: Teachers College Press, 1991. Heyneman S.P., Ransom A.W. Using Examinations and Testing to Improve
Educational Quality // Educational Policy. 1990. Korenev A. Through Borrowing We Create Our Own (social and cultural peculiarities of the Japanese National Center Test for University Admissions in English and German Languages) // Moscow State University Bulletin. Linguistics and Intercultural Communication. 2011. N 1.
Murphey T. The Tale That Wags (English Vfersion). Perceptia Press, 2010. Russian Federal Institute of Pedagogical Measurement. 2007—2013. RSE Demo Versions. RSE Specifications. URL: http://www.fipi.ru/view/ sections/92/docs/ (last accessed: 28.01.2013). Solovova E.N., Basova I.A The Need for Teaching Students of Specialized Language Schools / Faculties to Prepare Summaries of Academic Audio Texts // Moscow State University Bulletin. Linguistics and Intercultural Communication. 2012. N 2.
Suzuki S. Damning Report on School English // Asahi Evening News. 1998. September 20.
Ter-Minasova S. Learning and Teaching Languages in Russia: Old Traditions and New Problems // Moscow State University Bulletin. Linguistics and Intercultural Communication. 2013. N 2. Verbitskaya M. International Standards and National Examinations: The Russian Experience. The Language Teacher // JALT 2008. Pre-Conference Special Issue 2008. N 32/7. Wall D. Introducing New Tests into Traditional Systems: Insights from General Education and from Innovation Theory // Language Testing. 1996. N 13.
