CC BY
51
www.volsu.ru
ДИСКУССИИ
DOI: https://doi.Org/10.15688/jvolsu2.2019.4.18
UDC 811.161.1:37.02 Submitted: 11.07.2019
LBC 81.411.2-9 Accepted: 04.11.2019
THE RUSSIAN LANGUAGE TEST: TOWARDS ASSESSING TEXT COMPREHENSION 1
Kathryn Soo McCarthy
Georgia State University, Atlanta, GA, USA
Danielle Siobhan McNamara
Arizona State University, Tempe, USA
Marina I. Solnyshkina
Kazan (Volga Region) Federal University, Kazan, Russia
о
<N
> Fanuza Kh. Tarasova
Kazan (Volga Region) Federal University, Kazan, Russia
s
^
Roman V. Kupriyanov
a
^ Kazan National Research Technological University, Kazan, Russia
£¡4 Abstract. Reading comprehension relies on a variety of complex skills that are not effectively assessed by ^ existing Russian language tests. At the same time, Russian textbooks are criticized both for their low text quality | and high text complexity. This study addresses issues of Russian language proficiency and comprehension 2 assessment with the development of the Russian Language Test (RLT). The RLT was constructed to measure proficiency relevant to textbook comprehension, such as grammar and vocabulary knowledge, establishing ^ propositional meaning and inferencing. Results from this initial study including 81 fifth-grade and 94 ninth-grade cs students confirm that students struggle with grammatical inferences and identifying the main idea in a text. ^ Additionally, three standardized Russian exams, VPR, OGE, EGE are analyzed, affording an overview of the testing system for the Russian language from the elementary through high school education levels. This study demonstrates promise for the use of the RLT as a language proficiency assessment and provides a broad context for understanding the current state of Russian language tests for native speakers.
Key words: Russian Language Test, comprehension skills, grammar, vocabulary knowledge, gender, age,
£ m
^ general knowledge.
je
Citation. McCarthy K.S., McNamara D.S., Solnyshkina M.I., Tarasova F.Kh., Kupriyanov R. V. The Russian £ Language Test: Towards Assessing Text Comprehension. Vestnik Volgogradskogo gosudarstvennogo universiteta. § Seriya 2. Yazykoznanie [Science Journal of Volgograd State University. Linguistics], 2019, vol. 18, no. 4, pp. 231-
vi 247. DOI: https://doi.org/10.15688/jvolsu2.2019.4.18
£
tt
CÖ
o
o
УДК 811.161.1:37.02 ББК 81.411.2-9
Дата поступления статьи: 11.07.2019 Дата принятия статьи: 04.11.2019
ТЕСТИРОВАНИЕ ПО РУССКОМУ ЯЗЫКУ: ОЦЕНКА ПОНИМАНИЯ ТЕКСТА1
Кэтрин Су МакКарти
Университет штата Джорджия, г. Атланта, Джорджия, США
Даниэль Шиврон МакНамара
Университет штата Аризона, г. Темпе, США
Марина Ивановна Солнышкина
Казанский (Приволжский) федеральный университет, г. Казань, Россия
Фануза Харисовна Тарасова
Казанский (Приволжский) федеральный университет, г. Казань, Россия
Роман Владимирович Куприянов
Казанский национальный исследовательский технологический университет, г. Казань, Россия
Аннотация. Понимание прочитанного зависит от множества умений и навыков, которые не всегда эффективно оцениваются существующими тестами по русскому языку. Современные российские учебники подвергаются критике как за низкое качество текста, так и за высокую сложность текста. В представленном исследовании рассматриваются вопросы разработки теста по русскому языку (RLT), который выполнял бы функции входного теста для оценки уровня владения языком, а именно базовых грамматических знаний, объема словарного запаса, определения прямого и переносного смысла высказывания. Результаты данного пилотного исследования, в котором приняли участие 81 ученик пятого класса и 94 ученика девятого класса, подтверждают, что основные трудности вызывают задания, оценивающие умение делать умозаключения и определять основную идею текста. В статье также предлагается краткий обзор существующей системы тестирования по русскому языку и анализ заданий трех стандартизированных экзаменов по русскому языку: ВПР, ОГЭ, ЕГЭ. Исследование показало перспективность использования RLT в качестве теста для оценки уровня владения языком и предоставляет широкий контекст для понимания текущего состояния системы тестирования по русскому языку для его носителей.
Ключевые слова: тест на определение уровня владения русским языком, навыки понимания текста, грамматика, знание лексики, пол, возраст, общие знания.
Цитирование. МакКарти К. С., МакНамара Д. Ш., Солнышкина М. И., Тарасова Ф. Х., Куприянов Р. В. Тестирование по русскому языку: оценка понимания текста // Вестник Волгоградского государственного университета. Серия 2, Языкознание. - 2019. - Т. 18, №2 4. - С. 231-247. - (На англ. яз.). - DOI: https://doi.org/ Ш.15688/ггоки2.2019.4.18
Russian elementary schoolers outperform students from all other nations on the Progress in International Reading Literacy Study (PIRLS), but underperform on the Programme for International Student Assessment (PISA) [Ivanova, 2013]. One explanation for these differing levels of performance is that the majority of questions on the PIRLS are retrieval-based, whereas the PISA has a larger number of items directed at
Introduction
interpretation, integration, and evaluation [Zuckerman, Kovaleva, Kuznetsova, 2013]. The analyses of Russian textbooks have indicated that students tend to be encouraged to engage in rote memorization rather than meaningful comprehension and the language in these Russian books is often either overly simplistic [Meyer, 2016; Pinskaya, 2009] or too complicated for the target audience [Kuchma, Tkachuk, 2015; Otsenka didakticheskoy slozhnost..., 2016]. Text that is not matched to readers' abilities can
lead to issues such as lack of motivation, boredom, or frustration [Ponimanie shkolnikami...], which in turn can hamper students' acquisition of academic reading skills.
Russian academic texts have been described by experts as obscure, with excessive nominalization, ambiguous impersonal structures, as well as complicated, and even erroneous syntax [Integratsiya obrazovaniya; Khoutyz, 2013]. Thus, a major focus of educators in Russia is to assist students in developing sufficient reading comprehension skills and help them learn from more complex academic text. One limitation is that there is no short standardized placement test for native speakers that assesses both lower-level and higher-order proficiencies in Russian. This project outlines the first step in an iterative development of a Russian Language Test.
Study
The current study is a part of an on-going Project initiated by Kazan Federal University in 2017 and aimed at evaluating the quality and text complexity of textbooks used in the Russian Federation [Solovyev, Ivanov, Solnyshkina, 2018; Solovyev et al., 2019a; Solovyev et al., 2019b; Petrova, Solnyshkina, 2019]. The Project is outlined in three stages: (1) testing participants' reading comprehension proficiency in Russian and the language of the textbooks they use;
(2) measuring participants' individual differences in general knowledge and five cognitive domains that impact performance using the Wechsler's Intelligence Scale for Children (WISC; 5th graders, aged 11-12) and the Wechsler Adult Intelligence Scale (WAIS; 9th graders, aged 15-16);
(3) evaluating effect of different linguistic features of a printed text on participants' comprehension.
In this article we present the first two stages of the Project and proceed from the viewpoint that a student's limited proficiency in the language of the exam or test may become a barrier to assessing abilities within particular disciplines. That is, such content tests may be a measure of reading comprehension rather than an assessment of the students' knowledge of ability in the given domain (see: [Bowles, 2008]). In this project, we developed a Russian language test which could be used nationwide to differentiate students' reading comprehension skill. This test score can
then be used to examine the degree to which participants' scores in a subject comprehension test may be influenced by reading comprehension issues rather than content issues.
We first discuss the theoretical motivation of assessing multiple levels of proficiency. We then outline strengths and weaknesses of the extant assessments in the context of these theories. We then present the development of the Russian Language Test. Finally, we provide results from an initial implementation of the test with fifth and ninth grade students.
Theoretical Background
The test evaluation procedure is based on a generally accepted view that during reading "we process at different levels simultaneously and draw on both bottom-up and top-down processes in establishing meaning" [Research Notes]. We were guided by Weir's model of reading to examine seven aspects of reading comprehension [Weir, Khalifa, 2008], that reflect both lower-level processes (e.g., decoding) and higher-order processes (e.g., comprehension): 1) word recognition, the ability to "match the form of a word in a written text with a mental representation of the orthographic forms of the language" [Weir, 1993, p. 6]; 2) lexical access, the 'retrieval of a lexical entry from the lexicon, containing stored information about a word's form and its meaning'; the form addresses orthographic and phonological mental representations of a lexical item and possibly information on its morphology [Field, 2004]; 3) syntactic parsing involves grouping "words into phrases, and into larger units at the clause and sentence level to understand the message of the text" [Weir, Khalifa, 2008, p. 6]; 4) establishing propositional (core) meaning at the clause or sentence level, "a literal interpretation of what is on the page, the reader has to add external knowledge to it to turn it into a message that relates to the context in which it occurred [Weir, 2003, p. 6]; 5) inferencing as "a creative process whereby the brain adds information which is not stated in a text in order to impose coherence [Weir, 2003, p. 6]; 6) building a mental model "entails an ability to identify main ideas, to relate them to previous ideas, distinguish between major and minor propositions and to impose a hierarchical
structure on the information in the text" [Field, 2004, p. 241], it is when "the propositions representing the meaning of a text are linked together, usually by argument overlap, to form a hierarchical text base" [Kintsch, van Dijk, 1978, p. 374]; 7) creating a text (or discourse) implies not only recognizing the hierarchical structure of the whole text but also determining which items of information are central to the meaning of the text and which are secondary propositions, that is the ability to recognize significance of different parts ofthe text to the writer or reader [Weir, 2003, p. 6].
Modern Russian Language Tests
The system of extant Russian language tests for native speakers includes the following: (1) VPR (All-Russia testing paper), an end-of-year or promotion examinations, (2) OGE (Compulsory National Exam) enables examinees "to approach secondary education, either along a general university-preparatory track or a vocational-technical track" [Education...]; and (3) EGE (National Unified Examination), "a final graduation examination, as well as an entrance examination for higher education" [Education...]. Below, we describe the strengths and weaknesses of these assessments.
(1)VPR (All-Russia testing paper), Grades 4-6.
The All-Russian testing papers (VPR) are mandatory tests aimed at "identifying students' proficiency to meet the requirements of Federal (Russian) Educational Standard. VPRs are held to measure subject and meta-subject knowledge, universal learning skills and cross-curriculum concepts" [Opisanie kontrolnykh...].
The writing part of Grade 4 VPR in the Russian language consists of two sections: Part I is a dictation and two follow-up items, Part II comprises twelve items, nine of which are text-based tasks. Multiple choice questions are not used in VPR. The tasks of parts 1 and 2 are performed on different days. Each part lasts 45 minutes. The structure and the procedure of the test are similar in VPR5 and VPR6.
As specified by the test designers, the VPR assesses the following:
"3. Reading skills:
3.1. Text comprehension. Reading for specific information.
3.2. Reading for specific explicit information.
3.3. Drawing simple conclusions based on the information in the text.
3.4. Interpretation and synthesis of the information in the text.
3.5. Analysis and assessment of the content, language features and text structure.
4. Writing skills:
4.1. Writing letters, combinations of letters, syllables, words, sentences as part of literacy training.
4.2. Mastering neat writing.
4.3. Copying, dictation.
4.4. Reproducing a text (detailed, selective).
4.5. Writing short original texts (essays) on the topics that might interest children (based on impressions, literary works, pictures, series of pictures, video recording, etc.) [Opisanie kontrolnykh...].
Closer inspection of these tests (VPR 4, VPR5, VPR 6) indicates that they predominantly assess students' knowledge of language facts rather than comprehension skills. For example, in Part I VPR 4 students are expected to write a one-page dictation read by an examiner (item 1), identify sentences with similar predicates and underline them, copy those sentences and underline predicates (item 2); copy sentences from the dictation, underline and define parts of speech of the subject and predicate (item 3). The two first items in Part 2 assess students' knowledge on orthoepic norms: they are to mark the accent (stress) in the following four words (item 4); read the sentence below and copy the word with voiced consonants only (item 5). In items 6-8 students are asked to read a 150 word informational text with elements of narration followed by three open-ended type tasks: identify and write down the main idea of the text (item 6); make a 3-point plan of the text and write it down (item 7); write down a question for your classmates which would test how well they comprehended the text above (item 8). Items 9-15 assess students' vocabulary and grammar: students are asked to define the meaning of a word in Sentence 9 (item 9); change a word in Sentence 3 for the word with the same meaning (item 10); find a word in Sentence 1 which has the following morphological parts (which are coded with the accepted in the Russian tradition nomenclature as A □) (item 11);
copy all nouns from sentence 7, define their gender, declension, case, number of one of the nouns (item 12); copy all adj ectives with the nouns they modify from Sentence 1, define gender, declination, case, number of one of the nouns (item 13); copy all verbs from sentence 3 (item 14); describe a life situation in which you can use the proverb 'He who likes skiing downhill must enjoy climbing uphill' (item 15). The proverb is not used in the text. Thus, of the 15 items, only items 6-8 are testing participants' reading comprehension skills. Further, these items are open-ended, which means that they are more time and resource-intensive, making it difficult use this assessment for rapid evaluation (see Tables 1, 2).
(2) Compulsory (or Basic) National Exam (OGE) in Russian is a key element of the Russian system of assessing the quality of education. OGE scores are used to determine if students will have the opportunity to be enrolled in colleges (institutions of secondary vocational education) or proceed to high school. The test lasts 235 minutes.
The writing portion consists of three sections: Part 1 is a written account of a text, Part 2 (items 1-14) includes the following tasks: mark one of the four sentences which provides an explanation to the question ... (item 2); mark a sentences with a metaphor (item 3), in sentences 17 find a word with the prefix with a certain meaning (item 4); in sentences 14-16 find a word with the suffix the spelling of which is an exception (item 5); find a neutral synonym (item 6); substitute the phrase, based on agreement between the adjective and the noun, with a phrase the components of which are connected with governance (item 7); find and copy the clause in sentence 22 (item 8); in sentences 22-24 find one in which unattached attributes are used (item 9); mark the only parenthetical word in the list below (item 10); how many clauses are there in Sentence 51 (item 11); make a list of clauses from the compound sentences below (item 12); in sentences 1-6 find sentences with similar subordinate clauses (item 13); in sentences 4453 find one with an asyndetic and coordinating relations (item 14); Part 3: Essay (item 15).
The OGE mostly assesses students' abilities in lexical access and syntactic parsing. The 'inferencing' and 'building mental model' items (items 1 and 15) that examine reading comprehension are open-ended questions requiring
experts' examination and assessment of student responses. [Demoversii...] (see Tables 2, 3).
(3) EGE (National Unified Examination or Uniform State Exam) in Russian
The EGE is compulsory for high school graduates when they complete secondary education and receive a School Leaving Certificate (Attestat o Srednem (Polnom) Obshchem Obrasovanii). It is a standardized test released uniformly throughout the Russian Federation and rated by independent raters.
EGE in Russian is designed to measure proficiency in Russian regardless of how, when, and how well it has been taught. Thus, the exam has a broad proficiency orientation and its content is not supposed to be tied to any particular language-training program. The Construct of EGE 2019 in Russian defined in "Specification of Assessment Materials" includes 27 items and the following areas: 1. Speech. Text (5 test items).
2. Vocabulary and phraseology (2 items).
3. Speech. Spelling standards (7 items). 4. Speech. Punctuation norms (6 items). 5. Speech. Language norms (5 items). 6. The expressiveness of Russian speech (1 item). 7. The development of speech. Composition (1 item) (see the complete construct of EGE in Table 4). The test is long and takes 210 min [FIPI] (see the Specifications in Table 2, 4).
There are two limitations on using these assessments. The first is practical - the length of these exams renders them inefficient and uneconomical as a short "screening" tool. The second is more theoretically-oriented. While these tests evaluate important aspects of the Russian language, they do not focus on the higher-ordered reading comprehension skills that drive reading comprehension. These gaps drive the necessity to design a Russian Language test that is reliable, valid, feasible, and economical.
Developing the Russian Language Test (RLT)
Stages of the Russian Language Test Developing included the following: (1) determining the purpose of the test, (2) designing test specifications, (3) constructing test items, (4) evaluating and revising test items, (5) specifying scoring procedures, and (6) performing validity (content, construct, and concurrent) and reliability.
The purpose of the Russian language tests is to evaluate students' ability to comprehend grade-appropriate content. RLT5 and RLT9 are designed to promptly assess learners' proficiency for both lower-level skills, such as spelling, and higher-level skills, such as identification of a text genre.
The test constructs in RTL5 and RTL9 were defined to be based on the content and participants' performance [Sigott, 2004]. The development of test content was driven by two postulates. The first is that vocabulary tests and C-tests are valid predictors of reading skills and general language proficiency [Harsch, Hartig, 2016, p. 556; Alderson, 2005]. Both are viewed as being robust for placement and screening purposes [Milton, 2009; Harsch, Schr^er, 2007]. The second is that Russian is a syntactically complex language, which requires the ability to determine, and even change the gender of a noun, the punctuation of a sentence in connection with the context. Thus, one of the major aspects of basic comprehension in Russian is the ability to make a number of grammatical inferences.
With these factors in mind, we developed two grade-appropriate Russian language tests, for 5 th grade and for 9th grade students. Two grade appropriate difficulty levels were developed to examine language proficiency at different points and to serve as future prototypes of a non-leveled Russian language test taken as IELTS regardless of participants' period of schooling.
To evaluate test validity, we used grade-appropriate texts [Ivanov, Solnyshkina, Solovyev, 2018]. We identified appropriate texts by calculating the Flesh-Kincaid Grade level (FKGL). This formula, modified for the Russian language, is: FKGmod (ASL, ASW) = 0.36 x ASL + 5.76 x ASW - 11.97. ASL is the average number of words per sentence, and ASW is the average number of syllables per word [Solovyev, Ivanov, Solnyshkina, 2018]. In the RT5, the texts' FKGL ranged from 4.5 to 6.1. In the RT9, the FKGL ranged from 8.6 to 10.6.
The Russian language test (both RTL5 and RTL9) has two parts. Part I focuses on linguistic range, grammar accuracy, and local comprehension at the levels of decoding (word recognition, lexical access, and syntactic parsing) as well as establishing propositional meaning at the sentence and clause level. Part II in both versions of the test consists of one item only and offers a global comprehension
task in which participants are expected to identify macro-propositions and infer implicit information.
Students have 30 minutes to complete the RLT. Both RLTs are delivered on paper. The tests are administered in a group setting, typically with 20-25 participants in a classroom.
RLT5 and RLT9 comprise instructions on how to complete each item, examples on how to answer the questions, and the item sets. The text passages in RLTs are sampled of authentic sources such as fiction and mass media. Each item has an instruction for activity (e.g., Mark the correct sentence, Fill in the blanks with the derivatives of the word provided, Define the idiom).
All the materials were reviewed from a variety of settings and piloted with more than 50 native speakers of Russian. An initial pilot test of 30 potential items was reduced in the number of tasks to 11 items for the 5 th graders and 13 items for the 9th graders (See Tables 5 and 6).
The final version of RLT5 consists of 11 items and the RLT 9 comprises 13 items. The test items vary within lexical access, syntactic parsing, inferencing and building a mental model. Example items are shown in Table 7.
Item difficulty is measured in terms of the proportion of wrong responses for every item of a given test [Farhady, Jafarpur., Birjandi, 1994; Manual...]. All questions in RLTs are scored analytically with three rubrics: true - score 1, 1 minor error - score 0.5, wrong - 0.
Initial Validation Study
Once the Russian Language Test was constructed and refined, it was important to conduct a more ecologically-valid test. There were three goals of the current study. The first was to identify overall strengths and weaknesses in students' proficiency skills. The second was to pilot this version of the test with Russian students. The third goal was to explore how performance on the RLT might be to individual differences (e.g., gender, age, and general knowledge).
Participants
Participants were 5th grade (n = 81) and 9th grade (n = 94) students recruited from three high schools in Kazan, Russia, and two suburban
schools in Saby, Tatarstan. The participation in the pilot experiment was voluntary and the students were informed that they were free to stop participation at any time. The participants' parents were provided with the information on the purpose, procedures, confidentiality of the data received. The data collection began after the copies of consent forms from parents of all children were filed and the Ethics Committee of Kazan Federal University issued an approval to conduct the research in November 2018. The data were collected in November 2018 through February 2019 in Kazan and Saby, Russia.
Materials
The participants were given the grade-appropriate Russian Language Test and a general knowledge test. The 5 th graders completed the Weschler Intelligence Scale for Children (WISC) the 9th graders - the Weschler Adult Intelligence Scale (WAIS). Both versions of the test are based on the adapted for Russian-speaking respondents and standardized version of A.Yu. Panasyuk [Panasyuk, 1973] and Yu.I. Filimonenko [Filimonenko, Timofeev, 1994]. The Russian version of WISC contains 27 close-ended questions and the WAIS modification of the General knowledge test comprises 29 close-ended questions. WISC and WAIS are scored by giving one point for each correct answer.
WISC and WAIS assess participants' general knowledge rather than topic-specific or theoretical knowledge. The results of the tests provide an estimate of an individual's strengths and weaknesses associated with working memory, processing speed, and long-term memory. We examined the degree to which scores on the WISC and WAIS were related to Russian language proficiency and reading comprehension.
Results
Overall Performance. We first examined overall performance on the RLT. Scores were sectioned into four bands: Band I corresponds to 0-3.5 (0-25 %) tasks completed correctly; Band II is correlated with 4-6 (26-50%) tasks rated correctly, Band III was assigned if the participants performed 6.5-8 (51-75%) items correctly, Band IV reflects students who received 8.5-11 out of 13
(76-100 %). As shown in Figure 1 and Table 8, the 5th graders demonstrated a relatively normal distribution, with most students scoring within Bands II and III, with fewer students scoring in the lowest and highest bands. Figure 2 and Table 9 show a similar distribution for the 9th graders.
Item Analyses. To examine where students excelled and struggled, we analyzed participants' performance on each item of the RLT (see Figure 1, 2) and data on the proportion of students who correctly answered the questions of each item (see Figure 3, 4, Tables 8, 9). Tables 8 and 9 show the proportion of students who answered each question correctly. The majority of the 5th grade students (85 %) were able to accurately identify a correct conjunction. In contrast, only 11 % of the 5th graders were able to accurately convert from active to passive (Table 8). The 9th graders had little trouble identifying the genre of a text (95 %), but had difficulty with both items related to correcting finite and non-finite verbs (5.5 % and 12.5 %; Table 7).
General Knowledge Tests. Tables 10 and 11 present performance on the Russian language and knowledge assessments as a function of gender. The 5 th graders' comprehension and WISC general knowledge scores did not differ as a function of gender (Table 10). In contrast, the ninth graders showed a gender effect for proficiency, such that females scored higher than males (Table 11).
The participants' performance in RT5 presented in Figure 1 shows that all participants successfully managed the test demonstrating above average results. Scores on the Russian Language Test were moderately correlated with the respective general knowledge test scores for both the 5th graders, r = .33, and the 9th graders, r = .38 (Figures 3 and 4).
Discussion and Future Directions
This study tested the Russian Language Test, the first of its kind to assess students' basic Russian skills (e.g., vocabulary knowledge, grammar). Consistent with Zuckerman et al. [Zuckerman, Kovaleva, Kuznetsova, 2013]. The findings suggest that Russian students may need support when moving beyond recitation to deeper comprehension. This study further demonstrates that students are struggling to construct the
necessary grammatical inferences that help to maintain coherence across the text.
In the future, we are planning to iteratively revise the test. One aim is to construct additional items to assess higher-order inferences motivated by traditional discourse theory and situation model building [Kintsch, van Dijk, 1978]. We are also collecting more data to understand how Russian Language Test performance relates to performance on other reading comprehension tasks.
One cautionary note regarding the current study is raised due to the sample being limited to 11-12 year-olds and 15-16 year-olds from only two educational institutions (Kazan and Saby). Hence, further research is needed for other locations and age groups. It will also be of value to investigate the feasibility and accuracy of the Russian Language Test in an online environment. A computerized version of the test would allow for evaluation in broader contexts without a high increase of administration costs and thus afford assessment at scale.
In today's social and educational environments literacy is a crucial skill to both individual security and social well-being. Developing better language assessments helps to inform students' instructional needs and where educators should focus their attention. Our objective is to improve our capacity to better understand Russian language students' strengths and weaknesses across all ages. In turn, we hope to contribute to educators' capacity to address students' needs.
NOTE
1 The experiment on text comprehension and compilation of Russian Textbooks Corpus were funded by the Russian Science Foundation grant number 18-1800436 "Complexity of Texts in the Russian Language". The Russian Test development and conducting psychological tests were financed by the subsidy of the Russian Government to support the Program of Competitive Growth of Kazan Federal University.
The Russian Language Test: Towards Assessing Text Comprehension APPENDIX
Table 1
VPR, Grade 4 (2019). Specifications
№ Score The knowledge and skills assessed
Part 1
1 7 Ability to write a dictation, observing spelling and punctuation norms
2 3 Ability to recognize homogeneous members of a sentence
3 1 Ability to recognize clauses
4 3 Ability to recognize parts of speech
Part 2
5 2 Ability to recognize orthoepic norms
6 1 Ability to classify consonants
7 2 Ability to recognize the main idea in a written text; formulate the main idea in writing, following syntactic and semantic rules
8 3 Ability to draw up a plan of a text, reproduce a text in different forms in writing, following syntactic and semantic rules
9 2 Ability ask questions to a text and write them down
10 1 Ability to determine the meaning of a word by the context
11 1 Ability to select synonyms to eliminate overlaps in a text
12 2 Ability to perform morphological analysis of a word
13 3 Ability to recognize nouns in a sentence, recognize grammatical characteristics of a noun
14 3 Ability to recognize adjectives in a sentence, recognize grammatical characteristics of an adjective
15 1 Ability to recognize verbs in a sentence
16 3 Ability to infer implicit information
Total 38 16 items, max score 38, time = 90 min
Table 2
Item analysis of Russian VPR 4, OGE, EGE
№ Type of activity VPR4 OGE EGE
1 Word (form) recognition Items 1, 4, 5 - Items 4, 9, 10, 11, 12
2 Lexical access Items 9, 10 Items 3, 4, 5, 6 Items 3, 5, 6, 7, 8, 15, 24
3 Syntactic parsing Items 2, 3, 11, 12, 13, 14 Items 7, 8, 9, 10, 11, 12, 13, 14 Items 2, 13, 14, 16, 17, 18, 19, 20, 21
4 Propositional meaning Item 15 - Items 23, 25, 26
5 Inferencing Items 6 Item 2 Item 22
6 Building a mental model Items 7, 8 - -
7 Creating a text-level structure — Item 1 Items 1, 27
Table 3
Compulsory National Exam (OGE) in Russian: Construct and Structure (2019).
Specifications
№ Score The knowledge and skills assessed
1 7 Part 1. A written account of a text
Part 2. Text
2 1 Text as a piece of speech. Semantic and compositional integrity of the text. Text analysis
3 1 Expressive means of the Russian language (tropes, figures of speech)
4 1 Spelling of prefixes
5 1 Spelling of suffixes of various parts of speech
6 1 Vocabulary. Synonyms Classifications of words by origin and use
7 1 Types of subordination in a phrase (agreement, governance, adjoinment)
8 1 Clauses. Ways of expressing subject and predicate
9 1 A simple sentence with homogeneous and detached parts of a sentence (attributes, adverbial modifiers, adjunct and objects)
10 1 Punctuation marks in sentences with words and constructions that are not grammatically related to the sentence members (parenthesis, addresses, clarifications)
End of table 3
№ Score The knowledge and skills assessed
11 1 Types of clauses
12 1 Co-ordinate conjunctions and subordinate conjunctions.Complicated and complex sentences, types of punctuation. Composing and subordinate unions
13 1 Types of subordination in complex sentences. Subordination (homogeneous and heterogeneous) and consecutive subordination
14 1 Sentences with different types of connections between clauses (compound sentence, complex sentence, conjunctionless subordinate clause)
Part 3. Essay
15 10 Literacy and accuracy of an essay are assessed based on 5 criteria
Total 30 -
Table 4
National Unified Exam (EGE) in Russian: Construct and Structure (2019).
Specifications
№ Score The knowledge and skills assessed
1 1 The main idea and theme of a text
2 1 Connectors in a text
3 1 The lexical meaning of the word
4 1 Orthoepic norms (word stress)
5 1 The use of paronyms (lexicology)
6 1 Lexical norms of the Russian language (except paronyms)
7 1 Word derivation (Morphology)
8 Syntactic norms. Norms of agreement and governance
9 1 Spelling of word roots
10 1 Spelling of prefixes
11 1 Spelling of suffixes of various parts of speech (except -H - / - HH-)
12 1 Spelling of inflexional ending of verbs and suffixes of participles
13 1 Spelling of NE- and NI-
14 1 Joined-up writing, hyphenated writing, separate writing of words
15 1 Spelling of -H- and -HH- in suffixes of various parts of speech
16 Punctuation in a compound sentence and a simple sentence with homogeneous parts of the sentence
17 1 Punctuation marks in sentences with detached parts of a sentence
18 1 Punctuation marks in sentences with words and constructions that are not grammatically related to sentence members
19 1 Punctuation marks in a compound sentence
20 1 Punctuation marks in sentences with different types of connectors
21 1 Punctuation analysis of a text
22 1 Text as a piece of speech. Semantic and compositional integrity of the text
23 1 Functional-semantic types of speech (discourse)
24 1 Lexicology. Synonyms Antonyms. Homonyms. Phraseological Units. Groups of words by origin and use
25 1 Text coherence
26 4 Expressive means
27 24 Essay
Total 58 -
Table 5
RLT 5: Construct and participants' performance
№ Text of the item Texts of the task Type of the task Percent Correct
5 There is a spell in autumn early, One all too brief, of an enchantment rare: The nights are radiant and pearly, The days, pellucid, crystal-clear (F. Tutchev) Fill in the blank with the correct conjunctions provided: and, but Syntactic parsing 85.0 %
2 To play .... (a role, significance) Choose the correct word (semantic co-occurrence) Lexical access 82.0 %
11 There are two ks in me, please remember it Please always be as I am I exact, clean and neat Riddle: Identify the meaning of a term Lexical access 58.5 %
3 (put on, dress). a coat Choose the correct word (grammar co-occurrence) Lexical access 45.0 %
9 There landed a rook and probably absolutely unexpectedly he came across a plentiful of worms. Not for herself came (feminine) this:even here one may hear how loudly young are squeaking while waiting for their mother Fill in the correct grammar form of the noun Lexical access, proposition al meaning 45.0 %
10 Take/hold one mind Define an idiom Lexical access 43.5 %
7 It is found at the railways stations, all trains arrive there, it contains double r. what is it? Riddle: Identify the term Lexical access 43.0 %
4 What (time/ of time ) is it? Choose the correct case of the noun Lexical access 41.0 %
1 Mikhail Vasilievich Lomonosov (1711-1765) is a brilliant Russian scientist, writer, teacher. The main work among his linguistic heritage is his book on Russian Grammar (1755), the first research-based grammar of the Russian language, in which the scholar determined the speech norms of his time and laid the foundations of stylistics. Read a text and mark the main idea (MCQ). Building a mental model 39.5 %
6 I am a synonym to the word hospitable, my root refers to an old tradition to treat guests with bread and salt Riddle: Identify the synonym in a text Lexical access 19.0 %
8 The task performing by us, was easy Correct the sentence (active vs. passive participles) Syntactic parsing 11.0 %
Table 6
Percent correct for each item on 9th grade Russian Language Test
Item Percent Correct
1. Identify the text genre 90.5 %
2. Fill in the blank with the correct conjunction 63.0 %
3. Add a comma (identify the clause) 62.0 %
4. Read a text and mark the main idea 56.0 %
5. Substitution (non-finite constructions) 55.0 %
6. Correct the sentence (cohesive devices) 48.5 %
7. Correct the sentence (cohesive devices) 48.0 %
8. Read a text and mark the main idea 45.5 %
9. Substitution (attributive clauses) 38.0 %
10. Add a comma (identify compound sentence) 35.0 %
11. Change the punctuation (non-punctuated attributes) 33.5 %
12. Correct the sentence (non-finite verbs) 12.5 %
13. Correct the sentence (non-finite verbs) 5.5 %
Table 7
Example items from the Russian Language Test with Explanations of the Necessary Comprehension Skill
Test Item (Translated) Explanation
Fill in the blanks with derivatives of the word 'rook'. Прилетел грач и, видимо, вдруг наткнулся на массу червей. Не для себя прилетела эта : слышно и здесь, как орут молодые ... в ожидании матери (М.Пришвин). There landed a rook and unexpectedly he came across a plentiful of worms. Not for herself came (feminine) this : even The default gender of a rook is masculine. There is a cohesive gap between the 1st and the 2nd sentences: the masculine noun in the first sentence (erach, a rook) and the missing word to be filled in the blank in the second sentence refer to the same object. Test-takers are expected to (1) infer that both words refer to the same referent and (2) know the generic feminine noun - ptitsa, a bird. The word in the second blank refers to a group of rooks and a reader is expected to form the derivative accordingly - grachata, young rooks.
here one may hear how loudly young are squeaking while waiting for their mother.
Put punctuation marks choosing between the following: 1. A comma before 'and', 2. A comma after 'and', 3. A comma before and after 'and', 4. No comma. К ночи пошел дождь и подул резкий, студеный ветер (Г. Данилевский). By night it began raining and a strong wind began blowing. The general rule in Russian is that complex sentences consisting of two sets of a subject and a predicate are to be separated by a comma. However, if there is a member of the complex sentence referring to both clauses, then the rule does not apply. Thus, the right answer is 4 (no comma) and students are expected to detect the semantic relation between the adverbial modifier (by night) and the second sentence (a strong wind...).
Correct the sentence: Подъезжая к реке, мы остановили лошадей, быстро соскочили на землю и, наскоро раздевшись, бросились в воду. Arriving at the river, we stopped the horses, quickly jumped down to the ground and, having taken off the clothes, rushed to the water. A reader must infer that none of the actions is possible before one HAS actually arrived at the river.
Histogram.: Russian language (O 11) К С d=,Q8617, p> .20;Lilliefors р-=,Ю
О 2 4 6 8 10
Upper limit (x ■<= border)
Fig. 1. Participants' performance in RT5
Table 8
Participants' performance in RT5
Participants
Band I Band II Band III Band IV
female male female male female male female male
(n = 10) (n = 9) (n = 22) (n = 19) (n = 14) (n = 10) (n = 2) (n = 6)
Russian Language Test 3,00 2,78 4,98 4,89 7,32 6,90 8,50 8,83
Histogram.: Russian language (0-13) K-Cd=,10062, p> .20;Lilliefors p<,01
- Expected norm
45 -■-■......................................................... ■ - -....................................................
■2 O 2 4 6 8 10 12 _Upper limit (x border)_
Fig. 2. Participants' performance in RT9
Table 9
Participants' performance in RT9
Participants
Score Band I Band II Band III Band IV
female male female male female male female male
(n = 6) (n = 9) (n = 25) (n = 20) (n = 16) (n = 6) (n = 9) (n = 3)
Russian Language Test 3,00 2,06 5,02 4,85 7,31 7,50 9,61 9,00
Table 10
Means, Standard Deviations for Test Proportion Scores as a Function of Gender for 5th Graders
Participants
Score Overall Male Female t p
(n = 81) (n = 37) (n = 44)
Russian Language Test (11 items) .48 (.17) .48 (.18) .48 (.16) -0.01 0.99
Weschler Test (WISC) .54 (.11) .56 (.09) .52 (.11) -1.76 0.08
Table 11
Means, Standard Deviations for Test Proportion Scores as a Function of Gender for 9th Graders
Participants
Score Overall Male Female t p
(n = 94) (n = 38) (n = 56)
Russian Language Test (13 items) .44 (.17) .38(. 17) .48 (.16) 2.76 0.01
Weschler Test (WAIS) .46 (.16) .44 (.16) .48 (.15) 1.02 0.31
0.3 0.4 0.5 0.6 0.7
WISC Score
Fig. 3. Russian Language Test Proportion Score as a Function of General Knowledge Test (WISC) Proportion Score and Gender for 5th Graders
Fig. 4. Russian Language Test Proportion Score as a Function of General Knowledge Test (WAIS) Proportion Score and Gender for 9th Graders
REFERENCES
Alderson J.C., 2005. The vocabulary size placement test. Diagnosing foreign language proficiency. London, Continuum, pp. 79-96.
Bowles M., 2008. Standards-Based Assessment in the Native Language: A Practical Guide to the Issues. URL: https://www.edweek.org/media/ maz-guide%20to%20native%20language%20 assessment%20v15-blog.pdf (accessed 1 September 2019).
Demoversii, specifikacii, kodifikatory [Demoversions, Specifications, Codifiers].URL: http://fipi.ru/oge-i-gve-9/demoversii-specifikacii-kodifikatory (accessed 1 September 2019).
Education in the Russian Federation. URL: https:// wenr.wes.org/2017/06/education-in-the-russian-federation (accessed 1 September 2019).
Farhady H., Jafarpur A., Biijandi P., 1994. Test language Skills. From Theory to Practice. Tehran, SAMT Publ.
Field J., 2004. Psycholinguistics: The Key Concepts. London, Routledge (in preparation). Cognitive Validity. Taylor L. ed. Examining Speaking: (Studies in Language Testing). Cambridge, Cambridge University Press / Cambridge ESOL.
Filimonenko Yu., Timofeev V., 1994. Rukovodstvo k metodike issledovaniya intellekta u detey D. Vekslera. Adaptirovannyy variant [Guide to the Methodology of the Study of Intelligence in Children by D. Veksler (WISC). Adapted version]. Saint Petersburg, IMATON Publ. 94 p.
FIPI [FIPM].URL: http://www.fipi.ru (accessed 1 September 2019).
Harsch C., Hartig J., 2016. Comparing C-tests and Yes/ No Vocabulary Size Tests as Predictors of Receptive Language Skills. Language testing, vol. 33, no. 4, pp. 555-575. URL: https://pdfs. semanticscholar. org/bf8e/da858e81a93990c 450f5b0ccd661da6d5dac.pdf (accessed 1 September 2019).
Harsch C., Sch^der K., 2007. Text rekonstruktion: C-Test. Beck B., Klieme K., eds. Sprachliche Kompetenzen: Konzepte und Messungen. DESI-Studie. Weinheim, Beltz, pp. 212-225.
Integratsiya obrazovaniya [Integration of Education]. URL: http: //edumag.mrsu.ru/content/pdf/18-2.pdf (accessed 1 September 2019).
Ivanov V.V., Solnyshkina M.I., Solovyev V.D., 2018. Efficiency of Text Readability Features in Russian Academic Texts. Computational Linguistics and Intellectual Technologies, vol. 17, pp. 277-287.
Ivanova V.P., 2013. Razvitie intellekta kak osnovanie lichnostno-professionalnogo stanovleniya studentov: dis. ... d-ra psihol. nauk [Development of Intelligence as the Basis of
Personal-Professional Formation of Students. Cand. Psychol. sci. diss.]. Moscow. 430 p.
Khoutyz I., 2013. Engagement Features in Russian & English: A Cross-Cultural Analysis of Academic Written Discourse Teachers College, Columbia University Working Papers in TESOL & Applied Linguistics, no. 13 (1), pp. 1-20.
Kintsch W., van Dijk T.A., 1978. Toward a Model of Text Comprehension and Production. Psychological Review, no. 85, pp. 363-394.
Kuchma V.R., Tkachuk E.A., 2015. Gigienicheskaya otsenka uchebnykh tekstov: metodicheskie podkhody i otsenka trudnosti dlya detey obshcheobrazovatelnykh uchebnikov [Hygienic Assessment of Educational Texts: Methodological Approaches and Assessment of Difficulty for Children in Comprehensive Textbooks]. Vestnik RAMN, vol. 70, no. 2, pp. 214-221. DOI: 10.15690/vramn.v70i2.1315.
Manual for Language Test Development and Examining. URL: https://rm.coe.int/manual-for-language-test-development-and-examining-for-use-with-the-ce/1680667a2b (accessed 1 September 2019).
Meyer R.V., 2016. Ob izmerenii slozhnosti uchebnogo teksta po estestvenno-nauchnym distsiplinam [On the Measurement of the Complexity of the Text in Natural Science Disciplines]. Sovremennoe obrazovanie, no. 4, pp. 56-64. DOI: 10.7256/2409-8736.2016.4.19501.
Milton J. Measuring Second Language Vocabulary Acquisition. Bristol, UK, Multilingual Matters, 2009. 228 p.
Opisanie kontrolnykh izmeritelnykh materialov dlya provedeniya v 2019 godu proverochnoy raboty po RUSSKOMU YAZYKU 4 klass [Description of Russian Language Assessment Materials of 2019 ]. URL: https://fioco.ru/Media/Default/ Documents/%D0%92%D0%9F%D0%A0/ VPR_RU-4_Opisanie_2019.pdf (accessed 1 September 2019).
Otsenka didakticheskoy slozhnosti shkolnykh uchebnikov fiziki [Assessment of Complexity of School Physics Textbooks]. URL: https://top-technologies.ru/ru/article/view?id=35582 (accessed 1 September 2019).
Panasyuk A.Yu. 1973. Adaptirovannyy variant metodiki Vekslera (WISC) [Adapted Version of the Methodology of D. Veksler (WISC)]. Moscow, Institute of Hygiene of Children and Adolescents of the Ministry of Health of the USSR. 80 p.
Petrova A.A., Solnyshkina M.I., 2019. Pismennaya kommunikatsiya: tekstovyy i lingvisticheskiy podkhody k teorii i empirii [Writing Communication: Textual and Linguistic
Approaches to Theory and Empirism]. Science Journal of Volgograd State University. Linguistics, vol. 18, no. 3, pp. 276-281.
Pinskaya M.A., 2009. An Analysis of Elementary School Textbooks. Journal of Educational Studies, no. 1, pp. 137-161.
Ponimanie shkolnikami uchebnogo materiala [Comprehension of Learning Material by Students]. URL: http://www.voppsy.ru/issues/ 1986/861/861087.htm (accessed 1 September 2019).
Research Notes. URL: https://www.cambridgeenglish. org/Images/23150-research-notes-31.pdf (accessed 1 September 2019).
Sigott G., 2004. Towards Identifying the C-Test Construct. Frankfurt am Main, Lang.
Solovyev V., Ivanov V., Solnyshkina M., 2018. Assessment of Reading Difficulty Levels in Russian Academic Texts: Approaches and Metrics. Journal of Intelligent & Fuzzy Systems, vol. 34, no. 5, pp. 3049-3058.
Solovyev V., Solnyshkina M., Gafiyatova E., McNamara D., Ivanov V., 2019a. Sentiment in Academic Texts. 24th Conference of Open Innovations Association FRUCT, vol. 2019-April, pp. 408-414.
Solovyev V., Solnyshkina M., Ivanov V., Batyrshin I., 2019b. Prediction of Reading Difficulty in Russian Academic Texts. Journal of Intelligent & Fuzzy Systems, vol. 36, iss. 3, pp. 1-11.
Weir C.J., 1993. Understanding and Developing Language Tests. London, Prentice Hall. 341 p.
Weir C.J., Khalifa H., 2008. A Cognitive Processing Approach Towards Defining Reading Comprehension. Research Notes, Cambridge ESOL, no. 31, pp. 2-10.
Zuckerman G.A., Kovaleva G.S., Kuznetsova M.I., 2013. Between PIRLS and PISA: The Advancement of Reading Literacy in a 10-15-Year-0ld Cohort. Learning and Individual Differences, no. 26, pp. 64-73.
Information about the Authors
Kathryn Soo McCarthy, Assistant Professor, Department of Learning Sciences, Georgia State University, P.O Box 3978, 30602 Atlanta, GA, USA, [email protected], https://orcid.org/0000-0002-6277-7005
Danielle Siobhan McNamara, Professor, Department of Psychology, Arizona State University, Payne Hall, TEMPE Campus, Suite 108, 1104 Tempe, USA, [email protected], https://orcid.org/0000-0001 -5 869-1420
Marina I. Solnyshkina, Doctor of Sciences (Philology), Professor, Department of Theory and Practice of Foreign Languages Teaching, Kazan (Volga Region) Federal University, Kremlevskaya St., 18, 420008 Kazan, Russia, [email protected], https://orcid.org/0000-0003-1885-3039
Fanuza Kh. Tarasova, Doctor of Sciences (Philology), Associate Professor, Head of the Department of Romance and Germanic Philology, Kazan (Volga Region) Federal University, Kremlevskaya St., 18, 420008 Kazan, Russia, [email protected], https://orcid.org/0000-0001-9420-4503
Roman V. Kupriyanov, Candidate of Sciences (Psychology), Associate Professor, Department of Social Work, Pedagogical Sciences and Psychology, Kazan National Research Technological University, Karla Marksa St., 68, 420015 Kazan, Russia, [email protected], https://orcid.org/0000-0001-9794-9607
Информация об авторах
Кэтрин Су МакКарти, доцент кафедры теории и методики преподавания, Университет штата Джорджия, п/я 3978, 30602 г. Атланта, Джорджия, США, [email protected], https://orcid.org/0000-0002-6277-7005
Даниэль Шиврон МакНамара, профессор кафедры психологии, Университет штата Аризона, Пэйн Холл, Кампус TEMPE, комната 108, 1104 г. Темпе, США, [email protected], https://orcid.org/0000-0001-5869-1420
Марина Ивановна Солнышкина, доктор филологических наук, профессор кафедры теории и практики преподавания иностранных языков, Казанский (Приволжский) федеральный университет, ул. Кремлевская, 18, 420008 г. Казань, Россия, [email protected], https://orcid.org/0000-0003-1885-3039
Фануза Харисовна Тарасова, доктор филологических наук, доцент, заведующая кафедрой романо-германской филологии, Казанский (Приволжский) федеральный университет, ул. Кремлевская, 18, 420008 г. Казань, Россия, [email protected], https://orcid.org/0000-0001-9420-4503 Роман Владимирович Куприянов, кандидат психологических наук, доцент кафедры социальной работы, педагогики и психологии, Казанский национальный исследовательский технологический университет, ул. Карла Маркса, 68, 420015 г. Казань, Россия, [email protected], https://orcid.org/0000-0001-9794-9607
