CC BY-ND
175
National Research University Higher School of Economics Journal of Language & Education Volume 4, Issue 1, 2018
Kovalenko, M. (2018). The Validation Process in the IELTS Reading Component: Reading Requirements for Preparing International Students. Journal of Language and Education, 4(1), 63-78. doi:10.17323/2411-7390-2018-4-1-63-78
The Validation Process in the IELTS Reading Component: Reading Requirements for Preparing International Students
Marina Kovalenko
National Research University Higher School of Economics
Correspondence concerning this article should be addressed to Marina Kovalenko, National Research University Higher School of Economics, Malaya Pionerskaya, 12, Moscow, Russian Federation, 115054. E-mail: [email protected]
Although IELTS is coordinated under a framework for test development and validation, there is some controversy about exam results' correlation with students' post-admission intellectual, academic and professional performance. The theoretical part of the research aims to investigate the extent to which the IELTS reading component relates meaningfully to interpretations of validity. The empirical part addresses questions about perceptions of the impact of the IELTS reading preparation on adjustment to the challenges of academia and further academic performance and variances in these perceptions depending on the area of study and the level of language mastery. While having quite different views on assessing IELTS validity, the researchers agree that academic success is enhanced through and based on extensive substantive reading. The methodology relied on both qualitative and quantitative data derived from an anonymous online questionnaire: 133 international students with Russian citizenship, Global Education Programme (GEP) finalists, participated in the survey in summer 2017. Five different result interpretations were taken into account: overall results, those for sciences and humanities, higher and lower achievers' results. The discussion is built around test-takers' opinions on the IELTS exam, the reading component and scores. The issues discussed include, but are not limited to: reading strategies, information sources required at university, tasks effectiveness, exam preparation usefulness to academic adjustment and its influence on academic achievement, its resourcefulness for the formation of linguistic capabilities, and respondents' perception of extra factors for exam success. Potentially increasing jeopardy of negative washback is shown as an emerging problem. Although test-taking ability is not depicted as a crucially important factor affecting exam success, it is increasingly significant and its harmful effects may be expressed in illusionary higher levels of validity due to visually improved results. Quality preparation for the reading test can train a number of essential skills required in academia; however, preparation itself does not appear to be a significant factor for smoother adjustment to academic challenges, as it is highly dependent on preliminary linguistic background. There is a necessity to communicate broader information to learners through the IELTS handbook, website and other communication channels. EAP tutors should encourage their students to make efforts to cover the subject without framing it within boundaries of measurement, but with a clear understanding of future academic and professional challenges.
Keywords: IELTS reading component, academic reading, interpretations of validity, higher and lower achievers
IELTS is supposed to assess skills for following instructions, finding the main ideas and the relationships between them, identifying the underlying concept, and drawing logical inferences. According to Alderson (2000), IELTS's strength is in using multiple
methods of text understanding of any passage as in real life readers typically respond to reading texts in many ways (p. 206). By tradition, there have been two main approaches to the nature of the reading process. Bottom-up "information processing" focuses on the
This article is published under the Creative Commons Attribution 4.0 International License.
processing of more micro-level constituents of texts such as letters, words, phrases, sentences. Top-down "analysis-by-synthesis" centers around macro-level principles such as genre, text structure, as well as the role of background schematic knowledge (Moore et al., 2012, p. 8).
Weir (1993) proposes four types or levels of reading: reading expeditiously for global comprehension (summarize the core ideas independently and distinguish a specter of texts related to the subject), reading expeditiously for local comprehension (primary comprehension of core ideas of a text), reading carefully for global comprehension (identify the objective of an author, critically evaluate the ideas in a text and highlight ideas from various texts to maintain one's own reasons), reading carefully for local comprehension (understand an idea with the objective of using it) (Alderson, 2000; Grabe, 1999, as cited in Liu, 2010, p. 155; Moore et al., 2012, pp. 40-41). Urquhart et al. (1998) distinguish between five kinds of reading: scanning, skimming, search reading, careful reading and browsing. Careful reading appeared to be favored by many educationists and psychologists to the exclusion of all other types (Liu, 2010, p. 155). They noted the defining features of careful reading (global, i.e. obtaining an overview of the text in skimming versus local, i.e. locating specific information within a text in scanning): attempts to handle the majority of information in the text in a non-selective manner and build up a macrostructure, adopt a submissive role and accepts the writer's organization.
In its entirety, the absence of necessity to have background proficient knowledge on the given subject is regarded as a strong advantage and a justification of the IELTS claim to be unbiased. However, according to Khalifa and Weir (2009) knowledge stored in long-term memory indicates the efficiency of the reading process, which is enhanced from both "literal" and "interpretative" types of engagement with the text. This creates a remarkable limitation of the test itself, which is balancing between measuring higher order skills that require only a certain amount of creativity in order not to "over-interpret" the text and go far beyond it, but not too far from just "going outside it".
There is a certain amount of inconsistency and controversy about the exam results, mainly their correlation with student intellectual, academic and professional potential. Research indicates that the IELTS test "did not accurately predict test-takers' competence in the academic activities expected for university study" (Hyatt & Brooks, 2009, p. 35). Concerns emerged around the qualitative value of judgments made on a global score and their influence on post-admission academic performance. A vague description of the context behind the band score was highlighted and "a desire for a more in-depth evaluator
picture" (Hyatt & Brooks, 2009, p. 35) was expressed as potentially beneficial.
Although IELTS scores are supposed to broadly predict student language behavior in academic contexts, another paper indicates that "a number of factors must be considered, particularly with regard to ... students whose language proficiency meet, but do not exceed, required entry levels" (Ingram & Bayliss, 2007, p. 54). Balancing the need for income, course quality (content, delivery and grading) and reputation with the need to be competitive, was stated as a difficult task for universities (Ingram & Bayliss, 2007, p. 54).
IELTS-related research activities are coordinated under a framework for ongoing test development and validation. Cambridge English Language Assessment has responsibility for specific research and development (ielts.org, IELTS Academic Reading description). Test data is regularly analyzed to ensure that IELTS remains fair and unbiased and that it encourages, reflects and respects international diversity.
The theoretical part of the research aims to investigate the extent to which the IELTS reading component relates meaningfully to interpretations of validity. The empirical part addresses the following research questions:
1. What are the perceptions of the impact of IELTS reading preparation on the adjustment to the challenges of academia and further academic performance?
2. How different can these perceptions be depending on the area of study and the level of language mastery?
Hypotheses: Questions 7 and 10 were supposed to correlate notably with each other. Exam performance was expected to affect the amount of skepticism of the reading component and preparation effectiveness. The area of study was expected to influence mainly question 5, asking about the most relevant information sources at university. Generally, all the questions were anticipated to clarify test-takers exam perceptions and their commentaries were gently requested in order to shed light on the quantitative data outcomes.
Materials and Methods
In the past, models of reading have usually been developed with only careful reading in mind (Hoover and Tunmer, 1993; Rayner and Pollatsek, 1989, as cited in Weir et al., 2009). However, some IELTS tasks (i.e., section-summary match and gapped summary)
require skimming skills, which are not based on careful reading models. Khalifa and Weir (2009) suggest that along with comprehension the speed of reading is of significant importance. Weir et al. (2009), found that "for many readers reading quickly, selectively and efficiently posed greater problems than reading carefully and efficiently" (p. 162) and drew attention to the underestimation of the cognitive processing required to carry out test tasks by the majority of researchers.
Weir et al. (2009) identify potential limitations of the reading component in that "the major focus of... IELTS... test appears to be on careful reading" (p. 178). Their survey data suggest the necessity of expeditious skills and strategies for university students and the urgency of engaging both strategies in processing large amounts of information. A literature review can be a clear example of engaging various strategies. At the first stages, while careful searching is required within a restricted timeframe, students have to be selective and possess rapid reading skills. This can be exemplified in the Trinity College Dublin Master in Education handbook in recommendations for writing assignments: "A small number of well-chosen sources, which can be carefully analyzed, are preferable to a lengthy description of a broad range of literature (p. 39). Thus, not time- and length-consuming descriptions, but a "careful" approach to literature searching through scanning and then skimming appears to be an essential preliminary stage to further careful reading, promising insightful perceptions.
These strategies are hard to teach and learn in high school, as they require a trial and error method and thorough feedback as a means of assessment for learning. In addition, washback appears to be extremely harmful in practice. The test can narrow the text to an artificial structure, as its results must not be misinterpreted in order to reach a higher level of validity. Special concern should be pointed out for future students of the arts, humanities and the social sciences, as they definitely need to develop a critical literacy paradigm and engage in texts interpretatively, which requires the ability to cover, evaluate and summarize numerous sources of information as well as create authentic pieces of writing based on extensive reading. This ability goes far beyond test-wiseness and should be enhanced through high-quality teaching and learning practices.
Validation Process of Test Assessment Results
"Validity refers to the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests" (Plake & Wise, 2014, p. 11). It is not validity itself but the constant process of validation that is supposed "to provide a sound specific
basis for the proposed score interpretations" (Plake & Wise, 2014, p. 11). Thus, these interpretations should be evaluated. Nitko & Brookhart (2007) define validity as, "the soundness of ... interpretations and uses of student assessment results" (p. 38). They emphasize the requirements for evidence from a variety of sources that demonstrate these interpretations. The uses should be adequate and no serious negative consequences should follow as a result.
Principles of validation identified by Messick (1989, 1994, as cited in Nitko and Brookhart, 2007, p. 38) should be based on evidence, interpretation and use of test results supported by appropriateness and correctness. The consequences of the assessment results should be coherent with assigned values. Along with knowledge, achievement assessments should require important thinking processes and combinations of skills and knowledge to work on "real-life" applications as defined by the curriculum framework and state standards. Appropriate tasks should require higherorder thinking processes and skills, as the ability to solve sophisticated problems is vital for academic progress. The requirements that illustrate these points are emphasized by Nitko and Brookhart (2007), who include a detailed description of the processes and abilities being assessed; a clear demonstration of tools and measurements for the assessment and evidence of expected thinking processes and skills application for successful assignment implementation (p. 50).
Kane & Bejar (2014) draw attention to a developmental model of academic performance, "where the achievement levels are intended to represent qualitatively different levels of sophistication in the discipline" (p. 120). The cognitive model is also exemplified as an alternative, indicating a student's current state of mastery versus non-mastery of a topic. According to the cognitive model, individual learning progression should be measured by its task-requiring performances, indicating the level of achievement students are capable of mastering (Kane & Bejar, 2014, p. 120). The IELTS scale introduces results by the same principles: from a non-user (0 score, non-mastery) through to modest and competent users (5-6 scores) up to an expert user (9 score, mastery) with 0.5 increments (the IELTS scale). O'Loughlin (2012) points out that after reading the IELTS Guide (2009), all stakeholders generally find it to be informative. However, some wished "it could have included more information about the meaning and interpretation of IELTS test scores" (O'Loughlin, 2012, p. 34).
For instance, the level of independent functioning in an unfamiliar language environment and the level of operational command in the English language (the IELTS scale) as grading criteria cannot measure the full capacity of language mastery in a restricted timeframe precisely. There are a number of other components
that contribute to success, including: luck, guessing, anxiety levels, the speed of decision-making and test-taking capacity. In contrast, the absence of these components risks a decreasing chance for positive outcomes which can be illustrated by lower grade indicators such as occasional inaccuracies and the frequency of misunderstanding of problems (the IELTS scale). Consequently, the fluency between grades, even with 0.5 increments appears to be noticeable and inevitable. Kane & Bejar (2014) point out that, "large-scale assessments. are designed to produce scores that reflect each student's position on some continuum reflecting overall achievement in some domain" (p. 122). While understanding the practical use of "assessment of learning", which IELTS turns out to be, these authors question the limitations of the resourcefulness of these systems as "assessment for learning".
However, the differences between the levels linguistic capabilities of applicants to higher education are presented as rather vague assumptions, i.e. "some (Level 6) misunderstandings" or "occasional (Level 7) inaccuracies, inappropriacies and misunderstandings", disinguish a competent user from a good user. Ingram and Bayliss (2007) regard it as a difficulty for "university admissions staff and faculties to determine whether students are linguistically equipped to fulfill the task requirements of study disciplines" (p. 54). They also highlight the total absence of any applicants' linguistic uniqueness, as IELTS scores provide only generic comments. Examples of abilities to "write 3,000-word discursive essays" or "understand culturally-specific lectures or subject-specific journal articles, medical histories or legal arguments" (Ingram & Bayliss, 2007, p. 54) are not reflected in scores. This restricts the predictability of a student's academic trajectory. Kane and Bejar (2014) also offer to include differentiation within levels to distinguish between clear mastery and approaching mastery as evidence is required "to the attributes used to characterize each student's achievement" (p. 120).
These achievements tend to be strongly hierarchical and consistent. The researchers do not take into account that there may be exceptions for adjacent levels, which is more relevant to humanities than STEM subjects. The researchers' critique raises the question of IELTS validity in terms of its main mission - providing an indication of student capability in continuing further education in the non-native linguistic environment. Moreover, IELTS claims not to assess any discipline-specific knowledge gained for the exam preparation purpose that can be a strong argument against these kinds of "accusations".
Although assessments "for learning" and "of learning" are supposed to be interrelated, this is not always the case, especially when there is the obvious
pursuit of visually accountable results. Messick (1996) highlights the phenomenon of washback as a huge influence on language teaching and learning as "teachers might come to overemphasize those constructs that are well-represented [on the test] and downplay those that are not" (p. 252).
Washback is regarded as deleterious when there is a serious disconnection between a test's construct of reading and the broader demands of real world perception through language. It is essential for test developers claiming their tests are valid to strive to decrease construct underrepresentation and construct-irrelevant variance. Both test-developers and teachers could benefit from providing relevant resources to enhance test framework integration via the teaching and learning process, taking into account test-wiseness and other artificial restrictions on students' representations. In contrast, valid low scores resulting from poor teaching or limited opportunities to learn are not supposed to be test makers' responsibility. Such adverse consequences of validating assessment represent problems not of measurement, but rather of teaching and of educational or social policy.
Previous Study. Analysis of IELTS Academic Reading Component Validity
Moore et al. (2012) researched the IELTS academic reading construct validity. This study, which was awarded a grant in 2007, is an official IELTS report, is of high quality and its implications seem realistic and thought provoking. A decent level of criticality was noticeable, i.e., findings revealed that the majority of tasks required "mainly a basic comprehension of relatively small textual units" (Moore et al., 2012, pp. 2, 37).
In order to reinforce and encourage reading mastery, the subject-specific claim of becoming deconstructive readers was expressed as desirable (Moore et al., 2012, p. 59). The ways in which the meaning is created, how the words used in a text can carry particular nuances and how images create special impacts were mentioned as features of a sensible reader every student is supposed to emulate. Arguing against texts as "repositories of information and facts" and expressing the need for a definite seeing and constructing the world (Moore et al., 2012, p. 62) are an epistemological challenge for reading skill formation.
The lecturers that participated in the research commented most favorably on the relationship between the IELTS reading component and study on their courses and identified a relatively strong interdependence between some task types, indicating the multiple-choice format as the most common for tests. Another important finding for several informants was that the test showed "unexpected complexity" in
the science-based articles that required to be read more than once (Moore et al., 2012, p. 60). Khalifa and Weir (2009), indicating that efficiency of the reading process is largely enhanced by knowledge stored in the long-term memory, can support this.
Participants
The anonymous 10 question survey was conducted on the SurveyMonkey platform with 133 postgraduate (master and PhD level) Global Education Programme (GEP) finalists (global education, official documents) in July and August 2017. GEP is an innovative government funded programme offering Russian citizens an opportunity to do full-time courses at leading foreign higher education institutions. The most common countries for study are Australia, the UK, the USA and Germany. Russian citizenship is the compulsory condition of sponsorship. GEP finalists were selected for the research as they represent a group of contemporary Russians who were purposefully funded to study at universities with high expectations from students. The majority of the finalists had to take the academic module of IELTS (some universities accepted other language proficiency certificates) in order to get a non-conditional offer of acceptance.
According to the conditions of GEP, finalists are offered to choose the five priority areas of training, potentially beneficial to the challenges of the Russian economy. Overall, 79 respondents (59.4%) represented scientific specialties (Engineering, Health care and Science) and 54 respondents (40.6%) represented humanities (Education and Social work). All the participants took the exam no longer than 4 years before the time of the research; the vast majority of them took the exam in summer 2016. 72 participants were female and 61 were male. The age range was 2231 years old. 24 participants were PhD students, while 109 were doing their Master degree. Age, gender and the study level were known beforehand via both the GEP handbook and online profiles. They were not requested in the survey, as they were not considered to be of any particular significance.
Research Design
The research design relied on both qualitative and quantitative data derived from the survey results. Five different result interpretations were taken into account, where overall results were unfiltered. Two interpretations were filtered by 01: "Your specialty" where five areas of training were divided into sciences and humanities. Two second ones were filtered by respondents' exam scores, requested in 02, "What is Figure 2. Overall results for Questions 2 and 3.
There is an impressive diversity in reading results:
your IELTS overall score?" and Q3, "What is your IELTS reading score?" (5.5, 6.0 and 6.5 versus 8.0, 8.5 and 9.0). There were 19 (14.29%) higher IELTS achievers and 26 (19.55%) lower achievers.
Procedure
All the participants were sent personal emails offering consent forms and inviting them to consult with the author in case of difficulties with formulation clarity or emerged interest in the subject matter. Everybody had open access to the survey intermediate outcomes, which appeared on the screen after the survey was finished. The 10 compulsory survey questions featured three question types (see Appendix A) including matrix grading scale, multiple choice and scale range labels. All matrix grading scale labels and multiple choice questions included an extra option, Other (please specify). The survey took 5-15 minutes depending on the respondents' enthusiasm to fill in the option Other. Matrix grading scale labels and multiple choice questions allowed choice from either many options or the only one option from a set. Survey select rate provided the opportunity to download question summary data in 3 formats: PDF, PPT or Excel. Options for analysing individual responses and question summaries as well as filtering and comparing answers were used extensively in order to identify and interpret trends.
Results
Test-Taker Exam Scores
The first three questions requested general information about the respondents including their specialization and IELTS overall and reading component scores (Figures 1 and 2). The correlation between reading scores and overall scores was investigated by comparing and individual responses analysis.
Your specialization
30% 25% 20% 15% 10% 5% 0% 32% 30% 20% 11% 8%
Education Engineering Health care Science Social work
Figure 1. Specialization of all the respondents.
they range from 5.5 while overall scores are a lot more
IELTS overal score
5 5,5 6 6,5 7 7,5 8 8,5 9
IELTS reading score
20%
15%
10°%
5°%
0%
_I I J_ljp_
5 5,5 6 6,5 7 7,5 8 8,5 9
localised at 6.5, 7.0 and 7.5 (Figure 2). The respondents with 5.5, 6.0 and 6.5 overall received from 5.5 (one Engineering respondent) to 9.0 for the reading component. Although lower achievers range was broader (Figure 3) than higher achievers range (Figure 4). In the vast majority ofcases, reading scores are higher than overall scores by 1 band, as individual response analysis clearly illustrates. If reading results are lower than overall, the difference still lies within 1 band with extremely few exceptions (see the right picture on Figure 3). Comparing sciences and humanities exam performances there are no noteworthy differences in scores, although a wider range of higher overall scores is more noticeable for humanities respondents, while 1/3 of sciences respondents achieved 7.0, which is a purely average result.
Reading Strategies Most Applicable to Academic Challenges
The fourth question examined the applicability of the five kinds of reading strategies (scanning, skimming, search reading, careful reading and browsing) identified by Urquhart et al. (1998) to academic challenges. All the respondents chose search reading and scanning as "the most applicable" reading strategies. Careful reading and skimming appeared to be "very applicable", although fewer respondents identified them as "the most applicable". Browsing appealed to the respondents as well: 72.44% of them thought that it is either "somewhat applicable" or
"very applicable".
Higher achievers were remarkably more generous in evaluating the importance of all the strategies. There were much fewer opting for "not very applicable" and "not applicable". None of the respondents considered search reading to be "not very applicable" or "not applicable" and only 5.26% said that scanning was "not very applicable". A slight underestimation of skimming, careful reading and browsing was also of little significance. Lower achievers tended to be noticeably more skeptical about all the strategies: their most preferred option was "somewhat applicable" while "the most applicable" was the third prevalent option. Overall results demonstrate similar results for higher and lower achievers. The most widespread definition for all the strategies is "very applicable", "somewhat applicable" comes second" and "the most applicable" third. Some respondents also chose underestimating options, but their proportion does not neglect the significance of any strategy. The specialty did not affect respondents' choices to any worthy of note extent.
Information Sources Needed at University
The fifth question intended to clarify how relevant to academic studies 10 types of information sources, derived from the author's observation, were. The choices of higher and lower achievers were only slightly different from the overall responses. The four most demanded sources were the same: journal
IELTS overal score IELTS reading score
80% 40% ■
40% -. I- 20% -1 ■ 1-
20% 0% -1 l- 10% 0% .Ilia _
5 5,5 6 6,5 7 7,5 8 8,5 9 5 5,5 6 6,5 7 7,5 8 8,5 9
Figure 3. Overall scores for Questions 2 and 3 for lower achievers.
Figure 4. Overall scores for Questions 2 and 3 for higher achievers.
articles, online sources, course materials and textbooks with markedly lower primacy. Legal documents, social network news and public press appeared to be the least informative sources as approximately half of respondents said that they "quite rarely" or "hardly ever" use them. Personal correspondence demonstrated the most dissimilar reaction as, roughly speaking, each of five time options was chosen by 1/5 of all the participants.
Journal articles were the most valuable source for humanities respondents. They used online sources and course materials consistently, but with a lower level of frequency. However, respondents from the sciences indicated course materials, online sources and journal articles as three substantial sources. Approximately 75% of sciences respondents said that they use them "very often" or "quite often". Reference books and textbooks come fourth and fifth which indicates that almost all the respondents use them with a stable level of frequency.
Tasks Effectiveness for Academic Reading Preparation
The sixth question asked the extent of the effectiveness of the nine task types. Several websites, including but not limited to Take IELTS with British Council or IELTSEssentials, were consulted in order to choose these types carefully. All task types were highlighted as effective with a relatively small variance
in their importance. Matching causes and effects, summary completion and identifying the writer's views were chosen as the most effective tasks by a noticeably larger proportion of respondents.
All the other tasks were marked mostly as "somewhat effective". Although very few respondents marked these task types as "not very effective", their number was still larger than the number of people who negatively estimated the first three most effective tasks. Both humanities and sciences bar charts illustrate a very similar picture with a very coherent evaluation of all the tasks as "very effective" and "somewhat effective" by sciences respondents and more inconsistency and inclination to the option "somewhat effective" by humanities respondents.
IELTS Preparation Usefulness to Academic Adjustment and its Influence on Academic Achievement
Questions 7 and 10 featured relatively close issues: IELTS preparation usefulness to academic adjustment (7) and its influence on academic achievement (10). Overall, respondents were quite appreciative of IELTS reading preparation. 30.53% of participants said that preparation was "somewhat helpful" along with 26.72% and 10.69% of them estimating noticeable and significant usefulness respectively. The other 30% were not very grateful for this preparation or filled in the Other option. Education respondents' answers
Which of the reading strategies are the most applicable to your academic challenges?
■ ■
Scanning Skimming Search reading Careful reading Browsing
The most applicable ■ Very applicable ■ Somewhat applicable Not very applicable Not applicable
Which of the reading strategies are the most applicable to your academic challenges?
. .
к
Scanning Skimming Search reading Careful reading Browsing
The most applicable ■ Very applicable ■ Somewhat applicable Not very applicable Not applicable
U
Figures 5 and 6. Reading strategies estimated by lower and higher IELTS achievers.
10
20
30
40
50
60
70
Journal articles
Online sources
Cource materials...
Reference books
Social network news 4
Public press 3
36 12 32
22 16 13 1
18 16 9 2 6
18 16 19
9 16 18 6 3
7 10
15 18 3
9
10 11 10 11
4 13 13 11 11
3 9
20 10 10
3 8
18 16 7
Very often ■ Quite often ■ Sometimes ■ Quite rarely ■ Harly ever
0 10 20 30 40 50 60 70 80 90
Cource materials...
40 24 9 3 3
Online sources
36 27 10 4 2
Journal articles
37 21 14 4 3
Textbooks
20 23 17 11 8
Reference books 10
20 3 0 12 №
Official repotrs
4 27 22 16 10
Personal correspondence 10
18 17 12 21
Legal documents 4 1]
26 22 16
Social network news
8 13 12 2 4 22
Public press 3 13
18 5 2 2 23
Very often ■ Quite often ■ Sometimes ■ Quite rarely ■ Harly ever
Figures 7 and 8. The frequency of access to informational resources of humanities and sciences respondents (absolute numbers are featured instead of percentages to enhance better readability).
0
7 1
were more narrowly focused on the "somewhat helped" option (39.62%), while sciences representatives' responses were a lot more diverse and more positive about preparation - 30.77% of them specified noticeable help.
The difference in the perception of IELTS preparation usefulness was significant between higher and lower achievers. Higher achievers were markedly more sceptical about the preparation usefulness with 10.53% even choosing the most negative option. 57.89% of them admitted that preparation was helpful to a certain extent (first three options), while this number was 88.46% for lower achievers. Moreover, none of lower achievers thought that the preparation was absolutely "not helpful" and only 7.69% of respondents considered it a bit helpful.
Question 10 demonstrates a similar trend. The overall view of IELTS preparation resourcefulness to enhance academic achievement is 57%. While higher achievers appeared to be a lot more sceptical (47%), lower achievers tended to be significantly more thankful to the preparation (65%). The difference between sciences and humanities respondents is smaller: 60% versus 52% respectively. To summarise, Questions 7 and 10 indicate that the level of achievement is crucial to the perception and evaluation of exam preparation effectiveness: the higher the achievement, the more sceptically respondents act and the lower they tend to evaluate the impact of preparation on adjustment to the challenges of academia and further academic performance.
Respondents' Perceptions of Extra Factors of Exam Success
Question 8 was used to shed light on respondents' perception of extra factors potentially affecting their exam score. Good concentration was highlighted as the priority factor, speed of decision-making was second and anxiety level was third. Test-wiseness was also appreciated, although its importance was mostly "noticeably affected" or "somewhat affected". Overall, respondents felt less enthusiasm about guessing and luck with 16.54% of them rejecting the importance of guessing and 19.55% the importance of luck. 28.57% and 37.6% of respondents said guessing and luck as "a bit effective" respectively.
Although there is no significant difference between humanities and sciences respondents, higher and lower achievers demonstrated markedly different results. Lower achievers tended to evaluate all the factors, i.e. 19.23% and 11.54%, lower achievers pointed out that guessing and luck significantly affected their performance along with only 11.54% and 15.38 of them thinking that it did not affect their score. In contrast, higher achievers demonstrated a lot more disbelief towards these two factors: 36.84% and 47.37% of them fully rejected the influence of guessing and luck on their exam score respectively and none of them took their probable impact on exam scores seriously.
IELTS Preparation Resourcefulness for the Formation of Linguistic Capabilities
Question 9 intended to uncover respondents' opinions on the IELTS preparation effectiveness for forming the capabilities only partially assessed by IELTS itself, according to the findings of Moore et al. (2012). The ability to see what the author is trying to achieve through the text, the capability to
60,00% 50,00% 40,00% 30,00% 20,00% 10,00% 0,00%
Which IELTS reading tasks, in your opinion, are the most effective in preparing students for academic
reading challenges?
g_
■
■ 1 1 1
1 ■ 1 1 . 1 1 1 I 1 1 ■ 1
Summary Matching Identifyng Multiple Selecting comlpetion headings the writer's choice to... views
Table Matching Sentence Short factors completion causes and completion answer effects questions
Other
I The most effective ■ Very effective
Somewhat effective
A bit effective ■ Not very effective
Figure 9. Overall indicators of tasks effectiveness for academic reading preparation.
To what extent IELTS reading preparation helped you to adjust to academic
challenges?
35% 30% 25% 20% 15% 10% 5%
0%
Significantly helped
Noticeably Somewhat Helped a bit Was not helpful Other (please,
helped helped specify)
To what extent IELTS reading preparation helped you to adjust to academic
challenges?
50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0%
Significantly helped
Noticeably helped
Somewhat helped
Helped a bit Was not helpful Figures 10 and 11. IELTS preparation usefulness according to higher and lower achievers.
Other (please, specify)
understand deeply and interpret original sources of information and the ability to identify concepts in the reading for the purpose of applying them to another context emerged as the most desirable skills to all the respondents.
Three other abilities were appreciated as well, with only a slight drop in choosing "significantly resourceful" option. Enhancing creative writing development through various genre reading and reinforcing freedom to disagree with respected authors and developing of solid arguments were estimated by the majority of respondents as "somewhat resourceful" and "a bit resourceful", which may indicate the subject
specificity for these advanced skills. There were no notable differences between humanities and sciences respondents, but lower achievers showed remarkably more interest in all these skills than higher achievers. 7.69% of lower achievers said, "reinforcing the freedom to disagree with respected authors..." as "not very resourceful" while higher achievers were less generous in appreciating them and 4 abilities were considered "not very resourceful" by 1 (5.26%) or 5 (26.32%) respondents.
Discussion
Which IELTS reading tasks, in your opinion, are the most effective in preparing students for academic
reading challenges?
60,00% 50,00% 40,00% 30,00% 20,00% 10,00% 0,00%
ll
Good concentration
Appropriate anxiety level
High speed of decision making
Test-wiseness
Guessing
Luck
■ Significantly affected ■ Noticeable affected ■ Somewhat affected ■ Affected a bit ■ Did not affect Figure 12. All respondents' perception of extra factors of exam success.
Interpretations of the results are still a very arguable issue. On the one hand, they require more explicitness and guidance in their practical application. On the other hand, striving for both can lead to overinterpreting the scores, as the latter do not have enough potential to predict further trajectories of linguistic development. More clarity should be imposed on the meanings of the scores for the text-takers in order to prevent their probable disapprobation. However, according to the findings, the IELTS reading component generally succeeds in integrating numerous methods of text understanding and appears to be generally valid.
While having quite different views on assessing IELTS validity, all the researchers agree that academic success is enhanced through, and based on, extensive substantive reading. Potentially increasing the jeopardy of negative washback is an emerging problem. Although test-wiseness was not depicted as a crucially important factor affecting exam success, it is hard to deny that it is increasing significantly and its harmful effects may be expressed in illusionary higher levels of validity due to visually improved results. In contrast, it can distract from the key positive aim of validity, which is to control if the test assesses what it is supposed to assess.
Test-Takers' Opinions on the IELTS Exam, the Reading Component and Scores
Kane and Bejar (2014) highlight students' capability of continuing in further education in a non-native linguistic environment as the main mission of a language testing system. First, all the GEP finalists were accepted to leading universities. A smaller proportion had successfully finished their studies, while the vast majority were in the middle of their
courses. These facts imply that, regardless of the scale and the nature of probable difficulties with language, all the respondents can be named academic achievers as their IELTS results have sufficed and have not impeded their progression as international scholars.
Test-takers expressed challenging thoughts on the IELTS reading component in the option Other. The most extensive commentaries featured IELTS validity as a reliable evidence of current language mastery and a guarantee of further language improvement, along with IELTS comparisons with TOEFL and CPE exams from students who had the experience of taking more than one language test. The respondent with 7.0 overall and 7.5 for reading pointed out that "reading in TOEFL is almost the simplest part and getting almost maximum is not very difficult. TOEFL seems quite balanced in time, but in IELTS reading feels trickier than other sections".
The second issue, discussed by 5 test-takers, was the cost of the IELTS test in general and in particular for the university applicants from Russia, where the cost of the exam is highly dependent on fluctuating currency exchange rates. Cost was highlighted as a stress factor, potentially lowering the scores in case of the impossibility of retaking the exam.
The expiry period of two years did not seem reasonable for three students: "They give a certificate for 2 years. That's funny. It should be given for 5 years, at least 3. I don't understand how I will be able to forget the language?" The answer can be found in another respondent's opinion: "These exams are multi-billion dollar business and 2 years is definitely an extremely short period. It's just marketing and there's nothing to be done". Other students would prefer more detailed IELTS grades and comments that would make the test much less affordable and risk misinterpreting real
knowledge and skills, construct irrelevant variances inevitably influencing validity.
The third comment was the division of IELTS into academic and general modules. This, in one respondent's opinion, has marketing justifications as well: "You successfully passed IELTS academic and it happened that you have to migrate, but they will not accept it! Bring the general! It does not matter that the academic module requires more professionalism and very often suggests better qualifications".
The fourth opinion was that the IELTS test does not fully provide evidence of language mastery and more extended assessment procedures should be compulsory before providing a non-conditional offer of acceptance: "Everything should be controlled, i.e., a Skype interview with a candidate, where the commission will assess the level of the language proficiency, presentation skills and background knowledge! At the same time!" However, this student seemed to have neglected that the IELTS aim does not go further than marking the border of readiness to study in a foreign university and other methods of evaluating candidate applicability are engaged into this extended procedure.
Reading Strategies Most Applicable to Academic Challenges
Although careful reading has traditionally attracted more attention from researchers (Alderson, 2000, Urquhart et al., 1998, Weir, 1993), while the four other strategies could be less beneficial for scholarly challenges, the results did not reveal careful reading superiority to any extent. Moreover, only answering Question 4, higher achievers were notably more generous in grading than lower achievers. This finding implies that higher achievers manage to apply all the strategies with a high degree of regularity. An interesting comment was made by an Engineering student (overall 7.5; reading 8.5) about browsing: "In project-based education the most effective technique is quick browsing for interesting bits of relevant information, to build up a strong narrative behind the proposal. Deep immersion to theoretical works is less integrated in studies; however, thoughtful hours of reflection just make your overall performance and argumentation stronger"
The growing scale of accessible information automatically requires setting new, multileveled reading goals, i.e., being cautious and highly selective in terms of quality, appropriateness and the volume of information. These goals can explain search reading and scanning becoming the most applicable for academic challenges, as before reading carefully students need to go through the time-consuming process of evaluating the relevance and quality of
information and only after add a source to their reading lists. This issue is also related to Question 5 results, that textbooks did not appear to be in the list of three most used information sources for students. This implies striving for research activities versus getting the information in ready forms, which usually considerably reduces the cognitive load.
Khalifa and Weir (2009) suggest that along with comprehension the speed of reading is of significant importance. Weir et al. (2009) found that, "for many readers reading quickly, selectively and efficiently posed greater problems than reading carefully and efficiently" (p. 162) and drew attention to the underestimation of the cognitive processing required to carry out test tasks by the majority of researchers. According to Weir et al. (2009), imposing time constraints, seen by almost all the test-takers as predictors for lower grades, has the potential to enhance expeditious reading strategies, i.e. search reading and scanning, which are of great importance for research activities necessary for scholarship. These opinions are fully congruent with our findings.
Information Sources Needed at University
Comparison by specialty provided noticeably more variance while filtering by exam scores did not reveal any notable alteration from overall results. An Education student (8.0 overall, 8.0 reading) commented: "I am always reading something related to my field. Unfortunately, at this point I do not have time for any fiction. However, when I feel like catching up with recent novels I get them as audiobooks because my eyes are too tired to read anything extra". Generally, the test-takers were appreciative of a variety of sources highlighting the importance of studying relevant ones.
While online sources were on top of our list, the study by Moore et al. (2012) mentioned reviewing educational websites as not encouraging good reading practices as students do not read online in a serious and steady way (p. 44). This could be relevant to the problem of limitless information flow, a growing number of poorly referenced sources, which lack healthy criticism. In addition, the lecturers' concern raises the question of making implications on a shallow basis due to careless reading, which can prevent the enhancement of more global and advanced reading practices.
Tasks Effectiveness for Academic Reading Preparation
All the task types were appreciated to a considerable extent by all the test-takers with one of them pointing out that "the options overlap". A Science student
(overall 7.5, reading 7.5) commented: "I cannot specify and distinguish separate tasks... probably a mix of them helped somehow, but I'm not even sure whether separately they were very effective or not that much. That's why I chose "somewhat effective" almost everywhere". This perceptive point specifies the need for a constant reconsidering of the appropriateness of a combination of tasks for each particular test for the purpose of providing test-takers with the chance to demonstrate the broadest possible range of reading capabilities.
In the study by Moore et al. (2012) the required engagement was of highly "local" and "literal" kind for a number of common tasks. Agreeing with Alderson (2000, pp. 211-214), Moore et al. (2012) highlight that multiple-choice tasks have limitations, including the potential for guessing the correct option. Researchers suggested pushing test tasks in the direction of more global and interpretative domains of the analytical matrix featuring a range of views, available evidence, bringing a critical approach, examining and assessing the material and coming to personal judgment.
Weir et al. (2009) recommend ensuring that each test form includes a variety of tasks striving to engage expeditious and careful reading strategies (p. 179). For instance, the Not Given option in Yes/No/Not Given items can encourage the application of both strategies and reduce the possibility of guessing. The NG option requires the ability to make a distinction between one's own opinions and opinions of the others, which is of vital importance for university study where the critical appraisal of information sources is compulsory. Moreover, recommendations by Weir et al. (2009) to make efforts to minimize response strategy assumptions from tasks can be of great help for test developers as well as higher education applicants (p. 179).
IELTS Preparation Usefulness to Academic Adjustment and its Influence on Academic Achievement
The most striking point provided by Questions 7 and 10 was scepticism about the effectiveness of IELTS preparation for higher achievers while their very high results indicate their readiness for the exam. Such results definitely could not have been achieved without extensive language input. This intuitively clear finding requires further investigation for the purpose of identifying alternative forms of exam preparation, figuratively speaking, "without preparing exactly for an exam". Although the impact of linguistic
abilities is obvious and undeniable, further research could shed light on the more real-to-life strategies for increasing the exam score.
Special attention could be paid to the correlation between previous performance scores with IELTS score. Qualitative research also has a lot of potential in terms of individual case analyses, i.e., the one, found in the comment of the Education student (8.0 overall; 8.0 reading): "I think one of the best ways to enhance your academic achievement is through reading CPE materials (the old format). Constantly reading something that is even more difficult than what is given in IELTS textbooks helps you achieve very high IELTS score. I have dyslexia and yet I managed to get a very high reading score despite all the stereotypes that dyslexic people cannot learn languages. Practice makes it perfect. There is nothing too difficult about any part of IELTS. If you are resilient and persistent then you can achieve an impressive result".
Respondents' Perception of Extra Factors of Exam Success
Question 8 appeared to be the most commented on and provided a lot of room for interpretation. Exam success is definitely guaranteed when there is a combination of deep knowledge, test-wiseness gained through careful preparation and the advantages probably provided by extra factors. However, we can only strive to predict the proportion of these components in order to calculate a perfect formula for success.
For instance, the test-takers who provided commentaries had different specializations, very different opinions, but their reading scores did not differ significantly. Moreover, the three most chosen extra factors are very resourceful for the formation of numerous skills, going far beyond IELTS and are applicable to various fields of knowledge and work. As such, accusations that IELTS is over-dependent on extra factors do not look very convincing. In the majority of cases, sharp comments can be justified by a selfish desire of weaker students, undervaluing test difficulty to receive higher grades, while a smaller proportion of test-takers may tend to overestimate their "extra" skills and underestimate the real quality of knowledge. Whatever approach to these factors are chosen, they should never be ignored or downplayed by practicing tutors, who have to analyze each student's performance independently and predict developing trajectories with the greatest possible precision.
Table 1
Test-takers comments on 08
IELTS overall score IELTS reading score Specialisation and commentaries
Health care
"The most important things for IELTS are: 1) you MUST think fast. If you are perfect at English, but you are slow, you will not get enough points. This test checks the 7.0 7.5 speed of your thinking. It is not fair because some people are bright, smart, but
they need more time. 2) You must prepare for the test, its Questions, structure, and requirements. You will get more points this way".
Engineering
"Careful but fast reading! Reading Questions before and trying to answer them 6.5 7.0 while reading. I always start with reading the given answers/options and only then
start reading the text. Searching for the right part of the text one by one."
Science
"I believe that too little time is given for reading in the IELTS exam. You can know the language well and make mistakes. If this reading is given to me in Russian, I will 6.5 7.0 not get 9.0. Although this is my native language. There should be twice as much
time given. Because there are people who read and think fast, others do it more slowly. It should not be that much connected to speed, but language mastery. It's unfair."
Engineering
7.5 8.5 "It seems "deep knowledge" did not affect the results at all; the test is more about
these factors (concentration, speed)."
Social work
"I'm not so sure that IELTS reading is relevant to our competence in English. I had very mediocre English at the exam time, I had been preparing for a month with a 7.0 7.5 tutor, but I have a certain advantage - speed! I finished the task in 40 minutes, so I
searched for extra 20 minutes more and made corrections - the highest score was for this component."
Education
"I believe that IELTS in general and its reading component are not very reliable 6.5 7.5 indicators of English proficiency. I'm very interested in the results of the survey,
because at the time when IELTS was relevant, I was confused by its artificiality. If it really identified the knowledge of English, most likely, I would not have achieved my result!"
IELTS Preparation Resourcefulness for the Formation of Linguistic Capabilities
The most desirable linguistic capabilities, identified in Question 9, appeared to be congruent with the findings by Moore et al. (2012). A critical approach to knowledge advocated by a number of informants implies assessing arguments and not assuming that other points of view do not exist: "You might find a perfectly reasonable answer in a single book..., but you're in no position to evaluate that unless you've read alternatives" (Moore et al., 2012, p. 56). This
can be relevant to the pursuit of objectivity, based on exploring a focused topic in a broader context and a gradual movement from description to analysis and the evaluation of concepts and facts, indicating a global - interpretative level of language command and perception.
Along with the findings by Moore et al. (2012), our findings demonstrate that the higher order skills of using inductive and deductive methods of perceiving information, distinguishing what the main points are and summarizing them are scholarly capabilities of high significance. Moreover, both studies highlight
that extensive genre-based reading has the potential to improve learners' writing. This point is crucial for integrating different forms of language command and showing the difference in the requirements between high school and academia.
Conclusion
The literature sources reviewed elucidate various interpretations of validity applied specifically to IELTS reading preparation. Constantly becoming more challenging, requirements of academia demand the capability to switch swiftly between reading strategies and approaching academic texts in various, local and global, literal and interpretative ways. The importance of speed-reading, which evoked the most extensive, inconsistent and emotional commentaries in participants, is expected to be the foundation stone of future debate.
Reading competence may be an extremely broad category to be measured in sixty minutes, but quality preparation can still provide an impressive number of essential skills required in academia, especially for students with a more modest linguistic background. However, preparation for the IELTS reading test is not considered as an extremely significant factor of smoother adjustment to academic challenges and an impact factor on academic success.
There is a necessity to communicate broader information to learners through the IELTS handbook, website and other communication channels. As all forms of assessment have their advantages and limitations, EAP tutors should encourage their students to make efforts to cover the subject without framing it within boundaries of measurement, but with a clear understanding of future academic and professional challenges.
Additional areas for future research include, but are not limited to, elaborating on extra factors besides deep knowledge, which could positively affect IELTS scores in reading and three other IELTS components from the perspective of educational psychology and language teaching methodology. As test-wiseness and washback are subjects of high significance due to the growing number of available exam materials and preparation techniques, more in-depth methods, such as classroom observations, detailed interviews and self-reports could be used to explore their negative consequences and develop methodological recommendations for all the stakeholders.
References
Alderson, J. C. (2000). Assessing reading. Cambridge, UK: Cambridge University Press. DOI: 10.1017/ cbo9780511732935 Brookhart, S. M. (2007). Editorial. Educational measurement: Issues and practice, 26(1), 1-2. DOI: 10.1111/j.1745-3992.2007.00082_1.x Global Education Programme. (2017). Official documents. Retrieved from http://educationglobal. ru/en/ns/overview/officialdocuments/ Hoover, W. A., & Tunmer, W. E. (1993). The components of reading. In G. B. Thompson, W. E. Tunmer & T. Nicholson (Eds.), Reading acquisition processes (pp. 1-19). Clevedon, Avon, UK: Multilingual Matters. Hyatt, D., & Brooks, G. (2009). Stakeholders' perceptions of IELTS as an entry requirement for higher education in the UK. Journal of Further and Higher Education, 37(6), 844-863. DOI: 10.1080/0309877x.2012.684043 Grabe, W. (1999). Developments in reading research and their implications for computer-adaptive reading assessment. In M. Chaloub-Deville (Ed.), Issues in computer-adaptive testing of reading proficiency (pp. 11-47). Cambridge, UK: Cambridge University Press. IELTS. (2007). IELTS official practice materials.
Cambridge, UK: Cambridge University Press. IELTS Essentials. (2017). Retrieved from https:// www.ieltsessentials.com/global/prepare/ freepracticetests/academicreadingpracticetests IELTS. (2015). Guide for educational institutions, governments, professional bodies and commercial organisations. Retrieved from https://www.ielts. org/~/media/publications/guide-for-institutions/ ielts-guide-for-institutions-2015-uk.ashx Ingram, D., & Bayliss, A. (2007). IELTS as a predictor of academic language performance. Retrieved from https://www.ielts.org/~Zmedia/research-reports/ ielts_rr_volume07_report3.ashx Kane, M. T., & Bejar, I. I. (2014). Cognitive frameworks for assessment, teaching, and learning: A validity perspective. Psicología Educativa, 20(2), 117-123. DOI: 10.1016/j.pse.2014.11.006 Khalifa, H., & Weir, C. (2009). Examining reading: Research and practice in assessing second language learning. Retrieved from http://linguistics.uic.edu/ projects/reviews/page-126056771.html Liu, F. (2010). Reading abilities and strategies: A short introduction. International Education Studies, 3(3), 153-157.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-104). New York, NY: American Council on education and Macmillan.
Messick, S. (1994). Alternative modes of assessment, uniform standards of validity. Princeton, NJ: Educational Testing Service.
Messick, S. (1996). Validity and washback in language testing. Language Testing, 13(3), 241-256. DOI: 10.1177/026553229601300302 Moore, T., Morton, J., & Price, S. (2012). Construct validity in the IELTS academic reading test: A comparison of reading requirements in IELTS test items and in university study. IELTS Research Reports, 11, 1-80. Nitko, A. J., & Brookhart, S. M. (2007). Educational assessment of students (5th ed.). Upper Saddle River, NJ: Pearson Merrill Prentice Hall. O'Loughlin, K. (2012). Developing the assessment literacy of IELTS test users in higher education. IELTS Research Reports, 13, 1-80. Plake, B. S., & Wise, L. L. (2014). What is the role and importance of the revised AERA, APA, NCME standards for educational and psychological testing? Educational Measurement: Issues and Practice, 33(4), 4-12. doi: 10.1111/emip.12045 Rayner, K, & Pollatsek, A. (1989). The psychology of
reading. Englewood Cliffs, NJ: Prentice Hall. School of Education. (2015). Trinity College Dublin Course Handbook 2015/16 Retrieved July 6, 2016 from https://www.tcd.ie/Education/ courses/masters/M.%20Ed.%20Handbook%20 2015_16-September.pdf British Council. (n.d.). Take IELTS with BRITISH COUNCIL. Retrieved from http://takeielts. britishcouncil.org/prepare-test/practice-tests/ reading-practice-test-1-general-training Urquhart, A. H., Weir, C. J., & Frost, C. (1998). Reading in a second language: Process, product and practice. London, UK: Longman. Weir, C. J. (1993). Understanding and developing
language tests. London, UK: Prentice-Hall. Weir, C., Hawkey, R., Green, A., & Devi, S. (2009). The cognitive processes underlying the academic reading construct as measured by IELTS. Retrieved from https://www.ielts.org/Vmedia/research-reports/ ielts_rr_volume09_report4.ashx
