CC BY-NC-ND
30
Does Attending a Strong School Guarantee Good College Performance?
Evgeniya Popova, Marina Sheina
Received in Evgeniya Popova
December 2016 Junior Researcher, Laboratory of Interdisciplinary Empirical Studies, National Research University Higher School of Economics. Еmail: [email protected] Marina Sheina
Candidate of Sciences in Mathematical Physics, Senior Lecturer, Faculty of Economics, Management, and Business Informatics, National Research University Higher School of Economics. Еmail: [email protected]
Address: 38 Studencheskaya St,
614070 Perm, Russian Federation.
Abstract. The paper tests the hypotheses on better academic performance of graduates from stronger high schools and the nature of relationship between college students' achievements and their academic standing in high school (whether they performed above or below average) with due regard for school characteristics. Regression analysis is used to measure the effects of USE (Unified State Exam) scores, school type, and academic standing on college performance, while controlling for individual student characteristics and cases receiving the Governor's regional scholarship in addition to the standard student allowance. The sample includes 313 first-year Economics and Management students admitted to the National Research
University Higher School of Economics in 2012 and 2013. Cumulative first-year ranking points are used as an indicator of academic performance. As it turns out, graduating from an advanced high school or from a school with a high mean USE score in mathematics provides no guarantee of better educational outcomes for first-year students. High-school academic standing has positive effects on academic achievements in college, the strength of such effects varying depending on school characteristics. Educational outcomes of students who performed above average in low-performing schools can be explained by the high level of intrinsic motivation typical of academically successful students. Therefore, ignoring the information on the academic standing of graduates from low-performing schools may lead to underestimating their academic achievement in college. As for receiving the Governor's scholarship, this proves to be a significant factor in the academic performance of Management students only.
Keywords: Unified State Exam (USE), school quality, academic standing in high school, higher education, college performance, factors of academic performance.
DOI: 10.17323/1814-9545-2017-1-128-157
Empirical studies in education focus largely on analyzing the factors of academic performance. The research avenues include analysis of predictors of academic performance of college students, such as
standardized test results, high school performance, or the type of high school. The findings are controversial.
One of the advantages offered by standardized tests is that they assess student competencies using a unified standardized scale, allowing for comparison at the individual, school, municipal, and national levels. It is assumed that school leavers with better standardized scores have better competencies and will thus perform better in college.
High school grades represent the outcome of a long educational process and efforts made by students with individual levels of intellect and motivation. Hence, high school performance can be regarded as an indicator of student competencies, motivation and working ability [Gordeeva 2013:179]. However, high school grades cannot be compared directly, except perhaps at the individual level, within a group of students or a specific school. At the municipal level already, grade-based comparisons of student performance are impossible due to the diversity of school types, education programs offered, curricula and textbooks used, specific assessment standards and criteria applied, as well as specific teachers' requirements1. However, comparison is possible for relative academic performance indicators, which characterize the level of individual academic achievement as in ratio to the average level of school performance. Such indicators can also reflect the level of motivation for learning and thus serve as factors of academic performance in college.
Differences in school performance are initially attributed to the differences in education programs. Schools of advanced types—gymnasiums, lyceums, specialized schools—are ranked among the most effective educational institutions most often, according to Russian researchers [Konstantinovsky 1999; Cherednichenko 1999; Konstanti-novsky et al. 2006; Yastrebov et al. 2013]. However, the distribution of children among schools in Russia is not incidental. Advanced schools are selected by families with high levels of socioeconomic and cultural capital, firmly oriented at academic achievement. In addition, advanced schools select the most talented children to be admitted to primary, middle and high school. Why do advanced schools demonstrate on average higher levels of academic performance and standardized test (USE2) results than other educational institutions? "We don't know whether it's better teaching or better student population that makes advanced schools stronger." [Derbishir, Pinskaya 2016:114]
We analyzed the correlation between the academic performance of first-year students of a Russian university with their individual USE
1 For instance, when an A-student transfers from a general education secondary school to an advanced or specialized institution, in most cases they start to obtain lower grades, their competencies remaining the same.
2 Unified State Exam
scores in certain subjects as well as with school characteristics. We introduced and considered the indicator "academic standing" as a measure of academic performance of high school students, which uses USE scores in mathematics to show whether a prospective college student performed on average better or worse than their peers. Using the mean USE score in mathematics in a given cohort, we assessed school performance and identified two categories of schools: high performers and low performers. A regional law of 2010 grants an additional regional ("Governor's") scholarship to students scoring 225 (260 for economic majors) and more in three USE subjects cumulatively and who have stayed to obtain higher education in the region. The effect of receiving this scholarship was also considered in the analysis of the academic performance of college students.
Based on the data on the academic performance of Economics and Management students of the National Research University Higher School of Economics (HSE) (Perm) as well as the database on the USE results of 2012 and 2013 high school graduates in Perm Territory collected by the HSE Laboratory of Interdisciplinary Empirical Studies, we answer the following questions.
1. Does attending an advanced school guarantee a higher level of academic achievement in college?
2. How does academic standing in high school affect academic performance in college?
3. How does receiving the Governor's scholarship influence the academic performance of college students?
We have found that graduating from an advanced school does not guarantee high academic performance for first-year HSE (Perm) students majoring in Economics and Management. However, we have revealed a significant positive effect of academic standing, the extent of which depends on school quality. At the same time, the academic standing indicator mediates the relationship between school quality and academic performance in college: management students who graduated from low-quality schools but had high academic standing perform on average better than their peers from high-quality schools but with low academic standing, provided that the mean individual USE scores in mathematics are comparable. Receiving the Governor's scholarship, which exceeds the standard student allowance by almost four times, is a significant factor of academic achievement in Management but not in Economics.
The article is structured as follows: part one provides an overview of the key studies on correlations between standardized test results and school quality on the one hand, and academic performance of college students on the other; part two describes the sample and presents descriptive statistics; part three outlines the analysis tools; part four contains research results; and, finally, conclusions are drawn in part five.
1. Research on the factors of academic performance of college students 1.1. SAT and ACT in the USA
Extensive experience in studying the correlation between standardized test results and the academic performance of college students has been accumulated in the United States. Standardized tests, such as SAT (Scholastic Assessment Test) or ACT (American College Testing), are widely used for admissions to American colleges.
SAT was originally designed to measure the general innate intelligence of school students, in the first place. The alternative ACT was rather meant for assessing competencies in specific subjects and skills acquired in school. Gradual modifications in both have resulted in virtually no significant difference in their predictive power today [Atkinson 2009].
Tatyana Khavenson and Anna Solovyeva [2014] assessed the predictive power of SAT and ACT in American colleges based on an overview of publications over quite a long period of time to find that it explained 12-25% of variations in the academic performance of first-year college students. However, American educational researchers [Richardson, Abraham, Bond 2012] report high school grades to be a more reliable predictor of academic achievement in college. Consideration of standardized test results together with high school grades has been found to have greater predictive power [Kobrin et al. 2008; Patterson, Mattern 2012; Westrick et al. 2015].
According to Rothstein [2004], the predictive power of SAT is explained predominantly by its correlations with school characteristics. If demographic parameters and school quality are considered along with SAT, its significance as a factor of academic performance in college will decrease by 20% on average.
1.2. The USE in Two major problems have been solved by the introduction of the USE.
Russia First, the test has become a tool for the independent assessment of subject-specific knowledge and skills acquired in school, being used in calculating the final high school grades. Second, as soon as the USE is used for college admissions, it serves as a measure of school leavers' abilities: more capable students are expected to show better academic achievements in college.
The first Russian study [Derkachev, Suvorova 2008] analyzing the significance of USE scores as a predictor of the academic performance of college students was conducted in 2008. A number of subsequent publications presented their findings on the strength of correlations between the cumulative and subject-specific USE scores, on the one part, and college performance, on the other [Poldin 2011; Per-esetsky, Davtyan 2011; Zamkov, Peresetsky 2013; Khavenson, Solovyeva 2014].
Using regression analysis and meta-analysis of academic performance of about 19,000 first-year college students in five Russian universities between 2009 and 2011, Khavenson and Solovyeva estimated the predictive power of the cumulative USE score: the mean determination coefficient was found to be 0.20 in all majors, vary-
ing from 0.15 to 0.35 across the departments [Khavenson, Solovye-va 2014]. These findings are consistent with equivalent SAT and ACT estimations. The mean determination coefficient was 0.30 in economics (CI = 0.23-0.37) and 0.25 in management (CI = 0.22-0.27). The predictive power of USE results in specific subjects differs across departments as well, the highest being in mathematics and Russian in the great majority of the majors, while USE scores in major-specific subjects show low predictive capacity.
The predictive power of USE scores has been confirmed by data on the college performance of economics students [Poldin 2011; Pe-resetsky, Davtyan 2011; Zamkov, Peresetsky 2013]. All the authors agree that the USE score in mathematics has the greatest impact on the academic performance of first-year students. While analyzing the performance of economics students admitted to the HSE in 2009, Oleg Poldin came to the conclusion that subject-specific USE scores had higher predictive power than cumulative USE scores in a few subjects [Poldin 2011]. Oleg Zamkov and Anatoly Peresetsky assessed the influence of USE scores in mathematics, Russian and English on the academic performance of first-year students enrolled in the International College of Economics and Finance (ICEF)3 in 2009, 2010 and 2011. The authors used student gender, admission year, and region of graduation from high school (Moscow or other) as control variables. The results of their research are as follows: USE scores in all three subjects are significant at the level of 1%, while the region of graduation from high school has no significant effect on student achievement [Zamkov, Peresetsky 2013]. A similar finding—the insignificance of the region of graduation from high school in the academic performance of first-year college students—was reported by Derkachev and Suvorova [2008].
1.3. School Data obtained in Russian studies [Prakhov 2014; 2015; Derbishir, Pin-characteristics skaya 2016] demonstrate a strong correlation between USE scores and the type of educational institution. In particular, a positive correlation significant at the level of 1% has been revealed between the USE score in mathematics and the status of the lyceum or gymnasium offering advanced programs [Derbishir, Pinskaya 2016]. Ilya Prakhov [2014] found a 5% significance level positive correlation between the cumulative USE score in all subjects and attending a gymnasium or a specialized school. Of interest is analysis of the relationship between college performance and the type of high school.
1.4. Academic Russian psychologists believe that USE scores reflect the level of genstanding eral intelligence as well as basic learning skills and competencies ac-
3 ICEF is an HSE department that implements a joint Bachelor's program in Economics with the London School of Economics.
quired in school [Gordeeva, Osin 2012]. The effect of supplementary courses high school students attend to perform better in the USE is significant, yet rather low [Prakhov 2014; 2015]. Having analyzed the factors of academic performance of chemistry students in Moscow State University, Tamara Gordeeva and Yevgeny Osin [2012] found that students with better USE scores showed a higher level of general persistence and concentration skills. It means that high academic performance may be achieved either by innate mental abilities or by effort and perseverance. Meanwhile, these individual characteristics do not manifest themselves out of the blue during the first year in college but develop throughout the long process of schooling. Gordeeva [2013] demonstrated that the structure of motivation of academically successful students differed considerably from that of their peers4. Students showing high performance in school possess a higher level of intrinsic cognitive interest in learning and enjoy the learning process much more than low performers, seeing it as valuable and important5. The author states that "the most successful students show a much higher level of intrinsic motivation, particularly cognitive and achievement motivation <...> than their lower-performing peers." [Gordeeva 2013:179] since USE scores serve as high school grades, we can use them to establish whether a student was academically successful in school. To do this, we identify the academic standing of a student, i. e. the ratio of his/her individual USE score to the mean USE score in the cohort. Higher academic performance in college can be expected from students with higher academic standings [Gordeeva 2013].
Thus, the analysis of previous research allows us to predict the relationship between the factors examined and academic achievement of first-year university students. A significant positive correlation between USE scores and college performance can be expected. We also investigate how academic performance of freshmen correlates with school characteristics and academic standing in school (but not school grades).
4 The sample included students in grades 6-11 of two non-specialized, non-selective schools in Moscow with conventional education programs, well-reputed among parents. The category of academically successful students included those with an average USE score in Russian and mathematics of 4.25 and higher (on a five-point scale).
5 The author believes that such attitude towards learning stems from family values: "This perception of the importance of learning is obviously inculcated in academically successful children by their family environment and parents who demonstrate by personal example the value of learning, broad-mindedness, thinking skills, research activity, intellectual competence, problem solving skills, and academic achievement." [Gordeeva 2013:180]
2. Empirical data sources and descriptive statistics
The empirical basis of the research consisted of data on academic performance and individual and school characteristics of Economics and Management students admitted to HSE (Perm) on 2012 and 2013. The data was obtained from two sources: the HSE (Perm) administrative database and the databases on the 2012 and 2013 USE in schools of Perm Territory. The administrative database provided information on academic performance, USE scores in specific subjects, type of financing (government funding or tuition), school parameters (location, status, and number). The USE database was used to estimate the mean USE score in mathematics among final-year students in each sampled school. The mean USE score in mathematics is used below in this paper as a characteristic of school quality.
The final sample included data on 313 college students who graduated from Perm Territory schools and were admitted to the HSE as Economics majors in 2012 and 2013. The major-based structure of the sample is presented in Table 1.
Figure 1 shows the proportions of Perm Territory first-year students admitted to the HSE (Perm) as Economics and Management majors who graduated from Perm Territory schools with different mean USE scores in mathematics.
Students from advanced schools (lyceums, gymnasiums, specialized schools) account for 84% and 81% in Economics and Management, respectively. The school location statistics are given in Table 2.
The mean indicators of school quality are higher for advanced schools than for regular ones in both majors: 61.82 as compared to 47.12 points in Economics and 59.94 as compared to 47.21 points in Management (Fig. 2). However, the situation with subject-specific USE scores is less unambiguous. Overall, the mean individual USE scores are at least as high for the category of advanced schools as for the category of regular ones in both majors, except for the mean score in social theory, which is lower for advanced schools in the Economics department: 75.11 as compared to 77.05 points in Manage-
Table 1. Population of first-year HSE (Perm) Table 2. The structure of the sample students majoring in Economics and based on school location (%)
Management in 2012 and 2013 (people)
Admission year Major
Economics Management
2012 66 77
2013 84 86
The city of Perm Perm Territory districts
Advanced schools 76 24
Regular schools 19 81
Boys and girls accounted for 27% and 73% of the sample, respectively. Government-funded places were obtained by 77% of the sample: 105 students in 2012 and 136 in 2013.
Figure 1. The mean USE score in mathematics among Perm Krai schools in 2012 and 2013 and the proportions of their graduates admitted to the HSE (Perm) as Economics and Management majors
Percentage of students in
the major œ
Economics Management
School's mean USE score in mathematics
Figure 2. Mean school and individual USE scores in advanced and regular schools in the sample with due regard for majors.
£ r- œ
« w £ co co
m lo
Advanced schools, Economics Advanced schools, Managemen Regular schools, Economics Regular schools, Management
INDIVIDUAL USE SCORES
Table 3. Mean USE scores in school subjects and academic performance of first-year students in Economics and Management enrolled in 2012 and 2013
Admission year, major USE in mathematics USE in Russian USE in social theory USE in foreign language First-year educational outcome
2012
Economics 72.621 84.727 74.485 71.500 7.227
Management 65.065 80.558 68.987 67.896 6.783
2013
Economics 71.107 85.548 76.107 87.179 7.210
Management 66.651 85.779 74.105 87.674 7.007
Table 4. Results of USE score and college performance correlation analysis
Educational outcome in major USE in mathematics USE in Russian USE in social theory USE in foreign language
Economics 0.386** 0.311** 0.357** 0.268**
Management 0.322** 0.300** 0.315** 0.154*
Note: *, **, *** denote coefficients significant at the level of 10%, 5%, and 1%, respectively.
ment (Fig. 2). Still, graduates from advanced schools generally show better USE scores in university admissions, so they are naturally expected to perform better in college.
As a measure of academic performance, we estimated first-year cumulative ranking points as a sum of all grades in obligatory courses attended during the first year in college, weighted on credits in each course according to the relevant curriculum. The final educational outcomes of first-year students are presented in a ten-point system. The mean USE scores and indicators of first-year college performance are given in Table 3.
Correlation analysis reveals that college performance correlates with the USE score in mathematics as strongly as with USE scores in other subjects for both majors (Table 4).
Having compared the academic performance of HSE (Perm) students who graduated from schools of different categories, we have made a surprising finding: the mean educational outcomes values are virtually the same for graduates from advanced and regular schools in both majors (Fig. 5). Therefore, graduates from lower-status schools admitted with lower USE scores perform on average better during the first year of college than their peers who graduated from lyce-
ums, gymnasiums or specialized schools and had on average higher USE scores.
In order to find possible explanations for this paradoxical finding, we add the indicator of academic standing to the analysis. Given that the USE in mathematics is an obligatory test and that the USE serves as a school grade, we calculate academic standing in high school for each student as the ratio of their individual USE scores in mathematics to the mean USE score in mathematics among final-year students of the same school. The high-school academic standings of college students broken down by majors are presented in Figure 3. An average student in Economics had a better academic standing in high school than an average student in Management: 1.23 and 1.16, respectively; the difference between them is statistically significant at the level of 5%.
The mean academic standing and educational outcome indicators for graduates from advanced and regular schools are given in Figures 4 and 5.
Because school status was found to correlate negatively with the academic performance of first-year HSE (Perm) students, we divide the sampled schools into two groups based on their mean USE scores in mathematics. Schools with the mean USE score of at least 65 points are conventionally referred to as "mathematical" in our study. School grade 'A' was given to those on or above this 65-point threshold, when translating the USE results into the five-point scale. All "mathematical" schools have an advanced status and are ranked among the top seven schools of Perm Territory. Graduates from such schools account for 32% of all graduates from advanced schools in the sample. Schools whose mean USE score in mathematics is below 65 are conventionally referred to as "average", their graduates accounting for 76% of the sample. All regular schools in the sample belong to the "average" category.
The relationship between the academic performance of first-year college students and school quality is described in Figure 6. Figure 7 presents data on the quality of schools whose graduates are enrolled as Economics and Management majors.
The observed differences in the relationship between first-year educational outcomes and school quality between the two majors may be explained by differences in the high-school academic standings of college students. We divided all the students into two groups based on their academic standing in high school: students who performed below average (academic standing within the first quartile, i. e. less than or equal to 1.05) and those who performed above average (all the rest) (see Fig. 3). There are 119 above-average performers in Economics (79% of all Economists) and 111 in Management (68% of all Managers). The structure of the sample based on academic standing is presented in Table 5.
Consideration of academic standing sheds light on the difference in the mean educational outcomes that cannot be explained by dif-
Figure 3. The distribution of high-school academic standing of HSE (Perm) students estimated as the ratio of the USE score in mathematics to the mean USE score among final-year students, 2012-2013.
Percentage of students in the major
Economics Management
Advanced schools Regular schools
"Mathematical" schools "Secondary" schools
Figure 4. Mean academic standings of students from advanced and regular schools in the sample with due regard for major, 2012-2013.
■f 5
<35 .—
i i
Economics Management
Figure 6. Mean first-year educational outcomes of students majoring in Economics and Management depending on school quality, 2012-2013.
Is- Is-
I
Figure 5. Mean first-year educational outcomes of students from advanced and regular schools in the sample with due regard for major, 2012-2013.
k¡ £
^ r- r--
i
Economics Management
Figure 7. Mean indicators of school quality (mean school's USE score in mathematics) broken by majors, 2012-2013.
Economics Management
- œ
Economics Management
Table 5. The structure of the sample broken down by school quality and academic standings of students
(people)
Table 6. The proportions of HSE (Perm) students majoring in Economics and Management who received the Governor's scholarship in 2012 and 2013 (%)
Academic "Mathematical" í "Average"
standing school i school
Economics
Above average 23 ! 96
Below average 18 ! 13
Management
Above average 10 ! 101
Below average 24 i 28
Admission year ! Economics Management
2012 i 56 22
2013 i 36 40
ferences in school quality. For Management students from "mathematical schools" of similar quality (mean scores 72.30 and 73.43), a difference in the mean academic standing (1.11 and 0.94) results in a considerable educational outcome gap (7.30 and 6.63) (Appendix 1).
We consider receiving the Governor's regional scholarship to be a controlled factor. Students in Economics and Management with the cumulative USE score in three subjects of at least 260 points in 2013 (240 in 2012) are paid the Governor's scholarship during the year, beginning with the very first month. This scholarship supplements the regular student allowance received by all students in government-funded places before the first end-of-term examinations. The Governor's scholarship was 5,000 rubles paid monthly in 2012 and 2013. To continue receiving it, a student must have satisfactory or lower grades for the end-of-term exams. The conditions of qualifying for the Governor's scholarship after the first year are tougher: the mean grade for all the first-year exams must be at least 4.75 (in a five-point system), or 4.5 in case the student engaged in research activity. The Governor's scholarship serves, on the one hand, as an indicator of good high school achievement, while on the other hand it provides an external material incentive for college achievement: high academic performance indicators should be achieved to retain the scholarship for the second year. The proportions of students receiving the Governor's scholarship in Economics and Management departments are given in Table 6.
Next, we test the following hypotheses on how first-year college performance correlates with school characteristics and academic standing.
Hypothesis 1. Graduates from schools with a high mean USE score in mathematics perform better in college.
Hypothesis 2. Students with higher academic standings in high school perform better during their first year in college.
Hypothesis 3. The effect of high-school academic standing on college performance varies depending on school quality, being higher for above-average performers from "average" schools than for below-average performers from "mathematical" schools.
3. Empirical data To test the statistical hypotheses, we use the data analysis method analysis method widely applied in educational research, namely estimation of linear regression models, equivalents of the educational production function. The models are estimated using the method of ordinary least squares. The cumulative first-year ranking points (educational outcome) of a student are used as an indicator of academic performance, while individual student parameters and school characteristics serve as explanatory variables.
The following specification (1) is offered to estimate the effect of individual USE scores and school characteristics on college performance:
(1) Y = a + a1 X + PS, + e„
where Y indicates the academic performance of student i; X, is the vector of student i's USE scores; S, is the vector of student i's school characteristics; £, is error.
The following specification (2) is offered to estimate linear effects of academic standing in high school:
(2) Y = a + «1X, + Y + a*G< + a3C, + £,,
where Y, indicates the academic performance of student /;
X.t is the matrix of student i's USE scores, except for the score in mathematics;
MS indicates student i's academic standing in high school i (the
ratio of individual USE score in mathematics M, to the mean USE score in mathematics among final-year students of the same school MS );
G ,, is a dummy variable that takes the value 1 if the student received the Governor's scholarship during the first year and 0 otherwise; C, is the matrix of control variables, which include admission year, form of financing, and student i's gender; £.. is the error.
Binary variables are introduced to estimate the nonlinear6 effects of academic standing and school quality:
,H h, if M > 1,05 „ |l, if M < 1,05 ■ , x IH = 1 MS; and IL = i MS; are indicators of
10, otherwise 10, otherwise
academic standing in high school;
l"l, if MS, > 65 |"l, if MS, < 65
IM = u I™ = <{ are indicators of
10, otherwise I 0, otherwise
school quality, where IMstands for "mathematical" school and I"M denotes "average" school.
The following specification (3) is offered to estimate the nonlinear effects of academic standing with due account for school quality:
Y = a + aX. + Y1 M IHIM + Y2M IHINM + Y1 ~Ml ILIM + Y1 -Ml ILINM+ ' 1 ' 1 MS; ' ' ^ MS; ' ' ^ MS; ' ' ^ MS; ' '
(3) + a2G'. + a3C; + £; .
The analysis of academic performance factors in the two majors showed that student competencies differed a lot between the Economics and Management departments [Khavenson, Solovyeva 2014]. For this reason, regressions (1)—(3) are estimated separately for Economics and Management students.
The endogeneity problem is typical of most studies on the educational production function. Attempts to find the optimal solving tools depending on the context are made in a number of studies7. However, they are rarely used, as finding a good tool is rather difficult.
4. Regression Analysis of unstandardized regression coefficients of multiple regres-analysis results sions (1)—(3) allows us to assess and compare the relationship between college performance, on the one part, and individual USE scores in each subject, school characteristics, and academic standing, on the other part. The regressions are analyzed using the method of ordinary least squares; standard errors are corrected and robust; multicolline-arity is controlled for. The results of estimating the effects of USE and school status, i. e. regression (1), are presented in Table 1 of Appendix
6 "In social interaction models, environment effects are normally considered nonlinear if their strength (yX') depends on the relative position of a student in statistical distribution Xi or on distribution yX,'/' [Andrushchak, Poldin, Yud-kevich 2012:6] For instance, the effect of school quality on college performance may be different for students with high and low academic standings.
7 E.g. average monthly income per member of household is used as a tool for assessing cumulative material investments in additional exam preparation courses [Prakhov 2014].
2. For both majors, first-year performance has positive effects on USE scores in all subjects except foreign languages, which shows no significant effect on the academic achievement of Management students (specifications 1ea, 1ma). Estimated values of the model's explanatory power (0.284 for Economics and 0.187 for Management) are generally consistent with the conclusions made by Khavenson and Solovyeva [2014] about the predictive power of USE scores. As we introduce the dummy variable of school status into the model, we confirm the finding made during descriptive analysis: students from advanced schools perform significantly (at the level of 10%)—on average by 0.4 points-lower than their peers from lower-status schools, all other conditions being equal. Graduation from a "mathematical" school also has significant (at the level of 10%) negative effect on the college performance of Managers, while showing no significant effect on Economists' educational outcomes. Therefore, Hypothesis 1 has been disproved: graduation from a school with a high mean USE score in mathematics does not guarantee better college performance.
These findings contradict the results of studies by Prakhov [2014; 2015] and by Derbishir and Pinskaya [2016]. The contradiction may be due to sample bias caused by specific characteristics of a regional college:
• Due to a high level of competition in the national tertiary education market, strong graduates of advanced Perm schools seek to enter Moscow and St. Petersburg colleges, while Perm colleges enroll less academically successful graduates of high-status schools;
• Due to a high level of competition in the Perm tertiary education market, strong graduates of advanced schools are redistributed among Perm colleges depending on their level of attractiveness;
• Eighty-one percent of regular schools in the sample are located in districts of Perm Territory. Available research data indicates that district schools mostly supply academically successful grad-uates8 with high levels of academic motivation to the HSE (Perm) (according to findings [Gordeeva 2013]).
The results of estimating the effects of academic standing in high school, i. e. regression (2), are presented in Table 2 of Appendix 2.
The role of the USE as a significant factor of college performance was diminished by the introduction of control variables-student and school characteristics (as components of the academic standing indicator)—into the model. The USE score in Russian lost its significance for students of both majors, and the score in social theory became in-
8 The mean USE scores in mathematics of students majoring in Economics and Management are 72.64 and 65.87 points for graduates from advanced schools and 66.73 and 66.07 points for graduates from regular schools, the mean academic standings of the latter being 1.43 and 1.41, respectively.
significant for Management students only. This finding does not contradict those obtained by Rothstein [2004].
We revealed a significant (at the level of 5%) negative effect exerted by the USE score in foreign languages (specifications 2ma, 2mb, 2mc) and academic performance of Managers. Similar results were obtained by Khavenson and Solovyeva [2014] for humanities departments.
The results of estimating the 2ec and 2mc specification assessment prove hypothesis 2: we observe a significant positive effect of academic standing in high school on college performance.
College performance also has positive effects on individual USE scores in mathematics (specifications 2eb, 2mb) and academic standing (specifications 2ec, 2mc). These effects are significant at the level of 5%.
Changes in the USE score in mathematics contribute more to academic performance in college than changes in academic standing for students majoring in Economics. For Managers, the relationship between the effects of the USE score in mathematics and academic standing depends on school quality. The strength of the effect of academic standing means that an increase in the academic standing indicator by 0.1 improves college performance by the relevant coefficient divided by 10, all other conditions being equal. An increase in academic standing by 0.1 is equivalent to an improvement in individual USE score in mathematics Mi by 0.1-MSi points. For a Management student from a school with the mean USE score in mathematics of MSi points, an increase in academic standing by 0.1 provides an improvement of 0.108 points in college performance, while the improvement from an equivalent increase in the USE score in mathematics is only 0.02-0.1-MSi points. Equating the values academic performance improvement, we calculate school's mean USE score in mathematics MSi to be 54. This result means that an increase in high-school academic standing produces on average greater improvement in college performance than an equivalent increase in the USE score in mathematics for Management students from schools with mean USE scores in mathematics below 54 points in our sample, all other conditions being equal. Graduates from such schools account for nearly half of the Management major in the sample (47%). Disregarding the academic standing of Management students from schools with the mean USE score in mathematics below 54 points, we underestimate the level of their academic achievement.
To estimate the strength of effects of individual USE scores in mathematics and academic standings in high school on academic performance of first-year college students, we calculate effect strength estimation indicators f2 as described by Cohen [1988]:
D2 R2
f2— «»compl /»¡ncompl
' 1 - D2 '
¡ncompl
where R2ncompi is the determination coefficient for the initial regression with no regressors;
Table 7. Indicators f2 of the strength of the effects of the USE in mathematics and academic standing in high school on the academic performance of students majoring in Economics and Management
Major USE in mathematics Academic standing
Economics 0.074 0.046
Management 0.048 0.072
and Riompl is the determination coefficient for a regression including a regressor, the effect of which is estimated.
Indicator f2, calculated in this way, allows us to determine how much of the dispersion unexplained in the base regression can be explained by adding a new regressor. The estimated effects of the USE in mathematics and high-school academic standing for regression (2) specifications are given in Table 7.
f2 values below 0.15 show that the effects of both the USE in mathematics and academic standing in high school on first-year college performance are equally weak, other individual factors like gender, admission year, form of financing (government funding or tuition), and receiving the Governor's scholarship being controlled for. This finding does not contradict those obtained by Rothstein [2004].
Receiving the Governor's scholarship turns out to be a factor significant at the level of 1% for academic achievement of Management students and insignificant for first-year Economists.
All the control variables demonstrate expected coefficient signs. The admission year is insignificant in the analyzed regression models; academic performance of students in government-funded places is significantly higher than that of tuition-paying students; the gender variable is significant in the Management department, where girls perform better than boys.
We analyze regression model (3) to estimate the nonlinear effects of academic standing with due regard for school quality. The results are presented in Table 3 of Appendix 2.
Analysis of the effects of academic standing in high school on the college performance with regard for being an above- or below-average performer as well as for school quality shows that the resulting effects are significant (at the level of 5% in Economics and 1% in Management) and commeasurable in Economics and Management majors with one exception only: no significant effects of academic
Table 8. The mean improvement in college performance equivalent to an increase in the USE score in mathematics by 1 point with regard for school quality, academic standing, and major (points)
Type of school Academic standing Economics Management
Regression j Performance coefficient j improvement Regression | Performance coefficient j improvement
"Mathematical" Above average 1.517 0.217 1.718 0.241
Below average (1) 1.835 0.246 1.811 0.247
"Average" Above average (2) 1.321 0.248 1.432 0.276
Below average - - 1.528 0.273
standing on the college performance is observed in Economics students who performed below average in "average" schools.
All the academic standing coefficients are low except for below-average performers from "mathematical" schools. The effect of their academics standing on college performance exceeds the relevant effect for above-average performers from "average" schools. At first glance, this finding contradicts Hypothesis 3.
Considering that changes in academic standing are equivalent to those in the USE score in mathematics estimated by the school quality indicator (AMi =0,1-MSi), we compare the mean improvements in academic performance in response to a 0.1-point increase in academic standing by expressing them as a function of a 1-point improvement in the USE score in mathematics. The calculation results are shown in Table 8.
To test Hypothesis 3, we need to analyze two groups of students: (i) those from "mathematical" schools (i. e. schools with the mean USE score in mathematics of at least 65 points) whose USE score in mathematics is below the average value among their peers; and (ii) those from "average" schools (i. e. schools with the mean USE score in mathematics below 65 points) whose USE score in mathematics is above average.
On average, a 1-point increase in the USE score in mathematics results in greater performance improvement for Managers in the second group (0.276 points) than for Managers in the first one (0.247 points) and in similar improvements for Economists in both groups (0.247 and 0.246 points, respectively). These findings are consistent with those obtained in the regression (2) analysis. Therefore, Hypothesis 3 is proved for Management students in the sample and disproved for students majoring in Economics.
The findings obtained on Management students are consistent with the ones made by Tamara Gordeeva [2013]. Students in the second group were academically successful in their schools, and their better educational outcomes can be explained by a higher level of intrinsic motivation. Ignoring the information on high-school academic standings of such students may result in underestimating their academic achievement in college.
In our study, we only use data on two cohorts of students (admitted in 2012 and 213) in one college, which describes the academic performance of Bachelor's students in Economics and Management admitted to a specific college in specific years. Despite sample bias, the determination coefficients estimated for all the analyzed regressions correspond to the estimations of predictive power of standardized tests made in American studies and confirmed by Russian researchers.
5. Conclusion This study was primarily aimed at analyzing school characteristics and academic standing as factors of college performance. We also investigated how school characteristics affected the correlation between college performance and academic standing.
The analysis performed shows that graduating from an advanced school or a school with a high mean USE score in mathematics does not guarantee better academic performance for first-year HSE (Perm) students majoring in Economics and Management.
While modeling academic performance, the consideration of control variables—student and school characteristics—reduces considerably the effects of USE scores in Russian and foreign languages for both majors and additionally in social theory for Managers.
Academic standing in high school, i. e. whether a student performed above or below average, has significantly positively effects on the college achievements of first-year students in both majors. The strength of these effects varies depending on the type of school, i. e. whether its mean USE score in mathematics corresponds to the "excellent" school grade (65 points and higher in our case) ("mathematical" school) or is below 65 points ("average" school). Managers who were academically successful in "average" schools perform on average better than their peers who performed worse in "mathematical" schools. Comparable educational outcomes are demonstrated by students of both groups majoring in Economics.
As the regression analysis shows, the consideration of academic standing in high school allows for reporting that an equal increase in the individual USE score in mathematics results in a greater performance improvement for Managers who were above-average performers in "average" schools than for their peers who performed below average in "mathematical" schools. Academic standing contributes more to college performance than an equivalent improvement in the USE score in mathematics for Management students from schools with the
mean USE score in mathematics below 54 in our sample. Graduates from such schools account for nearly half of the Management major in the sample (47%). They were academically successful in their schools, and their better educational outcomes can be explained by a higher level of intrinsic motivation [Gordeeva 2013]. It means that ignoring the information on high-school academic standings of students from low-performing schools may result in underestimating their academic achievement in college. In our case, college performance of nearly half of Management students may be underestimated.
Consideration of such a predictor as the Governor's scholarship shows that material incentives do not always contribute to college performance. Receiving the Governor's scholarship turns out to be a significant positive factor of academic performance for Management students and an insignificant one for students majoring in Economics.
A similar study on a larger sample would clarify the role of school characteristics and academic standing in college performance. It would be of interest to analyze the influence of factors associated with the level of academic standing in high school, such as sociode-mographic characteristics of students, socioeconomic and cultural capital of their families, as well as their correlation with long-term academic performance.
References Alexandrov D., Baranova V., Ivaniushina V. (2012) Migrant Children in Russia. I. Migration, Ethnicity and Segregation in St. Petersburg. Sociology of Education and Science Laboratory Working Papersno WP 001. St. Petersburg: HSE. Androushchak G., Poldin O., Yudkevich M. (2012) Effekty soobucheniya v adminis-trativno formiruemykh studencheskikh gruppakh [Peer Effects in Exogenously Formed Student Groups]. Applied Econometrics, vol. 26, no 2, pp. 3-16. Atkinson R. C. (2009) The New SAT: A Test at War with Itself. Available at: http://www. rca.ucsd.edu/speeches/AERA_041509_Speech_Reflections_on_a_Century_of_ College_Admissions_Tests.pdf (accessed 10 January 2017). Cherednichenko G. (1999) Shkolnaya reforma 90-kh godov: novovvedeniya I sotsi-alnaya selektsiya [The 1990s School Reform: Innovations and Social Selection]. Sotsiologicheskiy zhurnal, nos 1-2, pp. 5-21. Cohen J. E. (1988) Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ:
Lawrence Erlbaum Associates, Inc. Derbyshire N., Pinskaya M. (2016) Upravlencheskie strategii direktorov effektivnykh shkol [Principals' Management Strategies in High-Performing Schools]. Voprosy obrazovaniya/Educational Studies Moscow, no 3, pp. 110-129. Derkachev P., Suvorova I. (2008) Ediny gosudarstvenny ekzamen kak sposob otsen-ki potentsiala k polucheniyu vysshego obrazovaniya [The Unified State Exam as an Instrument to Measure Potential for Higher Education]. Sb. Statey aspiran-tov GU-VShE [Collection of Articles by HSE Postgraduate Students], Moscow: HSE, pp. 34-64.
Gordeeva T. (2013) Motivatsiya uchebnoy deyatelnosti Shkolnikov I studentov: struk-tura, mekhanizmy, usloviya razvitiya [Motivating School and College Students to Learn: Motivation Structure, Mechanisms and Development Prerequisites] (PhD Thesis), Moscow: Moscow State University. Gordeeva T., Osin Y. (2012) Osobennosti motivatsii dostizheniya I uchebnoy motivatsii studentov, demonstriruyushchikh raznye tipy akademicheskikh dostizheniy (EGE, pobedy v olimpiadakh, akademicheskaya uspevaemost') [Specific Features of
Achievement and Academic Motivation of Students Showing Different Types of Academic Success (High USE Scores, First Prizes in Olympiads, Academic Attainment)]. Psikhologicheskie issledovaniya: elektronny nauchny zhurnal, vol. 5, no 24. Available at: http://psystudy.ru(accessed 10 January 2017).
Khavenson T., Solovyova A. (2014) Svyaz rezultatov Edinogo gosudarstvennogo ekzamena I uspevaemosti v vuze [Studying the Relation between the Unified State Exam Points and Higher Education Performance]. Voprosy obrazovaniya/ Educational Studies Moscow, no 1, pp. 176-199.
Kobrin J. L., Patterson B. F., Shaw E. J., Mattern K. D., Barbuti S. M. (2008) Validity of the SAT® for Predicting First-Year College Grade Point Average. College Board Research Report No 2008-5. New York: The College Board.
Konstantinovsky D. (1999) Dinamikaneravenstva. Rossiyskayamolodezh v menyayush-chemsyaobshchestve: orientatsiiiputi v sfereobrazovaniya (ot 1960-kh godov k 2000-mu) [The Dynamics of Inequality. Russian Youth in the Changing Society: Directions and Trajectories in Education (from the 1960s toward 2000)], Moscow: Editorial URSS.
Konstantinovsky D., Vakhshtayn V., Kurakin D., Roshchina Y. (2006) Dostupnost kachestvennogo obshchego obrazovaniya: vozmozhnosti i ogranicheniya [Accessibility of High Quality Secondary Education: Opportunities and Limitations], Moscow: Logos.
Patterson B. F., Mattern K. D. (2012) Validity of the SAT for Predicting First-Year Grades: 2009 SAT Validity Sample. College Board Statistical Report No 2012-2. New York: The College Board.
Peresetskiy A., Davtian M. (2011) Effektivnost EGE I olimpiad kak instrumenta otbora abiturientov [Russian USE and Olympiads as Instruments for University Admission Selection]. Applied Econometrics, vol. 23, no 3, pp. 41-56.
Poldin O. Prognozirovanie uspevaemosti v vuze po rezultatam EGE [Predicting Success in College on the Basis of the Results of Unified National Exam]. Applied Econometrics, vol. 21, no 1, pp. 56-69.
Prakhov I. (2015) Dinamika investitziy I otdacha ot dopolnitelnoy podgotovki k postu-pleniyu v vuz [The Dynamics of Investment in Pre-Entry Coaching and the Returns from Private Tutoring among University Applicants in Russia]. Applied Econometrics, vol. 37, no 1, pp. 107-124.
Prakhov I. (2014) Vliyanie investitsiy v dopolnitelnuyu podgotovku na rezultaty EGE [Effects of Investments in Preparation Courses on the USE Scores]. Voprosy obra-zovaniya/Educational Studies Moscow, no 3, pp. 74-99.
Richardson M., Abraham C., Bond R. (2012) Psychological Correlates of University Students' Academic Performance: A Systematic Review and Meta-Analysis. Psychological Bulletin, vol. 138, no 2, pp. 353-387.
Rothstein J. M. (2004) College Performance Predictions and the SAT. Journal of Econometrics, vol. 121, no 1, pp. 297-317.
Sawyer R. (2010) Usefulness of High School Average and ACT Scores in Making College Admission Decisions. ACT Research Report No 2010-2.
Westrick P. A., Le H., Robbins S. B., Radunzel J. M., Schmidt F. L. (2015) College Performance and Retention: A Meta-Analysis of the Predictive Validities of ACT® Scores, High School Grades, and SES. Educational Assessment, vol. 20, no 1, pp. 23-45.
Yastrebov G., Bessudnov A., Pinskaya M., Kosaretsky S. (2013) Problema kontekstu-alizatsii obrazovatelnykh rezultatov: shkoly, sotsialnyy sostav uchashchikhsya I uroven deprivatsii territoriy [The Issue of Educational Results' Contextualization: Schools, Their Social Structure and a Territory Deprivation Level]. Voprosy obra-zovaniya/Educational Studies Moscow, no 4, pp. 188-246.
Zamkov O., Peresetskiy A. (2013) YGE I akademicheskie uspekhi studentov bakala-vriata MIEF HSE [Russian Unified National Exams (UNE) and Academic Performance of ICEF HSE Students]. Applied Econometrics, vol. 30, no 2, pp. 93-114
Appendix 1
Economics
School quality
"Mathe- "Secondary" matical" schools schools
Academic standing
Educational outcome
CO CO ^
^ CO ^ CO ^ -f l-^LO
^ ^ ® ^ ^
O O ^^ to
"Mathe- "Secondary" "Mathe- "Secondary"
matical" schools matical" schools
schools schools
Management
School quality
Academic standing
Educational outcome
High academic standing Low academic standing
"Mathe- "Secondary" matical" schools schools
CO CO
;= ro 03
"Mathe- "Secondary" "Mathe- "Secondary"
matical" schools matical" schools schools schools
Appendix 2 Table 1. The correlations between college performance, USE
scores, and school characteristics. Dependent variable: first-year cumulative ranking points on a ten-point scale.
Economics Management
(1ea) (1eb) (1ec) (1ma) (1mb) (1mc)
USE in mathematics 0.034*** (0.008) 0.036*** (0.008) 0.035*** (0.008) 0.027*** (0.008) 0.026*** (0.007) 0.032*** (0.008)
USE in Russian 0.016* (0.008) 0.017** (0.008) 0.016* (0.008) 0.025*** (0.009) 0.024*** (0.009) 0.024*** (0.009)
USE in a foreign language 0.009* (0.005) 0.011** (0.005) 0.009* (0.005) -0.008 (0.005) -0.007 (0.005) -0.009 (0.005)
USE in social theory 0.036*** (0.009) 0.034*** (0.009) 0.036*** (0.009) 0.026*** (0.009) 0.025*** (0.009) 0.025*** (0.009)
Advanced school -0.401* (0.221) -0.416** (0.185)
School quality3 -0.079 (0.176) -0.324* (0.186)
R2 0.284 0.295 0.280 0.187 0.208 0.197
Observations 150 150 150 163 163 163
Note. Robust standard errors of the coefficients are given in brackets;
*, **, *** denote coefficients significant at the level of 10%, 5%, and 1%, respectively.
a school quality indicator.
Table 2. The correlations between college performance, USE scores, and academic standing in high school (linear effects). Dependent variable: first-year cumulative ranking points on a ten-point scale.
Economics Management
(2ea) (2eb) (2ec) (2ma) (2mb) (2mc)
USE in Russian 0.006 (0.011) 0.010 (0.011) 0.007 (0.010) -0.010 (0.010) -0.007 (0.010) -0.011 (0.009)
USE in a foreign language 0.008 (0.007) 0.012* (0.007) 0.011 (0.007) -0.014*** (0.005) -0.015*** (0.005) -0.010** (0.005)
USE in social theory 0.030*** (0.010) 0.034*** (0.010) 0.028*** (0.010) 0.008 (0.009) 0.009 (0.009) 0.008 (0.009)
USE in mathematics 0.030*** (0.009) 0.020** (0.008)
Academic standing 0.927** (0.374) 1.083*** (0.351)
Governor's scholarship 0.244 (0.227) 0.082 (0.230) 0.219 (0.222) 0.519*** (0.196) 0.452** (0.196) 0.508*** (0.187)
Control variables Gender, admission year, government funding
R2 0.266 0.320 0.300 0.371 0.401 0.416
Observations 150 150 150 163 163 163
Note. Robust standard errors of the coefficients are given in brackets; *, **, *** denote coefficients significant at the level of 10%, 5%, and 1%, respectively.
Table 3. The correlations between college performance, USE scores, and academic standing in high school with due regard for school quality (nonlinear effects). Dependent variable: first-year cumulative ranking points on a ten-point scale.
Economics Management
(3a) (3b)
USE in Russian 0.009 (0.010) -0.011 (0.009)
USE in a foreign language 0.014* (0.008) -0.011** (0.005)
USE in social theory 0.026*** (0.010) 0.009 (0.009)
M. Above-average performer x "mathematical" school x ms 1.517** (0.712) 1.718*** (0.569)
Above-average performer x "average" school x -MIS 1.321** (0.586) 1.432*** (0.475)
M Below-average performer x "mathematical" school x ms 1.835** (0.871) 1.811*** (0.628)
Below-average performer x "average" school x -MS 1.183 (0.812) 1.528** (0.651)
Governor's scholarship 0.145 (0.234) 0.459** (0.191)
Control variables Gender, admission year, government funding
R2 0.317 0.429
Observations 150 163
Note. Robust standard errors of the coefficients are given in brackets;
*, **, *** denote coefficients significant at the level of 10%, 5%, and 1%, respectively.
