CC BY-NC-ND
82
Thinking Skills in Teaching Practices: Relationship with Students' Achievement in Mathematics
G. Larina, A. Kapuza
Received in Galina Larina
July 2019 PhD in Education, Research Fellow, Centre for Psychometrics and Measurement in Education, National Research University Higher School of Economics. Email: [email protected] Anastasia Kapuza
Research Fellow, International Laboratory for Evaluation of Practices and Innovations in Education, National Research University Higher School of Economics. Email: [email protected]
Address: 20 Myasnitskaya St, 101000 Moscow, Russian Federation.
Abstract. The present-day knowledge society expects school education to ensure the development of higher-order thinking skills, such as novel problem solving. Experimental evidence shows that such skills can be developed in students by using classroom activities enhancing higher-order thinking skills more often. However, the impact of such activities on knowledge acquisition in specific disciplines, mathematics in particular, remains unclear. Data obtained in the longitudinal study Trajectories in
Education and Careers conducted on a TIMSS-PISA sample is used to evaluate the presence of teaching practices that promote higher- and lower-order thinking skills in the classroom and the correlations between those strategies, on the one hand, and teacher characteristics and mathematics achievement at the end of 9th grade, on the other hand. Teaching practices of both types were found to be related positively to student achievement in mathematics. Yet, teaching practices that promote higher-order thinking skills have a stronger positive effect on mathematics achievement gains between 8th and 9th grades, whereas the effects of practices implying lower-order thinking lose their significance or become negative a year later. It is also shown that the use of a specific type of teaching practices is not related to teacher credentials or qualifications. Keywords: higher-order thinking skills, secondary school, mathematics, teaching practices, PISA, TIMSS, structural equation modeling.
DOI: 10.17323/1814-9545-2020-1-70-96
This article was prepared as part of the Basic Research Program administered by the National Research University Higher School of Economics, using the Russian Academic Excellence Project 5-100 grant.
Today, economic transformations and technological progress increase the significance of higher-order thinking skills, classified among skills for the 21st century [Pellegrino, Hilton 2012; Griffin, McGaw, Care 2012]. Historically, education systems were focused on memorization and rote learning. However, as the role and incidence of automated labor and routine tasks have been progressively declining, pres-
ent-day high school graduates already need skills that will help them succeed in the ever-changing world [Froumin et al. 2018].
According to Benjamin Bloom's taxonomy of educational objectives [Bloom 1956], thinking skills required for learning can be represented as levels of thinking complexity. For example, analysis- and evaluation-oriented tasks imply higher cognitive effort and promote higher-order thinking skills, whereas procedural tasks, such as two-variable linear equations, involve lower-order thinking skills. Higher-order thinking skills include analyzing, evaluating, and creating, while lower-order thinking skills include remembering, understanding, and applying [Anderson, Krathwohl 2001].
Effectiveness of individual methods and techniques of promoting higher-order thinking has been widely discussed in literature. Yet, how more exposure to such teaching practices in the classroom will influence achievements in specific subjects is still an open question. Reducing the significance of lower-order thinking skills implies not only using other types of tasks but also shifting the whole paradigm of classroom interactions. Thus, tasks promoting lower-order thinking rather imply instructional influence, as primary focus is made on teacher's instructions on how to solve the task and what the correct answer should look like [Paniagua, Istance 2018; Obukhov 2014]. Tasks targeting higher-order thinking skills have no algorithmic solutions; they usually imply multiple steps and have more than one correct answer to them [Resnick 1987]. To be solved, they require a dia-logical teacher-student interaction [Barr, Tagg 1995]. In both cases, a teacher's role in school learning remains pivotal, as teaching practices should seek to structure students' learning activities and encourage their engagement.
This study aims at analyzing the teaching practices promoting higher- and lower-order thinking skills as well as their relationship to secondary school students' mathematics achievements. Data of two international assessments, TIMSS1 and PISA2, served as empirical basis for research. Russia is the only country where the same students participated first in the TIMSS-2011 and then in the PISA-20123. The case of Russia is also very curious because international data shows that Russian middle school students are bad at solving tasks that involve higher-order thinking skills (e. g. matching information in a text
1 Trends in International Mathematics and Science Study monitors trends in mathematics and science achievement every four years, at the fourth and eighth grades. It includes questionnaires for students, teachers, and school administrators: timssandpirls.bc.edu.
2 Programme for International Student Assessment measures 15-year-olds' reading, mathematical, and scientific literacy every three years. It includes questionnaires for students and school administrators: oecd.org/pisa
3 As part of the longitudinal study Trajectories in Education and Careers (TrEC), which involved TIMSS and PISA as its first waves. Detailed information on TrEC can be found in [Malik 2018] and at trec.hse.ru.
with information in a table), while being great at applying familiar algorithms and reproducing rote-learnt material [Tyumeneva, Valdman, Carnoy 2014]. This difference in the performance of tasks requiring different levels of thinking skills was demonstrated for mathematics as well as science. Education programs in Russia are mainly oriented at using standard well-structured problems and memorizing algorithms [Bolotov, Sedova, Kovaleva 2012; Kapuza et al. 2017; Larina 2016; Froumin et al. 2018]. According to a national survey, most Russian teachers consider the development of higher-order thinking skills to be beyond the scope of school education objectives [Dobryakova, Yurchenko, Novikova 2018]. Therefore, it appears important and relevant to explore the results of using teaching practices that promote higher- and lower-order thinking skills in the Russian context. The following research questions are addressed in this study:
1. What is the relationship between the teaching practices promoting lower- and higher-order thinking skills and the professional characteristics of math teachers?
2. Is there a difference in the relationship between the teaching practices promoting lower- and higher-order thinking skills and students' mathematics achievements at the end of secondary school?
1. Review of Available Literature on the Relation of of Teaching Practices with Students' Thinking Skills and School Achievements
The relationship between specific teaching methods and techniques and students' achievement in mathematics has been widely studied across different countries and education systems. Research has involved data obtained in large-scale assessments in education as well as experimental classroom studies. Classroom teaching practices are usually divided into two groups. The first one includes tasks encouraging students to memorize facts, formulae, and rules for solving routine problems, while the other one includes tasks requiring students to process information individually, use information technology, work in small groups, etc. International researchers usually refer to practices from the former group as traditional, whereas references to the latter one are less consistent, including "modern" [Bietenbeck 2014; Lavy 2016], "inquiry-based" [Miri, David, Uri 2007], "active learning" [Cordero, Gil-Izquierdo 2018], and others. Given the nature of tasks, these two groups can also be referred to as practices promoting lower- or higher-order thinking skills. Abbreviations "LO practices" and "HO practices" will be used hereinafter in this study to refer to the two types of tasks used by teachers in the classroom. Teaching practices are interpreted here as a set of teacher's classroom activities, including teaching methods, specific techniques, and forms of classroom organization.
Experimental evidence shows that exposure to teaching practices that target higher-order thinking skills is positively related to the
development of this skills in students. For example, a longitudinal experimental study showed a positive relationship between such HO teaching strategies as dealing with real-life problems and encouraging open-ended class discussions and the level of students' critical thinking skills, which are classified as higher-order thinking skills [Miri, David, Uri 2007]. Another study with a similar design revealed that using HO practices in the classroom promoted conceptual understanding of content, drawing connections between facts and ideas, and encourage students to use higher-order thinking skills in problem solving [Baumert et al. 2010]. As a result, HO practices turned out to be positively related to students' mathematics achievements at the end of the tenth grade even if nine-grade performance was controlled for.
At the same time, HO teaching practices are also positively related to the development of lower-order thinking skills. Jacquelyn F. Gamino with co-authors assessed the impact of the Science Mathematics and Research for Transformation (SMART) program on two types of reasoning skills, fact-learning (i. e. LO thinking skills) and gist-reasoning (i. e. HO thinking skills) [Gamino et al. 2010]. Within the SMART program, teachers used tasks targeting HO thinking skills, where students had to learn to abstract meaning from texts by omitting unimportant details and summarizing. In another group, teaching practices were designed to develop rote memorization. Use of the SMART program was found to be positively related to both fact-learning and gist-reasoning. Meanwhile, LO practices only contributed to fact-learning performance.
The positive effects of HO practices on lower-order thinking skills was also demonstrated on a sample of seventh- and eighth-grade students in the United States [Cohen et al. 1997]. Students in the classrooms where social sciences were taught using HO practices (open-ended group work activities encouraging student interactions) performed much better on the tasks requiring higher-order thinking than their peers in the control classrooms. Meanwhile, no difference in performance between the experimental and control classrooms was found in the tasks requiring fact memorization, i. e. lower-order thinking.
Finally, the use of HO practices correlates positively with progress in students' self-regulation skills. An experimental study showed that higher-order strategy training improved inhibitory control%ability to control one's impulsive and automatic responses%among 12- to 15-year-olds [Motes et al. 2014]. In its turn, inhibitory control as part of executive functioning is positively related to students' math achievement (see, for instance, [Bull, Lee 2014; Liew 2012]).
Given the confirmed positive effects of HO practices on various thinking skills, one may assume that they can also be positively related to academic achievements. Otherwise speaking, if a math teacher begins to use, say, real-world problems more often, students' outcomes are expected to improve. However, an overwhelming majority of stud-
ies only reveal a positive correlation between traditional (LO) teaching practices and achievement in secondary school, whereas the effects of HO practices are either insignificant or negative in both mathematics and science. For instance, Spanish researchers used PISA and TALIS4 data to find out that traditional teaching methods had a positive influence on PISA scores in mathematics, whereas more innovative active learning strategies had a negative impact on student achievement [Cordero, Gil-Izquierdo 2018]. Similar findings were obtained from TIMSS data in the United States [Bietenbeck 2014] (mathematics and science) as well as in longitudinal studies conducted in the U.S. [Schwerdt, Wuppermann 2011] (mathematics and science) and Israel [Lavy 2016] (mathematics, science, Hebrew, and English). Finally, the Russian longitudinal study Trajectories in Education and Careers (TrEC) based on TIMSS and PISA data demonstrated that PISA scores in mathematics were positively related to exposure to formal mathematics concepts in the classroom and negatively related to exposure to applied mathematics concepts [Carnoy et al. 2016].
Despite confirmed impact of teaching practices on student achievement, most studies exploring the factors of academic performance at school zero in on teachers' professional characteristics [Hanushek, Rivkin 2006; Ladd 2008; Wayne, Youngs 2003]. In particular, a significant positive correlation was found between student achievement and teacher credentials and qualifications [Carnoy et al. 2016; Clotfelter, Ladd, Vigdor 2007], years of experience [Clotfelter, Ladd, Vigdor 2007; Rivkin, Hanushek, Kain 2005; Tyumeneva, Kha-venson 2012], and the Russia-specific characteristic of teacher category [Carnoy et al. 2016; Zakharov, Carnoy, Loyalka 2014]. For instance, student performance was found to be positively related to teachers' mathematics preservice training in university mathematics departments rather than faculties of education [Carnoy et al. 2016]. Effects of such teacher characteristics manifest themselves most notably in the academic performance of students from families of middle and high socioeconomic status.
As we can see, the studies carried out so far rather indicate a negative or insignificant relation between HO practices and mathematics achievement. However, a number of limitations should be considered that are typical of data collection methods used in those studies. First, PISA mostly obtains information on classroom teaching practices from the student questionnaire, with the exception of few countries [0ECD2013]. Meanwhile, dispersed opinions among students may result in assessment bias. Second, surveys collect information on teaching practices simultaneously with testing, whereas experimental studies allow to evaluate the delayed effects of teach-
4 Teaching and Learning International Survey: http://www.oecd.org/edu/ school/talis.htm
ing practices, measuring the performance of both teachers and students at least twice. Attempts to assess the delayed effects using longitudinal data have been made in a number of studies, but they only analyze teachers' characteristics, not the practices that they use [Clotfelter, Ladd, Vigdor 2007; Rivkin, Hanushek, Kain 2005; Wayne, Youngs 2003]. Finally, the relationship between teaching practices and student achievement may differ depending on the characteristics and specific features of education systems [Caro, Lenkeit, Kyr-iakides 2016].
This study attempts to overcome the limitations of international surveys as sources of information on the relationship between teaching practices and school students' mathematics achievement. On the one hand, we use the TIMSS teacher questionnaire to find out what types of tasks teachers use in the classroom and how often, thus solving the problem of assessment reliability. On the other hand, our data is longitudinal, allowing to evaluate the delayed effects of different teaching practices. Longitudinal data will help us answer, using a value-added production function, the question, how exposure to tasks targeting higher- and lower-order thinking skills is related to students' mathematics achievements.
The study uses data from the first two waves of TrEC — the international assessments TIMSS-2011 (eighth grade) and PISA-2012 (ninth grade) conducted in Russia on the same representative sample of students. A total of 4,893 students from 231 classes participated in the study in 2011 and 4,472 students from 229 classes participated in 2012. For the purposes of this study, only teachers who had taught the participating students since at least the eighth grade were included in the sample. The resulting sample thus comprised 3,472 students and 185 teachers of mathematics.
2.2. Variables Student achievement was measured using the TIMSS and PISA instruments and converted to a 1,000-point scale. We used standardized TIMSS and PISA scores in mathematics. In both assessments, students' performance is represented as five probabilistic scores; Rubin's combination rules were applied to include probabilistic scores in our analysis [Rubin 1987].
In addition to student performance data, we also used data obtained from the 2011 and 2012 contextual questionnaires for students, teachers, and principals. Teacher questionnaires provided data on teachers' educational background and category, type of school and math program, and teaching practices. The question about teaching practices used in mathematics classes was asked as part of the TIMSS-2011 (eighth grade): "In teaching mathematics to this class, how often do you usually ask students to do the following?" (Question 19). Teachers were asked to choose one of four response options
2. Research Methodology
2.1. Data
("Every or almost every lesson", "About half the lessons", "Some lessons", or "Never") for each of eleven teaching practices:
a) Listen to me explain how to solve problems
b) Memorize rules, procedures, and facts
c) Work problems (individually or with peers) with my guidance
d) Work problems together in the whole class with direct guidance from me
e) Work problems (individually or with peers) while I am occupied by other tasks
f) Apply facts, concepts, and procedures to solve routine problems
g) Explain their answers
h) Relate what they are learning in mathematics to their daily lives
i) Decide on their own procedures for solving complex problems
j) Work on problems for which there is no immediately obvious method of solution k) Take a written test or quiz
Analysis also included control variables. Firstly, there was a students' socioeconomic status, because it has been proven by a number of studies to be a strong predictor of academic achievement, in mathematics in particular [Kuzmina 2016; Chirkina 2018; Khavenson, Chirki-na 2019]. Socioeconomic status of students was measured using two indicators from the student questionnaire, mother's education (1 for mothers who have a Bachelor's degree or above, and 0 if otherwise) and the number of books at home (1 for more than 100 and 0 for less) [Khavenson 2016; Bodovski, Chykina, Khavenson 2019]. Secondly, we used a population size obtained from the questionnaire for school principals as well as aggregated class characteristics, such as class size, percentage of female students, and average socioeconomic status of students. Descriptive statistics for all the variables used for analysis is given in Appendix 1, and for teachers' responses to questions on teaching practices, in Appendix 2.
2.3. Analysis Structural equation modeling (SEM) was used to answer the research Strategy questions. A few models of mathematics teaching practices were constructed and assessed using confirmatory factor analysis in order to construct scales for LO and HO teaching practices. Then, the relation of the two types of practices to teacher and class characteristics was assessed using SEM.
SEM was also used to find out whether there was any difference in the impact of relationship between HO and LO teaching practices and mathematics achievement at the end of secondary school (Figure 1). At this stage, relationship between the teaching practices and mathematics achievement in the eighth (TIMSS) and ninth (PISA) grades was assessed, while controlling for student, class, teacher,
Figure 1. Full analysis model
JS
^HO h
Student characteristics
Teacher and school characteristics
\
.V /
,-y TIMSS
PISA
(k
/ \
XS>
*©
V
TLO
J
3. Results
3.1. Constructing the types of mathematics teaching practices
and school characteristics. Since data was clustered, regression residuals were corrected using the Huber-White robust standard errors.
Three groups of models were created to analyze the relationship between teaching practices and student performance in the two assessments. Dependent variables were represented by TIMSS scores in mathematics in the first group, PISA scores in mathematics in the second one, and PISA scores while controlling for TIMSS performance in the third one (full model) (Figure 1). Three models were constructed in each group, LO practices being the independent variable in the first one, HO practices in the second one, and both types of practices in the third one. All the models controlled for student, teacher, and school characteristics.
According to a revised version of Bloom's taxonomy of thinking skills [Anderson, Krathwohl 2000], teaching practices were classified under one of the two types depending on whether they targeted lower-or higher-order thinking skills (Table 1).
Next, confirmatory factor analysis was used to test a two-factor model of teaching mathematics to a class (four practices within each factor). The total of three models were tested. Model 1 included all practices as a single factor, Model 2 discriminated between the LO and HO factors in compliance with the theoretical model, and Model 3 controlled for covariance between the two factors. As judged by the fit indices, Model 3 describes the data best of all (Table 2).
Figure 2 presents the confirmed factor structure and factor loadings for each teaching practice. Within the HO factor, the highest factor loadings are observed for j) "work on problems for which there is no immediately obvious method of solution" and i) "decide on their own procedures for solving complex problems." Within the LO factor, the highest factors loadings are observed for b) "memorize rules,
Table 1. Distribution of classroom activities listed in Question 19 of the teacher questionnaire between the two types of teaching practices*
LO factor HO factor
a) Listen to me explain how to solve problems b) Memorize rules, procedures, and facts d) Work problems together in the whole class with direct guidance from me f) Apply facts, concepts, and procedures to solve routine problems e) Work problems (individually or with peers) while I am occupied by other tasks h) Relate what they are learning in mathematics to their daily lives i) Decide on their own procedures for solving complex problems j) Work on problems for which there is no immediately obvious method of solution
"Lines c), g), and k) were excluded from analysis, as no response distribution was obtained for them. Table 2. Fit indices for the constructed models of teaching practices
One factor Two factors Two factors with covariance
RMSEA (Root Mean Square Error of Approximation) 0.145 0.097 0.090
CFI (Comparative Fit Index) 0.684 0.856 0.884
TLI (Tucker Lewis index) 0.557 0.799 0.829
SRMR (Standardized Root Mean Square Residual) 0.105 0.091 0.064
procedures, and facts", and the lowest ones for d) "work problems together in the whole class with direct guidance from me." The two factors are significantly related to each other, yet the correlation coefficient is low (0.28).
3.2. Relationship between mathematics teaching practices and teacher, class, and school characteristics
The use of the specified teaching practices is not related to such teacher characteristics as category or preservice degree, yet it is related to some of the class and school characteristics (Table 3). LO practices are less likely to be used in advanced math classes as well as in medium-sized cities and large towns (with populations of 100,000-500,000) as compared to densely populated urban areas (cities with populations of over 500,000). However, these coefficients have low levels of significance. As for HO practices, the relationship with population size is nonlinear: HO practices are more likely to be used in schools located in remote rural areas (less than 3,000 people) and small towns or villages (50,000-100,000) than in large cities. While controlling population size, teachers are using HO practices in large classes more often.
HO
4
Figure 2. Factor structure of the types of mathematics teaching practices
0.47.............> e) Work problems (individually or with
peers) while I am occupied by other tasks (3.6)
0.64.............> h) Relate what they are learning i
n mathematics to their daily lives (3.6)
0.72..............»- i) Decide on their own procedures for
solving complex problems (3.2)
0.70.............»- j) Work on problems for which there is
no immediately obvious method of solution (3.6)
0.78
0.59
©
0.48
0.51
0.28
LO
0.50
• a) Listen to me explain how to solve <■.............
problems (5.8)
0.72..............>■ b) Memorize rules, procedures, and facts (4) .............^s^) 0,
0.37.............> d) Work problems together in the whole
class with direct guidance from me (4.5)
0.43..............»- f) Apply facts, concepts, and procedures
to solve routine problems (5.7)
0.75
48
0.86
*.............©
0.82
3.3. Relationship between teaching practices and students' mathematics achievement
Let us first examine the results for the first and second groups of models in which the relationship between teaching practices and students' scores is analyzed separately for TIMSS and PISA (Table 4). LO practices show no significant correlation with TIMSS scores, while a one-SD increase in exposure to HO practices improves mathematics achievements by 0.7 SD (Table 4, Models 1 and 2). Initially, LO practices are significantly positively related to PISA scores. The relationship between HO practices and PISA performance is significantly positive, no matter whether LO practices are controlled for or not. In both cases, a one-SD increase in exposure to HO practices in the classroom improves mathematics achievements by 0.71 and 0.75 SD, respectively. Importantly, when both types of practices are included in the model, the LO correlation coefficient changes radically for both TIMSS and PISA and becomes significantly negative.
Analysis of the third group of models, with PISA scores as the dependent variable and TIMSS scores being controlled for, shows the same positive relationship between LO practices and PISA scores (Tab le 5) as without controlling for TIMSS scores (Table 4, Model 4).
Table 3. Characteristics of teachers and classrooms in which LO and HO practices are used
Type of teaching practices
LO HO
Teacher preservice (Reference category: math education -0.00 -0.09
degree): math degree (0.09) (0.07)
Teacher preservice: no math education -0.10 -0.06
(0.08) (0.07)
Type of school (1 = lyceum/gymnasium) -0.03 0.05
(0.10) (0.08)
Advanced math class (1 = Yes) -0.20* 0.01
(0.12) (0.07)
Teacher category (Reference category: highest): first 0.09 -0.00
(0.07) (0.06)
Teacher category: second or none -0.05 0.07
(0.11) (0.10)
Classroom size -0.01 0.02***
(0.01) (0.01)
Percentage female (%) -0.21 -0.02
(0.26) (0.23)
Books at home: 100+ (%) 0.09 0.17
(0.25) (0.20)
Mother's education: Bachelor's degree or above (%) 0.13 0.10
(0.22) (0.16)
Population (Reference category: over 500,000): 100,000- -0.23* 0.10
500,000 (0.13) (0.09)
Population: 50,000-100,000 -0.15 0.24**
(0.19) (0.12)
Population: 15,000-50,000 -0.03 0.15*
(0.11) (0.08)
Population: 3,000-15,000 -0.10 0.14
(0.11) (0.10)
Population: less than 3,000 -0.07 0.45***
(0.13) (0.17)
N of observations 3,414 3,414
Standard errors in parentheses * p<0.1; ** p<0.05; *** p<0.01.
However, the trend of the previous models persists: as soon as the use of HO practices is controlled for, the relationship between LO practices and PISA scores becomes negative, even though insignificant. Meanwhile, the relationship between HO practices and PISA perfor-
Table 4. The impact of teaching practices on TIMSS and PISA scores
TIMSS PISA
(1) (2) (3) (4) (5) (6)
LO practices 0.01 0.27*** 0.11*** -0.16***
(0.01) (0.01) (0.02) (0.03)
HO practices 0.70*** 0.78*** 0.71*** 0.75***
(0.02) (0.02) (0.02) (0.03)
Number of books at home 0.15*** 0.15*** 0.15*** 0.17*** 0.17*** 0.17***
(0.01) (0.01) (0.01) (0.02) (0.02) (0.02)
Mother's education 0.11*** 0.11*** 0.11*** 0.10*** 0.10*** 0.10***
(0.01) (0.01) (0.01) (0.02) (0.02) (0.02)
Control variables Yes Yes Yes Yes Yes Yes
Constant 0.29*** 0.20*** 0.24*** -1.00*** -0.97*** -1.01***
N of observations 3,394 3,413 3,357 3,394 3,413 3,357
Standard errors in parentheses * p<0.1; ** p<0.05; *** p<0.01.
Table 5. Relationship between teaching practices and PISA scores, with TIMSS scores being controlled for
PISA scores, when controlling for TIMSS scores
(7) (8) (9)
LO practices 0.10*** -0.01
(0.02) (0.03)
HO practices 0.32*** 0.31***
(0.02) (0.03)
TIMSS scores in mathematics 0.59*** 0.56*** 0.56***
(0.01) (0.01) (0.01)
Number of books at home 0.08*** 0.09*** 0.09***
(0.01) (0.01) (0.01)
Mother's education 0.03* 0.04* 0.04*
(0.02) (0.02) (0.02)
Control variables Yes Yes Yes
Constant -0.83*** -0.85*** -0.88***
N of observations 3,394 3,413 3,357
Standard errors in parentheses * p<0.1; ** p<0.05; *** p<0.01
mance remains positive, although the correlation coefficient is nearly twice as low as in the model that does not control for TIMSS scores, a one-SD increase in exposure to HO practices improving PISA scores by 0.31 SD.
While the coefficients of correlation between the number of books at home and PISA scores decrease little when controlling for TIMSS scores and retain the significance level of p<0.01, the relationship between mother's education and PISA scores loses an essential part of its significance (Table 4, Models 4-6; Table 5, Models 7-9).
4. Discussion The findings of this study indicate that the use of teaching practices promoting both higher- and lower-order thinking skills is positively related to students' mathematics achievements in the eighth and ninth grades. However, the effects differ in size, HO practices (such as "decide on their own procedures for solving complex problems") proving to be much more effective. Such results are valid only if LO or HO practices are analyzed individually. When both types of practices are controlled for, the relationship between LO practices and mathematics achievement becomes either insignificant (eighth grade) or negative (ninth grade), while the impact of HO practices becomes even stronger.
A value-added production function was used to analyze the delayed effects of both types of practices on mathematics achievement. At the end of the academic year, correlations between HO and LO practices and achievement gains remain the same as they were at the beginning of the year. The impact of HO practices remains positive, though less significant, and that of LO practices loses its significance when controlling for previous achievement. In other words, the positive effects of teaching practices promoting higher-order thinking skills maintain their significance over a year.
This study thus demonstrates that using tasks that target higher-order thinking skills in the classroom is more preferable for improving mathematics achievement. The larger positive impact of HO practices was also confirmed when they were used together with LO practices, and even one academic year later. Otherwise speaking, such practices as solving complex problems or relating knowledge to daily lives have positive effects on mathematics performance in any situation, which is not the case with practices promoting lower-order thinking.
No significant correlations were found between the preferred type of teaching practices and teacher or classroom characteristics (such as percentage of female students, percentage of students from families of low socioeconomic status, etc.). LO practices are less likely to be used in advanced math classes, but this difference does not relate to the observed effectiveness of HO practices. Consequently, the more effective HO practices can be applied by all teachers and will be useful for teaching mathematics to classes of any type.
Theoretical value of this study consists in shedding light on the relationship between teaching practices promoting higher-order thinking skills and students' mathematics achievement. Our findings are inconsistent with those obtained in other studies that are also based on international student assessments. To some extent, it may be due to using a value-added production function and drawing information about teaching practices from teacher questionnaires in this study. However, taking into account the possible effects of national contexts, the revealed effects should be tested in other education systems as well.
The inferences made in this study have a high practical value for the contemporary school education policies concerned about developing skills for the 21st century. However, correct interpretation of results obtained from such large-scale surveys would apply to trends at the level of the whole student population. Further research is needed to develop more detailed recommendations for teachers. For example, it would make sense to analyze the impact of LO and HO practices on subsamples differing in students' socioeconomic status and academic performance. Martin Carnoy and his colleagues, for instance, showed that the relationship between teaching practices and academic achievement of Russian students from low socioeconomic backgrounds differed as a function of their previous achievement [Carnoy et al. 2016]. For this group of students with middle and high initial (TIMSS) math scores, exposure to formal mathematics in the classroom improved their achievements more effectively than exposure to applied math and word problems. Meanwhile, no significant correlation was found between achievements of initially low-achieving students and the types of practices used by teachers in the classroom. In addition, a number of studies have shown that teachers tend to choose teaching practices promoting higher- or lower-order thinking skills depending on students' previous achievement [Zohar, Albo-her Agmon 2018; Zohar, Dori 2003]. Therefore, subsampling will make it possible to develop more specific guidelines on applying individual teaching practices to different groups of students.
Restriction of analysis to a single branch of knowledge limits the range of opportunities for interpretation. In our case, it is necessary to keep in mind that mathematics has always been regarded in Russia as a unique tool for promoting "intellectual development in mainstream school" [Kozlov, Kondakov 2011:36]. Mathematical education has been a significant factor of social segregation on the global scale [Jorgensen, Gates, Roper 2014]. Therefore, further research should measure the impact of different teaching practices on students' achievement in other subjects. Besides, it is important to point out that teacher self-report data was used in this study. Although such an approach may result in bias in assessing the exposure to particular practices [Kapuza, Tyumeneva 2016], direct observation would not have allowed answering the research questions asked in this study.
References Anderson L. W., Krathwohl D. (eds) (2001) A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives. New York: Longman.
Barr R. B., Tagg J. (1995) From Teaching to Learning — A New Paradigm For Undergraduate Education. Change: The Magazine of Higher Learning, vol. 27, no 6, pp. 12-26.
Baumert J., Kunter M., Blum W., Brunner M. (2010) Teachers' Mathematical Knowledge, Cognitive Activation in the Classroom, and Student Progress. American Educational Research Journal, vol. 47, no 1, pp. 133-180.
Bietenbeck J. (2014) Teaching Practices and Cognitive Skills. Labour Economics, vol. 30 (C), pp. 143-153.
Bloom B. S. (ed.) (1956) Taxonomy of Educational Objectives: The Classification of Educational Goals. London: Longmans, Green.
Bodovski K., Chykina V., Khavenson T. (2019) Do Human and Cultural Capital Lenses Contribute to Our Understanding of Academic Success in Russia. British Journal of Sociology of Education, vol. 40, no 3, pp. 393-409.
Bolotov V., Sedova Y.., Kovaleva G. (2012) Sostoyanie matematicheskogo obra-zovaniya v RF: obshchee srednee obrazovanie (analiticheskiy obzor) [The Status of Mathematics Education in Russia: Middle School (Analytical Review)]. Problems of Modern Education, no 6, pp. 32-47.
Bull R., Lee K. (2014) Executive Functioning and Mathematics Achievement. Child Development Perspectives, vol. 8, no 1, pp. 36-41.
Carnoy M., Loyalka P., Khavenson T., Zakharov A., Schmidt W. (2016) Revisiting the Relationship Between International Assessment Outcomes and Educational Production: Evidence From a Longitudinal PISA-TIMSS Sample. American Educational Research Journal, vol. 53, no 4, pp. 1054-1085.
Caro D. H., Lenkeit J., Kyriakides L. (2016) Teaching Strategies and Differential Effectiveness across Learning Contexts: Evidence from PISA 2012. Studies in Educational Evaluation, vol. 49, September, pp. 30-41.
Chirkina T. (2018) Sotsialno-ekonomicheskoe polozhenie i vybor obrazovatel-noy traektorii uchashchimisya: teoreticheskie podkhody k izucheniyu vzai-mosvyazi [Review of Theoretical Approaches to the Study of the Relationship between Students' Socioeconomic Status and Educational Choice]. Eko-nomicheskaya sotsiologiya / Journal of Economic Sociology, vol. 19, no 3, pp. 109-125.
Clotfelter C. T., Ladd H. F., Vigdor J. L. (2007) Teacher Credentials and Student Achievement: Longitudinal Analysis with Student Fixed Effects. Economics of Education Review, vol. 26, no 6, pp. 673-682.
Cohen E. G. et al. (1997) What Did Students Learn? 1982-1994. Working for Equity in Heterogeneous Classrooms: Sociological Theory in Practice (eds E. G. Cohen, R. A. Lotan), New York: Teachers' College, pp. 137-165.
Cordero J. M., Gil-Izquierdo M. (2018) The Effect of Teaching Strategies on Student Achievement: An Analysis Using TALIS-PISA-link. Journal of Policy Modeling, vol. 40, no 6, pp. 1313-1331.
Dobryakova M., Yurchenko O., Novikova E. (2018) Navyki XXI v. v rossiyskoy shkole: vzglyad pedagogov i roditeley [21st Century Skills in the Russian School: Teacher and Parental Perspectives]. Sovremennaya analitika obra-zovaniya, iss. 4 (21), Moscow: NRU HSE.
Froumin I., Dobryakova M., Barannikov K., Remorenko I. (2018) Universalnye kompetentnosti i novaya gramotnost: chemu uchit segodnya dlya uspekha zavtra. Predvaritelnye vyvody mezhdunarodnogo doklada o tendentsiyakh transformatsii shkolnogo obrazovaniya [Universal Competencies and New Literacies: What to Teach Today to Succeed Tomorrow. Preliminary Findings of an International Report on Trends in the Transformation of School Education]. Sovremennaya analitika obrazovaniya, iss. 2 (19), Moscow: NRU HSE.
Gamino J. F., Bond Chapman S., Hull E. L., Reid Lyon G. (2010) Effects of Higher-Order Cognitive Strategy Training on Gist-Reasoning and Fact-Learning in Adolescents. Frontiers in Psychology, vol. 1, December, pp. 1-16.
Griffin P., McGaw B., Care E. (eds) (2012) Assessment and Teaching of 21st Century Skills. London: Springer.
Hanushek E. A., Rivkin S. G. (2006) Teacher Quality. Handbook of the Economics of Education (eds E. A. Hanushek, F. Welch), Elsevier, vol. 2, pp. 1051-1078.
Jorgensen R., Gates P., Roper V. (2014) Structural Exclusion through School Mathematics: Using Bourdieu to Understand Mathematics As a Social Practice. Educational Studies in Mathematics, vol. 87, no 2, pp. 221-239.
Kapuza A., Kersha Y., Zakharov A., Khavenson T. (2017) Obrazovatelnye rezul-taty i sotsialnoe neravenstvo v Rossii: dinamika i svyaz s obrazovatelnoy poli-tikoy [Educational Attainment and Social Inequality in Russia: Dynamics and Correlations with Education Policies]. Voprosy obrazovaniya / Educational Studies Moscow, no 4, pp. 10-35. DOI: 10.17323/1814-9545-2017-4-10-35
Kapuza A., Tyumeneva Yu. (2016) Nadezhnost i struktura shkaly sotsialnoy zhe-latelnosti TALIS: otsenka v ramkakh sovremennoy teorii testirovaniya [Reliability and Dimensionality of the TALIS Scale of Social Desirability: Evidence from the Item Response Theory]. Monitoring obshchestvennogo mneniya: ekonomicheskie i sotsialnye peremeny, vol. 6, no 136, pp. 14-29.
Khavenson T. (2016) Kachestvo otvetov shkolnikov na voprosy o sotsialno-eko-nomicheskom polozhenii semyi [Quality of School Age Children Reports on Family's Socioeconomic Status]. Sotsiologicheskiy zhurnal / Sociological Journal, vol. 22, no 4, pp. 74-89.
Khavenson T., Chirkina T. (2019) Obrazovatelny vybor uchashchikhsya posle 9-go i 11-go klassov: sravnenie pervichnykh i vtorichnykh effektov sotsial-no-ekonomicheskogo polozheniya semyi [Student Educational Choice after the 9th and 11th Grades: Comparing the Primary and Secondary Effects of Family Socioeconomic Status], Zhurnal issledovaniy sotsialnoy politiki/ Journal of Social Policy Studies, vol. 17, no 4, pp. 539-554.
Kozlov V., Kondakov A. (eds.) (2011) Fundamentalnoe yadro soderzhaniya obsh-chego obrazovaniya [The Fundamental Guidelines for Middle School Curriculum Development], Moscow: Prosveshchenie.
Kuzmina Yu. (2016) Effekt «bolshoy lyagushki v malenkom prudu»: vsegda li khorosho, esli rebenok obuchaetsya v silnom klasse? [Big-Frog-in-a-Small-Pond Effect: Is It Always Good for a Child to Study in a Strong Class?]. Psik-hologiya. Zhurnal Vysshey shkoly ekonomiki / Psychology. Journal of the Higher School of Economics, vol. 13, no 4, pp. 712-740.
Ladd H. F. (2008) Teacher Effects: What Do We Know? Teacher Quality: Broadening the Debate (eds G. Duncan, J. Spillane), Evanston, IL: Northwestern University, pp. 3-26.
Larina G. (2016) Analiz prakticheskikh zadach po matematike: teoreticheskaya model i opyt primeneniya na urokakh [Analysis of Real-World Math Problems: Theoretical Model and Classroom Application]. Voprosy obrazovaniya / Educational Studies Moscow, no 3, pp. 151-168. DOI: 10.17323/18149545-2016-3-151-168
Lavy V. (2016) What Makes an Effective Teacher? Quasi-Experimental Evidence. CESifo Economic Studies, vol. 62, no 1, pp. 88-125.
Liew J. (2012) Effortful Control, Executive Functions, and Education: Bringing Self-Regulatory and Social-Emotional Competencies to the Table. Child Development Perspectives, vol. 6, no 2, pp. 105-111.
Malik V. (2018) The Russian Panel Study 'Trajectories in Education and Careers'. Longitudinal and Life Course Studies, vol. 10, no 1, pp. 125-144.
Miri B., David B. C., Uri Z. (2007) Purposely Teaching for the Promotion of Higher-Order Thinking Skills: A Case of Critical Thinking. Research in Science Education, vol. 37, no 4, pp. 353-369.
Motes M. A., Gamino J. F., Bond Chapman S., Patel N. (2014) Inhibitory Control Gains from Higher-Order Cognitive Strategy Training. Brain and Cognition, vol. 84, no 1, pp. 44-62.
Obukhov A. (ed.) (2014) Psikhologo-pedagogicheskoe vzaimodeystvie uchast-nikovobrazovatelnogo protsessa: uchebnikipraktikum dlyaakademicheskogo bakalavriata [Psycho-Pedagogical Relationship Among Stakeholders in Education: Textbook and Tutorial for Bachelor's Degree Studies], Moscow: Yurayt.
OECD (2013) PISA 2012. Assessment and Analytical Framework: Mathematics, Reading, Science, Problem Solving and Financial Literacy. Available at: https://www.oecd.org/pisa/pisaproducts/PISA%202012%20framework%20 e-book_final.pdf (accessed 31 January 2020).
Paniagua A., Istance D. (2018) Teachers as Designers of Learning Environments, Educational Research and Innovation. Paris: OECD.
Pellegrino J. W., Hilton M. L. (eds) (2012) Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century. Washington, DC: The National Academy.
Resnick L. B. (1987) Education and Learning to Think. Washington, D.C.: National Academy.
Rubin D. (1987) Multiple Imputation forNonresponse in Surveys. New York: John Wiley & Sons Inc.
Rivkin S. G., Hanushek E. A., Kain J. F. (2005) Teachers, Schools, and Academic Achievement. Econometrica, vol. 73, no 2, pp. 417-458.
Schwerdt G., Wuppermann A. C. (2011) Is Traditional Teaching Really All that Bad? A Within-Student Between-Subject Approach. Economics of Education Review, vol. 30, no 2, pp. 365-379.
Tyumeneva Y., Valdman A., Carnoy M. (2014) Chto dayut predmetnye znaniya dlya umeniya primenyat' ikh v novom kontekste. Pervye rezultaty sravnitel-nogo analiza TIMSS-2011 i PISA-2012 [What Does Subject Knowledge Give for Its Applying in New Context. The First Results from Studies TIMSS-2011 and PISA-2012]. Voprosy obrazovaniya/Educational Studies Moscow, no 1, pp. 8-24.
Tyumeneva Y., Khavenson T. (2012) Kharakteristiki uchiteley i dostizheniya shkol-nikov. Primenenie metoda firstdifference k dannym TIMSS-2007 [Teacher Characteristics and Student Performance at School. Applying the First Difference Method to TIMSS2007 Data]. Voprosy obrazovaniya / Educational Studies Moscow, no 3, pp. 113-140.
Wayne A. J., Youngs P. (2003) Teacher Characteristics and Student Achievement Gains: A Review. Review of Educational Research, vol. 73, no 1, pp. 89-122.
Zakharov A., Carnoy M., Loyalka P. (2014) Which Teaching Practices Improve Student Performance on High-Stakes Exams? Evidence from Russia. International Journal of Educational Development, no 36, pp. 13-21.
Zohar A., Alboher Agmon V. (2018) Raising Test Scores vs. Teaching Higher Order Thinking (HOT): Senior Science Teachers' Views on How Several Concurrent Policies Affect Classroom Practices. Research in Science and Technological Education, vol. 36, no 2, pp. 243-260.
Zohar A., Dori Y. J. (2003) Higher Order Thinking Skills and Low-Achieving Students: Are They Mutually Exclusive? The Journal of the Learning Sciences, vol. 12, no 2, pp. 145-181.
Appendix Appendix A. Descriptive statistics for variables included in analysis
Mean SE
Standardized TIMSS scores in mathematics 0.02 0.12
Standardized PISA scores in mathematics 0.04 0.11
Population size: > 500,000 0.31 0.46
Population size: 100,000-500,000 0.22 0.41
Population size: 50,000-100,000 0.08 0.27
Population size: 15,000-50,000 0.17 0.38
Population size: 3,000-15,000 0.14 0.35
Population size: < 3,000 0.08 0.27
Teacher preservice: math degree 0.14 0.35
Teacher preservice: math education degree 0.67 0.47
Teacher preservice: no math education 0.19 0.39
School type: lyceum/gymnasium 0.20 0.40
Class size 21.07 4.77
Percentage female 0.50 0.14
Books at home: 100+,% 0.33 0.19
Mother's education: Bachelor's degree or above,% 0.46 0.23
Advanced math class 0.13 0.33
Teacher category: highest 0.41 0.49
Teacher category: first 0.43 0.50
Teacher category: second or none 0.16 0.37
Books at home: 100+ 0.33 0.47
Gender=female 0.50 0.50
Mother's education: Bachelor's degree or above 0.46 0.50
Appendix B. Descriptive statistics for teachers' responses to Question 19 of the teacher questionnaire
Mean ; SE
In teaching mathematics to this class, how often do you usually ask students to do the
following?
Listen to me explain how to solve problems
Some lessons 0.08 ; 0.27
About half the lessons 0.21 ! 0.41
Every or almost every lesson 0.71 i 0.45
Mean SE
Memorize rules, procedures, and facts
Never 0.02 0.13
Some lessons 0.19 0.40
About half the lessons 0.43 0.50
Every or almost every lesson 0.36 0.48
Work problems together in the whole class with direct guidance from me
Some lessons 0.16 0.37
About half the lessons 0.33 0.48
Every or almost every lesson 0.50 0.51
Apply facts, concepts. and procedures to solve routine problems
Some lessons 0.09 0.29
About half the lessons 0.16 0.37
Every or almost every lesson 0.75 0.44
Work problems (individually or with peers) while I am occupied by other tasks
Never 0.02 0.13
Some lessons 0.49 0.50
About half the lessons 0.37 0.48
Every or almost every lesson 0.13 0.34
Relate what they are learning in mathematics to their daily lives
Never 0.01 0.11
Some lessons 0.49 0.50
About half the lessons 0.36 0.48
Every or almost every lesson 0.13 0.34
Decide on their own procedures for solving complex problems
Never 0.17 0.37
Some lessons 0.68 0.47
About half the lessons 0.13 0.34
Every or almost every lesson 0.02 0.16
Work on problems for which there is no immediately obvious method of solution
Never 0.19 0.39
Some lessons 0.72 0.45
About half the lessons 0.09 0.29
