What the New Measure of Thinking in School Students Has to Offer to Contemporary Education

Irina L. Uglanova; Irina N. Pogozhina

I. L. Uglanova, I. N. Pogozhina

Received in Irina L. Uglanova, Junior Research Fellow, Center for Psychometrics and Measure-

June 2021 ments in Education, Institute of Education, National Research University Higher School of Economics. Address: Bld. 10, 16 Potapovsky Ln, 101000 Moscow, Russian Federation. Email: iuglanova@hse.ru (corresponding author)

Irina N. Pogozhina, Doctor of Sciences in Psychology, Associate Professor, Department of Psychology of Education and Pedagogics, Faculty of Psychology, Lomonosov Moscow State University. Address: Bld. 9, 11 Mokhovaya St, 125009 Moscow, Russian Federation. Email: pogozhina@mail.ru

Abstract For the pedagogical principle of assigning comprehensible and adequate tasks to be implemented, allowance should be made for students' individual levels of logical reasoning, which requires diagnostic measures for objective and quick assessment. Today, the "clinical method" allows the most comprehensive assessment of logical thinking within the Piagetian framework. However, this diagnostic measure is extremely resource-consuming, hence unsuitable for large-scale testing. An overview of literature shows that the existing standardized diagnostic measures require a great number of highly-qualified experts to review the scores and prepare feedback for teachers, instructional designers, practicing psychologists and researchers.

The article describes design methodology of an instrument to evaluate levels of logical reasoning that will allow automated scoring without sacrificing score meaning, eventually facilitating and accelerating the diagnostic measurement procedure. Implementation of these principles is analyzed using the example of scenario-based tasks realized as computerized performance-based assessment in the form of stealth assessment of fifth- and seventh-grade pupils.

Keywords automated scoring, critical thinking, diagnostic measurement, formal operations, logical reasoning, logical thinking, performance-based assessment, Piaget, psychometrics, scenario-based problems, stealth assessment.

For citing Uglanova I. L., Pogozhina I. N. (2021) Chto mozhet predlozhit' novaya metodologiya ot-senki myshleniya shkol'nikov sovremennomu obrazovaniyu [What the New Measure of Thinking in School Students Has to Offer to Contemporary Education]. Voprosy obra-zovaniya/Educational Studies Moscow, no 4, pp. 8-34 https://doi.org/10.17323/1814-

9545-2021-4-8-34

Assigning comprehensible and doable tasks is a fundamental pedagogical principle. "All the subjects that are to be learned should be arranged so as to suit the age of the students, that nothing which is beyond their comprehension be given them to learn." [Comenius

1939:151]. Overall, the design of modern school curricula makes allowance for stages of mental development, so an "average" student encounters little difficulty in learning. At the same time, underachieve-ment in school education remains quite a pressing issue. One of its sources is that cognitive development rates of individual students are insufficient to enable them to learn what they are supposed to learn at a particular grade. There is empirical evidence of the relationship between academic gains in various disciplines and the level of intellectual development [Malhotra 2020; Watkins, Lei, Canivez 2007]. Comprehension of mathematical and physical concepts as well as of social norms and rules has been shown to require a certain level of logical reasoning [Inhelder, Piaget 1958; Piaget, Inhelder 2003].

Practices of the most successful education systems demonstrate that the problem of low academic achievement in school can be solved by personalizing the learning process, i. e. acknowledging that "every child is special" [Vainikainen et al. 2015; Hautamaki, Thuneberg 2019; Hienonen 2020]. Personalized learning requires, among other things, taking account of the individual levels of logical reasoning, indicated by students' performance on Piagetian tasks [DeVries 1974; Goldschmid 1967; Lawson, Renner 1975; Lovell, Shields 1967; Lawson, Blake, Nordland 1974]. Because conventional IQ test scores do not provide enough information on the composition and structure of logical operations to adapt curricula to individual levels of development, the modern school needs diagnostic measures that are suitable for mass application and objective, i. e. independent from subjective interpretations [Avila de, Pulos 1979; Hathaway, Hathaway-Theunissen 1975; Kaufman 1972]. A new diagnostic measure would help understand the cognitive sources of low achievement in individual children and adolescents so as to build personalized educational trajectories with due regard to individual psychological characteristics.

The present study seeks to identify and describe the stages in design of an automated-scoring instrument to measure school students' levels of logical reasoning.

1. The Construct of Logical Thinking: Composition and Structure

Before describing the instrument design methodology, it would be natural to define the construct that will be measured.

Logical thinking is not something we have by default; objectivity increases over the four stages of development between birth and maturity: (1) sensorimotor period; (2) pre-operational thought; (3) concrete operations; and (4) formal operations. From stage to stage, regular cognitive patterns that capture associations between various characteristics of reality are organized into increasingly generalized structures (logical operations). At the level of behavior, we observe changes in the individual's perceptions of the objects in the world around, their properties, space, time, motion, causality, etc. [Piaget 1994a; 1994b; 1994c].

A great contribution to success in middle and high school is made by logical structures that emerge at the stage of formal operations:

(1) (1) Propositional logic: ability to engage in hypothetical reasoning, i. e. identify (reflect) consistent patterns and associations not only between real-world objects and their visual representations but also between propositions of different language systems. Includes the following propositional operations: if ... then (implication), or (disjunction (either or both)), and (conjunction), not (negation), if and only if (equivalence), etc.;

(2) (2) Combinatorial logic: conditional combination of various objects in all possible configurations, e. g. when any six (seven, ten, ... n) objects are systematically combined into sets of two, three, ... etc. in all possible ways without repetitions and with rigorous control of the outcome;

(3) (3) Synthesis of the two kinds of reversibility into a single cognitive structure of four transformations INRC (I is identity, or direct operation; N is negation of change, or inverse operation; R is reciprocity, or reciprocal operation, allowance for the impact of mutually related factors; and C is negation of reciprocity, the inverse of the reciprocal). The adolescent gains the ability to analyze problematic situations using all the four transformations at the same time.

At the latter level, the adolescent mind develops new operational schemata necessary for successful learning in school disciplines:

• Proportions (commensurability, equality of two or more ratios);

• Mechanical and homeostatic equilibrium (equality between action and reaction);

• Relative motion (motion in relation to a fixed system);

• Probabilities (odds of an event occurring in certain circumstances);

• Ability to go beyond the observed data, which involves hypothesizing (what if ...?), constructing a system of probable regular patterns, etc. [Piaget 1994a; 1994b; 1994c; Piaget, Inhelder 2003; Pia-get 2008].

As we can see, logical thinking represents a sophisticated system of interrelated structures.

2. Using the "Clinical Method" to Measure Logical Thinking

Logical thinking has been traditionally assessed using structured or semi-structured clinical interviews in which questions are asked by the interviewer (specifically trained expert) individually or in small groups. The purpose of a clinical interview is not restricted to scoring the product of problem solution as in standardized diagnostic measures; it also involves finding out how exactly the solution was reached, i. e. which cognitive processes were involved or not involved, and in what order.

Mistakes made while solving a problem indicate deficiencies in specific logical operations and thus are of high importance when interpreting the results. Using the clinical interview data, the expert makes inferences about the levels of specific logical operations and the overall level of logical thinking ability [Bringuier 2000; Piaget 1994a; 1994b; 1994c; Piaget, Inhelder 2003].

The disadvantage of the "clinical method" is its low level of resource efficiency, especially when using it on large samples or in large-scale assessments. First, there should be a sufficient number of highly-qualified experts capable of producing as unbiased diagnostic results as possible. Second, individually administered tasks make diagnosis essentially more time-consuming. In most contemporary educational situations, there are just not enough personnel and time for the administration of Piagetian tasks via the "clinical method" [Avila de, Pu-los 1979; Meyer 1972].

Therefore, researchers face the need to develop a new measure of logical thinking that would be a decent alternative to clinical interview. It should be easy to use for teachers, instructional designers, practicing psychologists, and researchers doing large-scale data collection. It should also preserve construct specifics, i. e. allow observing how cognitive processes forming part of logical operations get involved in problem solving.

Scalability of the instrument can be achieved by standardized testing, where all behaviors observed in a test are interpreted using diagnostic criteria that are uniform, pre-determined, and objective (unaffected by expert bias).

3. Using Standardized Methods to Measure Logical Thinking

Since the 1960s, Piaget's followers have attempted to replace individual testing (clinical interviews) with group testing in order to check correspondence between mandatory school curricula and pupils' cognitive levels, e. g. in teaching life science disciplines [Lovell 1961; Shayer 1978; Shayer, Küchemann, Wylam 1976].

Researchers enquired whether group testing could yield results as good as those of clinical interviews. It was shown that clinical interview and frontal assessment (e. g. when stimulus material is displayed on a screen and respondents write down their answers) produce comparable outcomes [Faust 1983; Renner et al. 1978; Rowell, Hoffmann 1975; Shayer 1979]. Yet, frontal assessment is not equal to standardized testing as it allows variations in how stimuli are presented and how results are interpreted.

We analyzed a number of studies describing the use of standardized diagnostic measures of logical thinking in the period of transition between the Piagetian stages of concrete and formal operational thinking in large-scale group-administered quantitative assessments. The main objective of their authors was to ensure the same degree of interpretation as in clinical interviews, while reducing the time and re-

sources required for test administration. The standardized instruments analyzed differ by a few critical parameters:1

(1) Test content:

a. Construct measured (logical operations);

b. Scope of measurement: test outcomes (product) and/or reasoning (process);

(2) Response format:

a. Selected response (multiple choice or yes/no (closed-ended) questions);

b. Essay or short constructed response (open-ended questions);

(3) Scoring:

a. Automated;

b. Manual (expert);

(4) Stimuli: physical objects, pictures, text, video.

The tests differ both in the set of logical operations assessed and the scope of measurement. Despite some differences in the types of logical operations that they measure, all the tests diagnose both concrete and formal operational thinking. As for the scope of measurement, the tests can only measure participants' final scores (product) [Tisher 1971; Raven 1973; Milakofsky, Patterson 1979; Avila de, Pulos 1979; Roberge, Flexer, 1982; Bergling 1998; Bakken et al. 2001]. However, some authors developed diagnostic instruments that assess both the product and process [Longeot 1962; 1965; Staver, Gabel 1979; Lawson 1978; Tobin, Capie 1981; Roadrangka 1991]. The latter is diagnosed via open-ended questions, which require manual scoring by experts.

On the one hand, the obvious benefits of closed-ended questions are reduced testing time and lower student stress as writing skills are not involved. In addition, multiple-choice testing ensures score objectivity, whereas manual scoring of constructed responses by different experts may cause inconsistency of scores and reduce their reliability [Roadrangka 1991]. On the other hand, the closed-ended format does not allow to obtain justifications for choosing the specific answer, thereby curtailing the diagnostic potential of the instrument and leaving open the possibility of answering correctly by making a random choice.

Stimuli are presented not only as plain text but also as a combination of text and pictures [Longeot 1962; 1965; Tobin, Capie 1981; Bergling 1998], pictorial material alone [Milakofsky, Patterson 1979; Avila de, Pulos 1979; Bitner-Corvin 1988], a combination of text and video [Staver, Gabel 1979; Tobin, Capie 1981], a combination of textual de-

1 The table at doi:10.17632/vxt3237yvt.1 presents a systematization of tests the psychometric characteristics of which are publicly accessible in peer-reviewed publications [Uglanova I. L., Pogozhina I. N. (2021) What the New Measure of Thinking in School Students Has to Offer to Contemporary Education. Mendeley Data, V1].

scriptions and physical objects [Roberge, Flexer, 1982], physical objects alone [Tisher 1971; Lawson 1978], etc.

Entirely verbal stimulus material is valid for assessing the development of formal operational thought; however, it implies a lot of reading, which can lead to semantic bias in the assessment of concrete operational structures. Furthermore, in the absence of physical objects as stimuli, students may stop perceiving the problem as significant [Lawson 1978]. Verbal stimulus materials and open-ended questions represent the greatest challenge for younger children [Lawson 1978; Roadrangka 1991].

On the whole, standardized tests as measures of the level of logical structures prove to be reliable and valid. As one of the ways of measuring their validity, results obtained in a standardized test are compared to results obtained in a clinical interview. Final reliability coefficients vary from insignificant on some scales [Staver, Gabel 1979] to significant at the level of 0.88 on others [Avila de, Pulos 1979; Bit-ner-Corvin 1988].

As we can see, the use of standardized measures allows to reduce testing time without sacrificing the scope of the logical reasoning construct to be measured. However, the problem of automated scoring has not been solved so far. In addition, there are no studies measuring logical operations in realistic situations, only in laboratory contexts.

Computerized performance-based assessment is a promising diagnostic measure of logical thinking (process and product) in technology-enhanced learning environments. This type of assessment allows to apply automated scoring, evaluate both the outcome and the reasoning process, and create real-life systems in every item [Wang, Shute, Moore 2015].

4. Requirements for the New Methodology

The new design methodology for diagnostic measures of logical thinking should, first of all, allow diagnosing the process of arriving at the solution and, second, provide feedback (which implies analysis of the results) for teachers, instructional designers, and practicing psychologists without the involvement of experts. Furthermore, the new measures should follow a principled assessment design framework [American Educational Research Association, American Psychological Association, National Council on Measurement in Education 2014; Messick 1992].

4.1. The process measurement requirement

Positively, "the nature of the construct guides the selection or construction of relevant tasks as well as the rational development of con-struct-based scoring criteria and rubrics" [Messick 1992:17]. Traditional multiple-choice items are not suitable for assessing the process as they only register the outcome (correct/incorrect) without making allowance for how it was achieved [Griffin, McGaw, Care 2012; Messick 1994; Razzouk 2011].

Prominent among the alternative modes of assessment are performance-based tasks [Messick 1994] which focus on both the product and process of solution. Today, computerized performance-based assessment is widely used to measure complex constructs in the form of games, simulations, and scenario-based tasks [Klerk de, Eggen, Veld-kamp 2016; Graesser, Kuo, Liao 2017; Sun et al. 2020]. However, performance-based diagnostic measures of logical thinking within the framework of Piaget's theory have not been described in literature yet.

4.2. The Transition from the "clinical method" to performance-based standardautomated scoring ized testing provides a unique combination of benefits of both multi-and feedback ple-choice and constructed-response items: the analysis limitation is requirement overcome by matching the examinee's product to the correct answer and assessing the solution process without involving experts.

Another advantage of performance-based testing is that it can be implemented in the format of stealth assessment, whereby test situations are embedded seamlessly into a computer-based learning or gaming environment such that the learner is unaware of being assessed [Wang, Shute, Moore 2015]. Stealth assessment helps to reduce test anxiety and maintain learners' engagement, while at the same time providing authentic contexts to make interpretation of assessment results as close as possible to everyday real-life problems.

Finally, the diagnostic framework should allow the use of test results to create psychological assistance programs for students whose levels of logical thinking are not enough to succeed in a discipline. The approach proposed in this article suggests decomposition of formal operational structures (activity-based diagnostic measurement, in some authors' terminology [Ilyasov 1986; Talyzina 2018]), which will allow assessment of not only the development of a logical operation as such but also the development of its constituent cognitive processes. Just as in clinical interviews, a lot of importance is assigned to analysis and interpretation of mistakes made during the solution process. Mistake is understood here as non-completion or incorrect completion of a cognitive task that is part of the logical operation measured and has to be completed to progress through the scenario. The developmental stage of the logical operation is determined based on the data collected.

In the future, the obtained diagnostic results can be used to elaborate an approximate action plan for promoting the development of the missing components of a "problematic" logical operation as well as to design and implement, within the activity-based approach to learning and socialization, a system of teaching practices required for such operation to be "internalized". The proposed methodology for diagnosing the composition and structure of logical operations will make it possible to accurately identify the "problem areas" in as many students as possible, so that each of them could be offered an appropriate personalized formative learning program.

5. The New Methodology and Instrument for Diagnosing Logical Thinking in School Students

The new methodology for diagnostic measurement of logical thinking proposed here involves the following:

• Identifying the logical operations as defined by Piaget [Piaget 1994a; 1994b; 1994c], their composition and structure as a diagnostic framework for activity-based diagnostic measurement and as the basis for design of formative learning programs in the future [Ilyasov 1986; Talyzina 2018];

• Analyzing the benefits and limitations of the currently applied measures of logical thinking;

• Applying modern psychometric techniques and digital technology.

To measure logical thinking in school students, we suggest using scenario-based tasks that put students into a technology-enhanced environment akin to learning and real-life contexts. In scenario-based tasks, examinees consecutively perform tasks that are interconnected within a context-enhanced story (scenario). The process and product of solution in such problems act as behavioral indicators allowing to assess the development of a specific cognitive process within a logical operation. Scenario-based tasks are similar in their format to computer games; unlike games, however, they involve less variation, which results in a more standardized assessment. Such an approach allows implementing the principles of stealth assessment for diagnostic measurement of logical thinking.

The diagnostic profiles constructed on the basis of test results describe levels of development of each logical structure (combinatorial logic, INRC group) measured. A teacher, psychologist, or parent can view information about the developmental stage for each logical operation separately to construct a comprehensive personal profile of logical thinking. As of now, scenario-based tests have been developed for fifth- and seventh-grade pupils.2

The new diagnostic methodology thus meets the design requirements for tasks measuring logical thinking and at the same time ensures that diagnostic measurement results can be scored without the involvement of experts.

To measure concrete operational thinking, test items are supported with physical objects or their visual representations (videos, pictures), as pupils mostly think in images at this stage of development. Emergence of formal operations in the adolescent's mind enables them to

2 The 4C's Project for assessment of 21st-century skills (critical thinking, creativity, communication, and collaboration) designed by the Centre for Psychometrics and Measurements in Education (Laboratory for New Construct Measurement and Test Design) of the HSE Institute of Education under an R&D contract with the Charitable Foundation "Investment to the Future".

6. Meeting the Process Measurement Requirement

6.1. Stimulus presentation

build complex logical relations mentally without interacting with physical objects or their visual representations. That is why a verbal mode of stimulus presentation is more relevant than a pictorial one for diagnosing the developmental stage of formal operations [Avila de, Pu-los 1979; Piaget 1994a; 1994b; 1994c].

The computerized scenario-based tasks that we propose allow to combine verbal and pictorial stimuli and simulate interactions with physical objects, which makes it possible to assess the examinee's ability to mentally manipulate various types of materials (physical, pictorial, symbolic) and ensure a more accurate measurement of the level of logical reasoning (concrete or formal operations stage).

6.2. Combinatorial Combinatorial operations are a logical structure emerging within the problems stage of formal operations. An example of a combinatorial problem for fifth-grade pupils is presented in Figure 1. The problem offers an imaginary scenario, asking students to make fuel for a space ship. The instructions are partially verbal, while the stimulus material imitates interaction with real-life objects, allowing examinees to try out all possible combinations of ingredients one by one while avoiding repetitions and omissions.

In the course of the solution process, we systematically evaluate the development of cognitive processes within combinatorial operations: (1) identification of variables to be combined in the problem scenario; (2) combination of variables presented in the problem scenario with regard to instructions (into sets of two, three, etc.); (3) control of variable combinations (systematic combination of variables into all possible sets in accordance with the given conditions while avoiding repetitions).

As the first step, the examinee explores the planet and collects ingredients for the fuel (Figure 1a), i. e. identifies the variables. After that, the examinee is given illustrated instructions for research (testing of all possible combinations of the ingredients collected) to produce fuel. Particular focus is placed on making sure that students fully understand the instructions (Figure 1b), i. e. they realize that combination of variables should be part of the solution process. Next, the examinee solves the problem (Figure 1c) by combining the variables into groupings that satisfy the given conditions and exercising control over the combinations.

As students are asked to combine stimulus components systematically in search for candidate solutions and allowance can be made for mistakes that they make, not only the product but also the process of solution can be assessed.

The final assessment of the development of combinatorial operations is based on the scores assigned for the product (the number of correct variable combinations) and the process of solution (absence of repetitions). Three stages of combinatorial operations development are identified in the diagnostic profile:

Figure 1. An example of a scenario-based problem measuring the development of combinatorial operations.

(a)

Collect the elements for making fuel and put them into the backpack. By clicking on planet locations, you can visit them and find various elements. Be mindful that there is little room in the backpack. Take only the four elements that are necessary to produce fuel. Open notebook Done

Collect the elements for making fuel and put them into the backpack.

By clicking on planet locations, you can visit them and find various

elements. Be mindful that there is little room in the backpack. Take

only the four elements that are necessary to produce fuel.

Open notebook

Approach 'Lake'

Close

(b)

Click this button to start a new combination

Create a combination

Back

Remember! Combinations are duplicate if they consist of the same

objects.

Back

Create all possible combinations of two flasks. Make sure you have no duplicate combinations. Be mindful that there should be no identical flasks and no empty boxes in a combination. Remove (x12) Create a combination Done

Create all possible combinations of three flasks. Make sure you

have no duplicate combinations. Be mindful that there should be no

identical flasks and no empty boxes in a combination.

Remove (x12)

Create a combination

Done

14

Voprosy obrazovaniya/ Educational Studies Moscow. 2021. No 4

Stage 1: Combinatorial operations are not developed. The student is able to identify the variables to be combined in a problem scenario

(1) but fails to produce combinations satisfying the given conditions

(2) as well as to control the process of combination by avoiding repetitions (3);

Stage 2: Combinatorial operations are partially developed. The student is able to identify the variables to be combined in a problem scenario (1) but makes mistakes in the process or fails to combine all the variables (2) as well as to control the process of combination by avoiding repetitions (3);

Stage 3: All the cognitive processes within combinatorial operations are fully developed. The student systematically generates all the correct configurations satisfying the problem statement and avoids repetitions.

6.3. INRC problems The emergence of the cognitive structure of four transformations INRC manifests itself in the ability to analyze the problem situation and examine the impact of all the given conditions one by one. By checking whether the situation is affected by a specific factor, the adolescent mentally performs two divergent operations, varying the factor characteristics (modifying the variables) and at the same time holding all the other factors constant (fixing the variables).

For instance, in a classic Piagetian task, the bending of rods with weights applied [Piaget, Inhelder 2003], to find out whether length, diameter, shape, and material have a difference in bending, the adolescent tests each variable one by one. To test for the effects of length, they select for comparison long and short rods that are alike in every other aspect: both are thick, round, and made of steel. Next, they bend the two rods and analyze the results. If the rods bend equally, a conclusion is drawn that length has no impact on bending, and vice versa. Effects of all the other variables in the scenario are tested in a similar way.

The same principle underlies the problem designed for seventh-graders (Figure 4). In a fantastic scenario, the examinee is asked to find out which residents of an old house have a chance of turning into a ghost.

Just as in combinatorial problems, this one allows to assess not only the outcome (correct/incorrect) but also the process of solution by measuring the development of processes within the logical operation of variable fixation as an indicator of INRC group [Ilyasov 1986; Baldina 1987; Pogozhina 2006]:

(1) Identify variables and their values in the problem scenario;

(2) Fix the variables: to determine the impact of variables on the outcome, the examinee has to compare situations in which all the variables except one are held constant;

Figure 2. An example of a scenario-based problem measuring the development of INRC group.

If you haven't sorted the evidence into three columns yet, it's just about time to do it. Classify all the pieces of evidence. Got it

I'm sick and tired of this ghost! And, most importantly, why does it sneeze? Sneezing should be done quietly, into a tissue. I remember my late uncle Vladimir Alexeevich who coughed in a very cultured manner. He would wrap his scarf around his neck, brew some herbal tea, and sit there groaning. You know, he would always suffer some imaginary illnesses, but I never saw him actually sick in his whole life

At least this ghost doesn't do any damage. We have so many fragile things here, like this antique porcelain tea set™ It belonged to Sergey Kon-stantinovich. He loved porcelain and was rarely seen without his favorite cup. Such a nice little thing Evidence

(a)

Evidence

So that's what you're up to, Sergey! Well, deep down inside, I knew this would come™ For what you've done, you will never find peace anymore and will be roaming forever like a restless ghost! Insufficient evidence Photo

Evidence Not a ghost

Natalya Vasilyevna getting dressed up for the theater

Tea