CC BY
3ПЕДАГОГИКА И ОБРАЗОВАНИЕ (PEDAGOGY & EDUCATION) УДК 371.261
Nyamgerel Ch.
Associate Professor Department of Chemistry, School of Arts and Sciences, National University of Mongolia (Ulaanbaatar, Mongolia)
CAN LEARNING SITUATION AND ITEM'S HINT BE FACTORS THAT DETERMINE THE KNOWLEDGE TYPE AND COGNITIVE PROCESS ASSESSED BY MULTIPLE-CHOICE ITEMS?
Abstract: in this article presents result of the study on knowledge types and cognitive process levels of chemistry four parallel test items of GEE-2013, based on previously done two studies. Each test (2013-A, B, C, D) consisted of 31 selected-response items. Tests were administrated in June 2013 to N=2930-2964 examinees. The Educational Evaluation Center of Mongolia using parallel tests to ensure that measures the same knowledge, skills, or abilities. According to the methodology we developed, each item was determined that refers to which of the Knowledge type of Bloom's Taxonomy. Due to the difference of "the learning situation", only item 2 belongs to different knowledge type. Items 2, 6, and 12 of the tests depend on different cognitive process level due to differences in learning status and hints. The result of this study suggests retaining "the learning situation" and "item's hint" as an important part of the test construction procedure.
Keywords: bloom's taxonomy, knowledge type, cognitive process, learning situation, item's hint
Introduction
Large-scale studies in the following fields are being conducted in countries: Developing selected-response items; Classifying items into cognitive levels of Bloom's taxonomy; Determining the compatibility between classifications made by different
subjects such as students and teachers; and Cognitive level validation of university entrance and secondary school graduation test structures [1, 2, 3, 4, 5, 6].
However, there are almost no published research articles or academic reports on the second dimension of Revised Bloom's Taxonomy [7, p. 28], which is the identification of knowledge types. It's critical for educators to align the assessment tasks with the cognitive process levels and the knowledge types they want to assess. Before the reader can classify a particular test exercise, he must be aware of or at least make some assumptions about the learning situations that preceded the test. In addition, he must also actually attempt to solve the test problem and note the mental processes he utilizes [8, p. 51]. We developed the "Analytical methodology for selected-response item" [9, p. 65], which considers the learning situation and hints in the item to determine its knowledge type and cognitive process level. After that, we did confirmatory factor analysis on the knowledge-type model and cognitive level model [10, 11].
The results of these two studies are compared with the report of the items analysis from the Educational Evaluation Center of Mongolia to explain some of the characteristics of the task defined as taking into account differences in Bloom's taxonomy, types of knowledge, and cognitive processes.
Methodology
This article describes tasks 2, 6, and 12 that were found to belong to different knowledge types and of cognitive process levels in the analysis of the selected-response items (4x31=124) of the four parallel tests (2013-A, 2013-B, 2013-C, and 2013-D) [10, p. 22, 11, p. 419].
The task coding used in the previous study was directly applied here. It includes:
2013-B-6 - 6th item of the 2013-B test; KN10102 - the 1st (01) of the items related to factual knowledge (KN1), the 2nd item of the test (_02); CL203_12 - the 3rd (03) of the items related to the "understanding" (CL2) of cognitive process, the 12th item of the test (_12)
First, the rationale for determining what knowledge type and cognitive process level the given items belong to is explained. Then it will be presented whether this definition is confirmed or rejected by the confirmatory factor analysis. Finally, based on the report of the distractor efficiency analysis, some features that influence the task to be an indicator of the knowledge type and cognitive process level (latent variable) are explained for each item [10, p. 25, 11, p. 420].
Result and Discussion
When analyzing 31 items for four parallel tests in 2013, only item-2 was classified into different knowledge types due to the difference in a learning situation. However, item-2 and item-12 were classified into different cognitive levels due to the difference in a learning situation, and item-6 was classified into different cognitive levels due to the hints in questions [9, 10, 11].
ITEM 2. According to the researcher's document analysis through the methodology for the selected-response item, due to learning situation item 2013-A-2 and 2013-B-2 belongs to the level of remembering (CL1) factual knowledge (KN1), while item 2013-C-2 and 2013-D-2 belongs to the applying (CL3) procedural knowledge (KN3) (Table 1). Because the ratio of 0.5 mol to 11.2 L and 2.0 mol to 44.8 L is frequently utilized in educational activities [9, p. 103].
Table 1. Item 2
2013-A-2 (KN101-02, CL101-02) 2013-B-2 (KN101-02, CL101-02)
Calculate how many liters of volume 0,5 mol of any gas will occupy under normal conditions. A*. 11,2 l B. 33,6 l C. 44,8 l D. 56,0 l E. 67,2 l Calculate how many liters of volume 2,0 mol of any gas will occupy under normal conditions. A. 11,2 l B. 33,6 l C*. 44,8 l D. 56,0 l E. 67,2 l
2013-C-2 (KN302-02, CL302-02) 2013-D-2 (KN302-02, CL302-02)
Calculate how many liters of volume 2,5 mol of any gas will occupy under normal conditions. A. 11,2 l B. 33,6 l C. 44,8 l D*. 56,0 l E. 67,2 l Calculate how many liters of volume 3,0 mol of any gas will occupy under normal conditions. A. 11,2 l B. 33,6 l C. 44,8 l D. 56,0 l E*. 67,2 l
According to the results of the confirmatory factor analysis, our classification of item 2013-A-2 and 2013-B-2 as a item that tested " remembering (CL1) fact knowledge (KN1)" is confirmed. However, item 2013-C-2 assessed "understanding (CL2) conceptual knowledge (KN2)" while item 2013-D-2 examines the features of "understanding (CL2)" and "applying (CL3)" of "factual (KN1)" and "procedural knowledge (CL3)" (Table 2) [10, p. 25, 11, p. 418].
In fact, it is computed that "2.5 moles of gas has a volume of 56.0 L" is considerably less than "3 moles of gas has a volume of 67.2 L (2013-D)". However, adding one decimal place when multiplying numbers 22.4 by 3.0 can make the calculation more challenging to remember than numbers like " 11.2" and "44.8."
Table 2. The knowledge types (KN) and cognitive level (CL) of the Item 2
Items Knowledge type Cognitive process
By researcher's analysis According to the confirmatory factor analysis By researcher's analysis According to the confirmatory factor analysis
2013-A-02 KN1 KN1 CL1 CL1
2013-B-02 KN1 KN1 CL1 CL1
2013-C-02 KN3 KN2 CL3 CL2
2013-D-02 KN3 KN1, KN3 CL3 CL2, CL3
The middle (MG1) and lower groups (LG2) of test takers perform worse on item 2013-C-2 and 2013-D-2 than on item 2013-A-2 and 2013-B-2 (Table 3).
Table 3. Distribution of the Responses for item 2 (percentage)
Test Group (number of examinees) A B C D E
11,2 л 33,6 л 44,8 л 56,0 л 67,2 л
2013-A-2 Total (2964) *58,9 12,1 21,2 4,6 2,2
UG (800) 90,2 2,4 6,1 0,7 0,1
LG (800) 29,7 22,5 33,9 8,2 4,2
MG (1364) 57,6 11,7 22,6 4,8 2,1
2013-B-2 Total (2951) 16,8 10,2 *65,5 4,8 2,0
UG (797) 4,1 1,3 93,8 0,5 0,4
LG (797) 31,3 18,4 34,5 9,9 4,4
MG (1357) 15,8 10,6 67,0 4,3 1,5
2013-C-2 Total (2939) 17,1 11,4 20,1 *44,5 5,6
UG (794) 2,8 3,3 4,8 87,3 1,5
LG (794) 28,2 17,0 31,0 13,2 8,6
MG (1351) 18,9 12,9 22,7 37,8 6,2
2013-D-2 Total (2950) 12,0 20,2 15,2 4,8 *46,6
UG (797) 1,6 2,6 2,4 1,1 91,8
LG (797) 22,8 32,3 25,5 7,7 10,7
MG (1356) 11,8 23,4 16,7 5,4 41,3
Source: Center for Educational Evaluation, Mongolia *- correct response
1 Lower group, 27% of the examinees at the bottom
2 Middle group, 46% of the examinees at the middle
The MG and LG of test takers performed worse on items 2013-C-2 than on items 2013-A-2, 2013-B-2, and 2013-D-2. In a range of 90.2-93.8% in the upper group (UG3), examinees correctly answered the 2013-A-2, 2013-B-2 (KN1, CL1), and 2013-D-2 (KN3, CL3) items. Somewhat lower (87.3%) examinees of the UG's correctly responded to the 2013-C-2 item. In other words, "0.5 mol of gas is 11.2 L," "2 mol of gas is 44.8 L," and "3 mol of gas is 67.2 L" were factual information for the examinees in the higher group.
ITEM 6. Items 2013-A-6, 2013-B-6, and 2013-C-6 are categorized as belonging to remembering (CL1) factual knowledge (KN1), but item 2013-D-6 is categorized as belonging to comprehending (CL2) factual knowledge (KN1). All are checking the factual knowledge of the color of the two-color indicator phenolphthalein. Due to the hint in the item, only item 2013-D-6 is considered CL2 [9, p. 114]. Because there are two options: "C. Colorless and pink" and "E. Pink and colorless." Examinees must choose the correct answer from two options according to the given order of acid and alkali in the item's stem (Table 4, Table 5).
Table 4. Item 6
2013-A-6 (KN104-06, CL104-06) 2013-B-6 (KN 104-06, CL104-06)
To each solution of hydrochloric acid (HCl) and caustic soda (NaOH) added phenolphthalein, what will be the color of the solution? Choose the correct answer. A. Red and orange B*. Colorless and pink C. Red and blue D. Pink and blue E. Blue and colorless To each solution of caustic potash (KOH) and hydrochloric acid (HCl) added phenolphthalein, what will be the color of the solution? Choose the correct answer. A. Red and orange B*. Pink and colorless C. Blue and colorless D. Red and blue E. Pink and blue
2013-C-6 (KN102-06, CL103-06) 2013-D-6 (KN102-06, CL201-06)
To each solution of sulfuric acid (H2SO4) and caustic soda (NaOH) added phenolphthalein, what will be the color of the solution? Choose the correct answer. A. Red and orange B*. Colorless and pink C. Red and blue D. Pink and blue E. Blue and colorless To each solution of caustic soda (NaOH) and hydrochloric acid (HCl) added phenolphthalein, what will be the color of the solution? Choose the correct answer. A. Red and orange B. Red and blue C. Colorless and pink D. Blue and colorless E*. Pink and colorless
3 Upper group, 27% of the examinees at the top
According to the confirmatory factor analysis results, our classification of items 2013-A-6, 2013-C-6, and 2013-D-6 is confirmed (Table 5). However, item 2013-B-6 was assessed as a "remembering (CL1) and understanding (CL2) factual knowledge (KN1)." The only difference was that KOH was provided; the reason why this happened could not be explained [10, p. 25].
Table 5. The knowledge types (KN) and cognitive level (CL) of the Item 6
Items Knowledge type Cognitive process
By researcher's analysis According to the confirmatory factor analysis By researcher's analysis According to the confirmatory factor analysis
2013-A-06 KN1 KN1 CL1 CL1
2013-B-06 KN1 KN1 CL1 CL1, CL2 (KOH)
2013-C-06 KN1 KN1 CL1 CL1
2013-D-06 KN1 KN1 CL2 CL2
The answers provided by UG, LG, and MG of examinees to items 2013-A-6, 2013-B-6, and 2013-C-6 are similar (Table 6).
Table 6. Distribution of the Responses for item 6 (percentage)
Test Group (number of examinees) A B C D E
2013-A-6 Total (2964) 18,7 *40,4 12,8 14,0 13,0
UG (800) 11,6 61,6 9,5 9,5 7,0
LG (800) 25,4 23,7 15,0 15,7 18,5
MG (1364) 18,9 37,8 13,5 15,6 13,3
2013-B-6 Total (2951) 21,3 *35,6 16,5 9,3 16,3
UG (797) 11,0 60,2 7,9 6,3 13,7
LG (797) 26,9 19,6 24,1 11,7 16,7
MG (1357) 24,0 30,6 17,2 9,6 17,7
2013-C-6 Total (2939) 21,4 *36,1 13,6 15,6 12,1
UG (794) 11,7 56,6 10,2 12,0 8,4
LG (794) 31,4 21,3 14,2 16,9 14,9
MG (1351) 21,2 32,8 15,2 16,9 12,6
2013-D-6 Total (2950) 19,7 16,6 20,5 13,7 *27,6
UG (797) 9,9 11,4 17,5 8,3 51,9
LG (797) 26,0 19,5 23,5 17,6 11,8
MG (1356) 21,7 18,0 20,5 14,7 22,7
Source: Center for Educational Evaluation, Mongolia *- correct response
However, in the 2013-D-6 task, relatively few examinees from UG, LG, and MG answered correctly (Table 6). Examinees must answer that phenolphthalein will be seen as "E. Pink and colorless" in the question's alkali and acid order. The performance of the test takers (Table 6) supports our definition of this item as an "understanding factual knowledge" because it evaluates the examiner's understanding of the distinction between options C and E.
ITEM 12. Items 2013-A-12 and 2013-B-12 are at the understanding (CL2) conceptual knowledge (KN2), while items 2013-C-12 and 2013-D-12 are at the remembering (CL1) conceptual knowledge (KN2) are defined respectively (Table 7). According to the researchers, the only given ionic compounds separate these four items from one another. Tasks 2013-C-12 and 2013-D-12 are the "remembering" items because NaCl and AlCl3 are classic examples of ionic compounds in textbooks. However, items 2013-A-12 and 2013-B-12 are the "understanding" items because ZnO and CuO are not even mentioned in textbooks and curricula [9, p. 116].
Table 7. Item 12
According to the confirmatory factor analysis results, the knowledge type classification of four items is confirmed (Table 8). All of the cognitive process classifications are rejected, except item 2013-D-12. According to the analysis results, tasks 2013-A-12, 2013-B-12, and 2013-D-12 require "remembering," while task 2013-C-12 requires "understanding" [10, p. 25].
Table 8. The knowledge types (KN) and cognitive level (CL) of the Item 12
Items Knowledge type Cognitive process
By researcher's analysis According to the confirmatory factor analysis By researcher's analysis According to the confirmatory factor analysis
2013-A-12 KN2 KN2 CL2 CL1
2013-B-12 KN2 KN2 CL2 CL1
2013-C-12 KN2 KN2 CL1 CL2
2013-D-12 KN2 KN2 CL1 CL1
ZnO and CuO are no longer affected because the formulation (Table 7) of the options allows the examinees to reach the correct answer to items 2013-A-12 and 2013-B-12 in a 2-step process (identify the chemical bond in CO, Zn, HCl, and Cu). The
examinees' performance on items 2013-D-12 differed from that on items 2013-A-12 and 2013-B-12 (Table 9) in a way that this study could not explain.
Table 9. Distribution of the Responses for item 12 (percentage)
Test Group (number of examinees) A B C D E
2013-A-12 Total (2964) 14,5 12,4 11,3 12,5 *48,7
UG (800) 3,4 2,9 2,5 6,0 85,1
LG (800) 23,2 20,5 17,9 16,9 20,1
MG (1364) 15,9 13,3 12,6 13,7 44,1
2013-B-12 Total (2951) 9,0 *51,3 9,9 17,2 11,9
UG (797) 2,4 80,1 1,8 11,5 3,9
LG (797) 15,4 26,0 16,9 22,2 17,9
MG (1357) 9,1 49,3 10,6 17,5 13,1
2013-C-12 Total (2939) 9,4 *45,9 17,6 13,2 13,3
UG (794) 2,1 75,1 11,2 4,5 6,6
LG (794) 17,6 24,4 20,8 19,7 16,8
MG (1351) 8,9 41,3 19,5 14,6 15,2
2013-D-12 Total (2950) *31,7 9,6 12,2 22,0 23,9
UG (797) 55,9 2,4 6,3 13,9 21,3
LG (797) 17,3 16,3 18,8 24,7 22,0
MG (1356) 25,9 9,9 11,8 25,1 26,5
Source: Center for Educational Evaluation, Mongolia *-correct response
Item 2013-C-12, which starts identifying the chemical bond type in NH3, is a "understanding" item, on the other hand. NH3 is a well-known example of covalent bond's donor-acceptor mechanism. The above results can be explained by the common misconception that covalent bond's donor-acceptor mechanism is considered an independent "bond type".
This study was limited to 124 items in the four parallel tests, and comments were made on twelve items. However, it will be crucial to develop an optimal selected-response items for the assessment of chemistry as well as other subjects.
Conclusion
The examination of the 12 tasks covered by the study demonstrated that "familiarity," "classic and frequently used examples," and "memorized information and knowledge" have a significant impact on the knowledge type and the cognitive process level of the item. According to the study result, "the learning situation" and "item's hints" are indispensable factors in determining the knowledge type and cognitive level.
It's important for educators to align the assessment tasks with the cognitive process levels and the knowledge types (intended learning outcomes) they want to assess. By considering the learning situation and its impact on the types of knowledge students acquire, educators can design assessment tasks that effectively measure students' understanding and skills in a particular subject area. The item's hint in a multiple-choice item can influence the knowledge type and cognitive process required to answer the question. Depending on the nature of the hint, it can direct the student towards recalling factual knowledge, applying conceptual understanding, or utilizing procedural knowledge. The hint should support and enhance the assessment task without overly simplifying or compromising the intended cognitive demand. Careful consideration should be given to ensure that the hint appropriately guides students while still requiring them to engage in meaningful thinking and problem-solving.
REFERENCES:
1. Domin D. S. A Content analysis of general chemistry laboratory manuals for evidence of higher-order cognitive tasks // J Chem Educ. 1999. 76(1): 109-111.
2. Azar A. Analysis of Turkish high-school physics-examination questions and university entrance exams questions according to Bloom's taxonomy // J. Turkish Sci. Educ. 2005. 2(2): 144-150.
3. Liman M. A. and Isma'il Y. A structural equation model examining the relationship among Remembering, Understanding, Applying and Students' Achievement in Mathematics // J Educ. Stud. 2015. 3(1): 70-91.
4. Curkovic N., Using of structural equation modeling techniques in cognitive levels validation // Interdisciplinary Description of Complex Systems. 2012. 10(3): 270-283.
5. Kim M.K., Patel R. A., Uchizono J. A. and Beck L. Incorporation of Bloom's taxonomy into Multiple-Choice examination questions for a Pharmacotherapeutics Course // Am. J. Pharm. Educ. 2012. 76(6): 114-124.
6. Rodrigues S., Taylor N., Cameron M., Syme-Smith L. and Fortuna C. Questioning Chemistry: The role of level, familiarity, language and taxonomy // Science Education International. 2010. 21(1): 31-46.
7. Anderson L. W., Krathwohl D. R., Airasian P. W. and Cruikshank K. A. A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives. New York: Longman, 2001.
8. Bloom B. S., Engelhart M. D., Furst E. J., Hill W. H. and Krathwohl D. R. Taxonomy of Educational Objectives: The Classification of Educational Goals. Michigan: David McKay, 1956.
9. Nyamgerel C. The Methodology of Analysis of Selected-response Items (dissertation) [in Mongolian]. Ulaanbaatar, 2016.
10. Nyamgerel C. and Tumenbayar D. Analysis of the compatibility of the level of cognitive process of the items (on the case of the chemistry test of the GEE) [in Mongolian] // Issues and solutions in Educational metrology conference. 2016. Ulaanbaatar. 21-27.
11. Nyamgerel C., Tumenbayar D., Nergui N. and Oyuntsetseg N. Knowledge type of Chemistry test items of General Entrance Exam-2013 // Educational Evaluation International conference. 2016. Ulaanbaatar, 417-424
