CC BY
33
Perspectives of Science & Education
International Scientific Electronic Journal ISSN 2307-2334 (Online)
Available: https://pnojournal.wordpress.com/2022-2/22-05/ Accepted: 16 June 2022 Published: 31 October 2022
X. He, C. K. S. Singh, N. A. Ebrahim
Quantitative and qualitative analysis of higher-order thinking skills in blended learning
The problem and the aim of the study. Higher-order thinking skills is a very popular concept in education field around the world. Among the broad range of higher-order thinking skills, recent studies are concentrating on fostering higher-order thinking skills in education. The number of related publications demands a bibliometric study for finding the keywords, sources, countries, and research clusters. And the study carried out a bibliometric analysis of the relevant publications on higher-order thinking skills from Scopus database between 2011 and 2020.
Research methods. A visual system of top keywords in higher-order thinking skills research has been produced by analyzing of keywords co-occurrence. And by using bibliography inquiry, the most common topic of blended learning has been found. This study also investigates the recent publication trend of higher-order thinking skills in blended learning.
Results. The annual number of publications demonstrates an uptrend in the area of higher-order thinking skills, and the publication rates have increased distinctively in 2018 and 2019. Indonesia and the United States have already become the two major research centers while Malaysia and Australia are joining the research circle in the near future. Related publications had been related to the field of education and metacognition from the very beginning in 2016, and then to the field of higher education, critical thinking and assessment, followed by active thinking and learning in 2017, and eventually to blended learning and flipped classroom in 2018. The systematic structure shows that the United States and Korea have more papers from joint efforts than other countries. In particular, Korea became the most collaborative country. The Journal of Physics: Conference series has signified an increasing occurrence in publications on Higherorder thinking skills in blended learning with "student" acting as the most influential word. Fostering higher-order thinking skills cultivation in flipped classroom and scaffolding higher-order thinking skills activities in learning approaches are both crucial areas involved.
In conclusion, the quantitative analysis on higher-order thinking skills through bibliometric study has revealed the recent subfield of blende learning. Thus, the dataset is further refined by blended learning, considering publication year, sources, countries and organizations, keywords and topics. The annual output of publications on higher-order thinking in blended learning have risen in the past ten years, showing it is more likely to keep developing in the near future.
Keywords: higher-order thinking skills, blended learning, bibliometric analysis, education
For Reference:
He, X., Singh, C. K. S., & Ebrahim, N. A. (2022). Quantitative and qualitative analysis of higher-order thinking skills in blended learning. Perspektivy nauki i obrazovania - Perspectives of Science and Education, 59 (5), 397-414. doi: 10.32744/pse.2022.5.23
_Introduction
\ \ / /ith human society entering the intelligent industry 4.0 era, everything is \ \ / / connected through the Internet. Information technology has tremendously \A/ changed the requirements of many industries for workers' knowledge, ability, and accomplishment; thus, simply relying on memory to recite declarative and procedural knowledge is far from adapting to the new development [1]. Higher-order thinking is the embodiment of the advanced comprehensive ability to think from concrete to abstract thinking, referring to the cognitive ability at a higher cognitive level. In Bloom's taxonomy, analyzing, evaluating, and synthesizing are all considered as higher-order thinking skills that need distinctive learning strategies instead of those just involving memorizing facts and concepts [2]. The framework of Bloom Taxonomy was re-examined in 2001. Although this new version still involves six categories, it differs in the higher level of cognition which consists of analyzing, evaluating, and creating [3]. Higher-order thinking skills are usually regarded as thinking critically, logically, reflectively, and creatively at a much higher level [4]. Generally, higher-order thinking skills are defined as transferring the knowledge and corresponding skills into newly different situations, thinking critically for making decisions, and solving problems which were set both by the teachers and students [5]. According to G. Huang, C. Lai, J. Liang, H. Chu, C. Tsai, higher-thinking ability consists of problem-solving, critical thinking, and creativity [6]. A. Lewis, D. Smith think higher-order thinking skills occurs when a person absorb new information and rearranges or extends it to the problem solution in complicated situations [7]. Higher-order thinking is defined as an encompassing term, which is widely used in education field and play an important role in the development of universities. As O.J. Alkhatib states, whether a university is successful or not depends on their effectiveness in preparing graduates with sufficient higher-order thinking processes, in which the improvement of problem-solving, critical thinking, and creativity, innovation, and decision-making are all involved [8]. Also, B.B. Yazar Soyadi points out that the use of creative and critical thinking to approach problems has become one of today's requirements [9]. With a little bit of difference, most researches all mention three most important parts in higher-order thinking skills, namely, critical thinking skills, problem-solving strategy, and creativity. Higher-order thinking skills are highly valued, just as it was mentioned by the World Economic Forum in 2016 that critical thinking and complex problem-solving will sure be the most demanded job qualifications [10]. Wu Yan (2018), the Director of Higher Education Department of Ministry of Education, put forward the "golden lesson" standard in the National University Teaching Forum, in which higher-order is interpreted as a fusion of knowledge, ability, and quality with the aim of cultivating students' advanced thinking ability and comprehensive ability to solve complicated problems [11]. Unlike the reviews stated above, bibliometric analysis has been conducted in this study to demonstrate the visualization of the trends of the existing literature.
In contrast to the traditional narrative literature reviews, bibliometric analysis has developed a unique sort of reviewing the theoretical literature in many fields and become complementary to help the researchers [12]. The continuous new emerging technologies
and bibliometric software packages invention have led to bibliometric visualization of journals [13]. A bibliometric study is useful in examining the annual research productivity, most frequent keywords, top-cited publications, and the trend in publications to learn about particular research filed quantitatively [14]. A qualitative analysis plus a bibliometric study help to gain deeper insight into scholarly outputs as quantitative and qualitative approaches are both used together to visualize the information and relationship among the academic papers by a systematic analysis [15]. Therefore, it is necessary to select the top-cited papers for qualitative analysis.
As a significant approach, citation analysis is of great significance in gauging how influential a research article is [16]. Citations are from the references in the literature, so the use of them as seen as research performance indicators can reflect their influence and quality [17]. Hence, this study explores the research status of higher-order thinking skills in blended learning through both the quantitative and the qualitative approaches. This study will shed light on the research that focuses on higher-order thinking skills.
Materials and methods
Web of Science (WoS) and Scopus are two major databases widely used to collect scholarly publications data [18]. As for those researchers who carry out bibliometric analysis, the two databases have raised the relevant issues of the comparability and stability of statistics [19]. WoS is regarded as one of the most outstanding databases to conduct literature retrieval and analysis. However, the range of journals in the Scopus seems to be more comprehensive with enriched data than WoS [20]. To select the appropriate database, a topic search of (''higher-order thinking skills" OR ''Higher-order thinking'') was conducted on both the WoS database and Scopus database. Compared with 1,391 publications retrieved from WoS, 2,429 publications were retrieved from the Scopus database. All of the publications contain the phrases of "higher-order thinking skills" or "higher-order thinking" in the title, abstract, or key-words. And the publications were retrieved from the Scopus database on 23rd of July 2021. To ensure the most recent documents are selected, publications since the year 2011 have been considered. Thus, the number of retrieved documents has decreased from2,429 publications to 1,911. In 2021, as the documents are not included, the final data consists of 1,687 documents. They are the documents that were published about higherorder thinking skills during 2011-2020. VOSViewer software [21] is adopted to demonstrate the visualization of the annual number of publications, top productive countries and top keywords. VOSViewer is a freely available software for constructing and viewing the graphical representation of bibliometric mapping for easy interpretation [22]. The total number of 2,429 documents was limited to 184, which are directly relevant to higher-order thinking skills in blended learning.
The publications available in the Scopus database include higher-order thinking skills or higher-order thinking on their topic (title, abstract or key-words). During the data collection process, some limitations cannot be avoided. Although the final data is refined to "blended learning" or "hybrid learning", other authors may use possible synonyms of these phrases as a replacement. Therefore, this study will probably cover all the concerned pieces of literature
from the Scopus database. Generally, all the publications in the bibliometric analysis are fully representing the major research results in higher-order thinking skills in blended learning. The data is analyzed by Bibliometrix-R-package, open software for analyzing and mapping bibliographic data in quantitative research [23]. The data collection procedure is shown in Figure 1, which demonstrates some key elements for systematic reviews as well as a Prisma flow diagram. They are two important ways to sum up the scientific literature with priori-specified criteria to answer a particular research question [24]. The quantitative results from the bibliometric analysis of the publications are further explained by a qualitative analysis. By categorizing these 184 publications into two main subjects, this study finally chooses the highly cited publications for a qualitative literature review.
Figure 1 The Prisma flow graph on higher-order thinking skills in blended learning
_Literature review
Higher-order thinking skills (HOTs) is a very popular concept in the world, which was firstly developed by Benjamin Bloom in the year of 1956, who proposed three domains to gauge the learning accomplishments of learners: the cognitive, the affective and the psychomotor. A.L. Weay, M. Masood, S.H. Abdullah mention that educational goals are divided by Bloom into the psychomotor, affective and cognitive domain [25]. The psychomotor domain has a few levels including perception, instructed response, mechanism adaptation and origination, and affective domain consists of receiving, organizing, characterizing, valuing and responding. According to N. Omara, S.S. Harisa, R. Hassana, H. Arshada, M. Rahmata, N.F. Ainun Zainala, R. Zulkifli, six cognitive skills in Bloom's Taxonomy are categorized into another new six sequences of categorists, it has generally been acknowledged by instructors as a rule for developing proper test questions related to different psychological categories [26].
The concept of higher-order thinking skills is very broad as Bloom's Taxonomy is a part of it. Just as R. B. King, D.M. Mclnerney, D.A. Watkins state, logical thinking, critical thinking, reflective thinking as well as creative thinking all belong to higher order thinking skills [27]. S. Broomkhart defines higher-order thinking as the process of transferring the knowledge and skills into new situations, thinking critically for making decisions, and solving problems which were set both by the teachers and students [28].
With a little bit of difference, G. J. Hwang, C. L. Lai, J.C. Liang, H.C. Chu, C.C. Tsai believe there are only three important parts in HOTS, namely problem-solving, creativity and critical thinking [29]. K. Lu, H. H. Harrison, Y, Shi, X. Wang explain elements of higher-order thinking in more details [30], that is, problem-solving is the ability to gather and analyze data, creativity is the ability to make new products and develop creative ideas, and critical thinking is the ability to do critical and objective analysis of information with clear and rational thought.
X. Lin, C. Hmelo, C.K. Kinzer, T.J. Secules summarize four design features of process displays, process prompts, process models and reflective social discourse, which provide scaffolds for reflective thinking by integrating video, the internet and telecommunication systems [31]. According to R. Paul and L. Elder, much of people's thinking is not just informed and impartial, but biased, distorted, uninformed and prejudiced. Yet the quality of our life and that of which we produce, make, or build depends largely on the quality of our thought [32]. Therefore, higher-order thinking has become what people talk about when they want to be evaluative, creative and innovative.
Information technology profoundly changes the requirements of various industries for workers' knowledge, ability and accomplishment. A.C. Saputri, Sajidan, Y. Rinanto, N.M. Prasetyanti advocate that people should know how to use technology as a tool to serve themselves or others, and possess competencies such as critical thinking skills, problemsolving skills, communication and collaboration skills, and information and digital literacy skills [33]. J. T. Bruer points out that although being benefited from technology, students must have the reflective thinking when they are facing enormous information for adapting learning flexibly to new situations [34].
As the reform of education and teaching are carried out to meet the demands of fostering talented people, there is increasing number of researches on blended learning to foster higher-order thinking. A. Churches draws Bloom's digital map in which some new elements are required to be added for dealing with the new objectives [35]. S.H. Vavilina mentions that analyze, Evaluate and Create, the three levels of Bloom's taxonomy will probably benefit the most from the embedment of digital literacies [36]. Similarly, Sneed agrees that the updated Bloom's Digital Taxonomy with examples of tools are helpful for creating online learning activities to satisfy students' need [37].
Thus, as the development of new technology has greatly reduced the burden of human learning of lower-order knowledge and skills, learners' high-level thinking has been taken into consideration, which is related to whether individuals can tap into their potential and realize their self-worth. C.D. Lin points out that the construction of developing students' core quality system is urgent for a comprehensive implementation of quality education such as critical thinking and creative thinking [38]. These abilities are viewed as an important universal skill that need to be taught and cultivated.
As options for supporting higher-order thinking activities in the analyzed studies, it is proposed to use bibliometric analysis to grasp the most common topic of the recent publication trend of higher-order thinking skills in blended learning. This method is a tool for searching and processing information for discovering patterns, hotspots, analogues and finding optimal solutions.
_Quantitative Analysis
1,687 publications on the topic of "higher-order thinking skills" are from 2011 to 2020. In Figure 2, the annual number of publications shows an uptrend in the area of higherorder thinking skills. In 2011, the number of publications was just above 50. And then the research productivity of higher-order thinking skills went up slowly over the first five years. Publication rates have increased distinctively in 2018 and 2019, with 200 and 338 papers. As what was shown in Figure 2, more and more scholars' research interest was in this field from 2015, onward and the number of publications has increased steadily.
450 400 350 300 250 200 150 100 so 0
2020 2019 201S 2017 2016 2015 2014 2013 2012 2011
Year
Figure 2 Analysis of publication years in higher-order thinking skills from 2011 to 2020
Figure 3 demonstrates the visualization of the top 10 productive countries and regions which are involved in the research area of higher-order thinking skills. The documents published in the top 9 countries were reported as 82.9% of the total publications. There are 3 western countries and 6 Asian countries in the top list. Indonesia ranks first, collaborating with 414 papers published. The second top country is the United States collaborating with 394 publications, and Malaysia is the third with 203 publications. To be specific, Indonesia has the highest publication in Asia, the United States represents North America, and Australia ranks highest in Oceania. The data above shows the leading country of each continent in the research of high-order thinking skills. It is obviously seen that Indonesia and the United States have already become the two major research centers in this research field. Meanwhile, Malaysia and Australia are joining the research circle of higher-order thinking skills in the near future.
Figure 3 Density Visualization of top 20 countries collaboration in higher-order thinking skills
With the assistance of VOSViewer software, co-occurrence analysis has been used among the author's keywords, keywords plus, and keywords index. In this study, keywords with a minimum number of 20 occurrences were adopted. There were a total number of 3,245 keywords in the bibliometric analysis of higher-order thinking skills, and 18 keywords met the requirement. Figure 4 depicts the systematic structure of 18 authors' keywords of high-frequency. Moreover, keywords in different colors demonstrate the information about the specific time of the related publications, while these lines indicate the co-occurrence connections among the keywords. Higher-order thinking skills-related publications had been used in the field of education and metacognition from the very beginning in 2016. Higher-order thinking skills research started related to the field of higher education, critical thinking and assessment, followed by active thinking and learning in 2017, and eventually to blended learning and flipped classroom in 2018.
After co-occurrence analysis with keywords, the hottest topics of blended learning as well as flipped classrooms colored in yellow are revealed. Therefore, the paper focuses on higher-order thinking skills in blended learning.
fhihti 1
—f Bi^rMtd laming)
[ higher order thinkin j
^ÊtÈÊÊÊÊÊÊmà ¡^^^ ( ffipp.dd^m]
^B» iMM^j^^ ■•■a vL i'.L •:""=,
^ thinknf>. sj
- VOS viewer
h-gfier-order thirokin
mu 20iT.a 2017.5 2Û1S.D
Figure 4 The Systematic structure of top 18 author's keywords through bibliometric
analysis of higher-order thinking skills
Refining the original 1687 publications to blended learning led to 184 publications retrieved from the Scopus database, which can be seen in table 1. 506 active authors explore higher-order thinking skills in blended learning. Statistically, the authors per document is 2.42, suggesting that many documents are mainly the work of collaborative research. In contrast to 41 single-authored documents on higher-order thinking skills in blended learning, 143 documents are the work of joint efforts by 410 different authors. The global collaboration index is estimated as 2.87, which shows there is an uptrend for co-authorship in the research of higher-order thinking skills in blended learning. In Table 1, the average number of citations per document is 7,163, which means there are much more low-cited publications than highly cited publications.
Figure 5 reveals that the number of publications increased in higher-order thinking skills in blended learning. As can be seen in the cumulative graph, the field of higher-order thinking skills in blended learning is reaching 20 publications in 2018. Meanwhile, in conform with the steady rate of research on higher-order thinking skills, the number of research productions of higher-order thinking skills in blended learning has increased slowly. As shown in Figure 5, the blended learning approach for higher-order thinking skills has expanded just after the year of 2018, where the number of documents has gone up from only 7 in 2011 to 40 in 2019. In 2020, there is a decrease due to the impossibility of face-to-face class as the outbreak of the Covid-19 pandemic at the end of 2019.
Figure 6 gives the full image of a number of authors affiliated with different countries. The red curves imply the joint works published by the most productive countries in the field of higher- order thinking skills. As is shown on the map, international cooperation is restricted to several countries, which depicts the need for wider collaboration with more nations worldwide. As the collaboration path demonstrates, the United States and Korea have more papers from joint efforts than other countries.
Table 1
The relevant data for the bibliometric analysis on higher-order thinking skills
in blended learning
Description Results
Timespan 2011-2020
Sources (Journals, Books, etc) 125
Document 184
Average years from publication 3.66
Average citations per documents 7.163
Average citations per year per doc 1.442
References 7307
DOCUMENT TYPES
article 102
book 1
book chapter 14
conference paper 59
conference review 4
editorial 2
review 2
DOCUMENT CONTENTS
Keywords Plus (ID) 662
Author's Keywords (DE) 506
Authors 445
Author Appearances 511
Authors of single-authored documents 35
Authors of multi-authored documents 410
Single-authored documents 41
Documents per Author 0.413
Authors per Document 2.42
Co-Authors per Documents 2.78
Collaboration Index 2.87
2020 2019 2018 2017 2016 2015 2014 2013 2012 2011
Year
Figure 5 A cumulative diagram of publication years in the field of higher-order thinking skills
0>
X)
c
o
Latitude
Figure 6 Country collaboration map
Figure 7 shows the number of multiple country publications as well as single country publications in higher-order thinking skills in blended Learning in the top-cited countries which are chosen in terms of the nationality of the corresponding author. 16 single country publications appear in Indonesia. Chinese and American corresponding authors demonstrate equal research productivity of 12 articles in the area of higher-order thinking skills in blended learning. Interestingly, both of the two countries contain just one multiple country publication. The total articles were written by Malaysian corresponding authors, which include 8 multiple country publication and one single country publications. Korea became the most collaborative country. As Figure 7 shows authors prefer to collaborate with those who are working in the field from the same countries.
Corresponding Author's Country
■u
INDOfleSIA CHINA LF5A MALAYSIA KOREA SJUDi ARABIA AUSTRALIA GREECE INDIA b MEXICO
5 NETHERLANDS (J NEW ZEALAND SPAIN THAILAND TURKEY UNITED KINGDOM COLOMBIA DFNMAftK EGYPT HONG KONG
Col la bo rat
MCP
SCP
10
15
N. of Documents
Figure 7 The number of single country publications and multiple country publications
The Annual occurrences of top 10 sources of higher-order thinking skills in blended learning from 2011 to 2020 are shown in figure 8. The Journal of Physics: Conference series has signified an increasing occurrence in publications on Higher-order thinking skills in blended learning. Many researchers focused on higher-order thinking skills are inclined to publish papers in Journal of Physics" Conference series, where 17 related papers in 2020. Besides, the occurrences of other sources like the Universal Journal of Educational Research have grown fast since 2018, arriving at over eight annual publications in 2020. Although Proceedings of the International Conference on E-learning and Computers and Education started publishing papers related to Higher-order thinking skills in blended Learning in 2013 and 2016, respectively, they are producing fewer articles nowadays, instead.
ts-
VI
<u 0}
8 10"
u
O
m ij
Is-=j
U
o
PI
o
o
in
o
J) Q
o
m o
№
o rv
o
- AlPCQNFERENCEPROCiEOINGi - 6RTT1SH JOURNAL Q^S^fcro+JAL TKHNOiOiV - COMPUTERS AND EDUCATION
-INTERNATIONAL JOURNAL OF MOBILE LEARNING Ar ¡C ORGANISATION -JOURNAL OF PHYS ICS: CONFERE NCE SERIES
- PROCEEDINGS OF THE EUROPEAN CONFEHENCE ON E-LEARNiNG, ECEL -UNIVERSAL JCORNAL OF EDUCATIONAL RESEARCH
Figure 8 Annual occurrences of top-ranked sources of higher-order thinking skills
in blended learning
Table 2 shows the top author's keywords from 2011 to 2020, which have increasing occurrences in recent publications. The list has the keywords that arose at least seven times in all publications. Meanwhile, only 20 have been up to the standard among these authors' keywords in higher-order thinking skills in blended learning. Being the strongest keyword with 76 occurrences, "student" acted as the most influential word to illustrate the elements of the concept. As Table 2 shows, more and more researchers are studying on higher-order thinking skills from the perspective of teaching in the field of education.
Figure 9 demonstrate the keywords index in the Scopus database by clustering them into categories with the scale of multidimension. The visualization of the top keywords index shows the publications related to higher-order thinking skills in blended learning are organized into two main clusters. The blue cluster shows the keywords index of all publications, considering the process of higher-order thinking skills in blended learning. The cluster consists of 18 top index keywords like Education, Student, Teacher, Academic performance, Motivation, Collaborative learning, Human experiment, Learning, and others.
Table 2
The occurrence of top-ranked author's keywords in the bibliometric analysis of higher-order thinking skills in blended learning
Author's Keywords Occurrences
students 76
teaching 36
education 33
Higher-order thinking 28
higher-order thinking skills 28
e-learning 23
engineering education 21
curricula 19
computer aided instruction 18
learning systems 16
education computing 15
learning 14
blended learning 10
teaching and learning 9
thinking 9
adult 8
humans 8
problem solving 8
article 7
human 7
The red cluster comprises the remaining 32 top keywords index, mainly including higher education, problem- solving skills, critical thinking, flipped classroom, online system, and scaffolds. Research documents of this cluster have suggested different blended learning-based approaches. And these articles include many discussions about the online and social elements in which different cognitive thinking skills are highly related.
Conceptual Structure Map - method: MCA
ytoblem.solving.skills , software.design
rlticaT.th inking ..... roble m .solving
inking, SKiílBritical.thinking.
student performance
lassroom^-*' medical.education
Ivation é human.experiipç^Ci
collaborative .learning
, , onl, e. learning. envir<
learning
ie.systems humans ' imént
teacher
student
artig^ procedures medical human
Mt female male
f i
*coI labors
sreducation ^V--"""""" thinking
:i^I,rietworkingvronline.learning models
0 12 2 4
Dim 1 (75.91%)
Figure 9 Conceptual structure map of the top 30 Scopus keyword index in the bibliometric analysis of higher-order thinking skill in blended learning
_Qualitative Analysis
The analysis of keywords cluster assists the researchers by identifying the research trends and hotspot in the area of higher-order thinking skill in blended learning. The clusters demonstrate the important themes and point out the research directions in term of the extracted keywords index. The most productive literature of higher-order thinking skills generally concentrates on thinking in education through the flipped classroom. The conspicuous research clusters from the bibliometric analysis are analyzed in a qualitative approach.
All these documents of higher-order thinking skills in blended learning from the Scopus database are further refined by annual citation rate. Analysis and evaluation of the top-cited papers will be helpful to know the most important topics in the field. After excluding review papers, other publications with the highest citation rate are analyzed in terms of all titles and abstracts. Two major research topics of higher-order thinking skills in the flipped classrooms and learning approaches for higher order thinking skills are carried out during the analysis. The refined research area also in accordance with the revealed research clusters in the previous factorial analysis.
A. Fostering higher-order thinking skills cultivation in flipped classroom
With the increasing advancement of technology, different approaches to teaching and learning have occurred in education. As one form of blended learning, flipped classroom has changed the way traditional teaching and learning is carried out since it is increasingly used to assists student-centered learning.
Students prefer to accept new teaching models. Attending an information and communication technology class on 3D modeling, students are involved in a public secondary school in Hong Kong through the flipped classroom. The teacher provided the students with instructional videos, designed and administrated online quiz with learning materials delivered. By collecting both quantitative and qualitative data through survey, online tests, and focus group interview, the study revealed that flipped classroom approach was helpful to foster students' higher-order thinking ability [39]. Based on personalization, self-direction, and collaborative work and higher-order thinking, a theoretical framework was set up to assist student-centered learning. The flipped classrooms personalized learning through different resources and teacher access, and brought about positive outcomes from students, not only developing their higher-order thinking but also making them engaged in learning activities collaboratively through both peer groups and design groups [40].
The flipped classroom needs to combine with creativity. Flipped classroom approach is adopted as a method to foster students' higher-order thinking skills, such as a question, critical thinking, and creativity. Through flipped teaching practice, better learning outcomes were obtained compared with traditional learning. Students were contented with this teaching approach and even expected it to be an indispensable part of the curriculum [41]. As the students watched the interesting and relevant videos online and finishing the pre-class tasks, they have been motivated, and carried out the learning activities in the
face-to-face lessons with the knowledge they had acquired in class easily. Students could be allowed to spend their own time on their study freely and the teacher could promote interaction, collaborative spirit and higher-order thinking by checking individual learning process and task performance [42].
To apply flipped classroom in the teaching, a smart learning diagnosis system was designed, and a learner-centered environment had been created in a software engineering course. An experimental group was formed in which one group of students in the experimental groups were taught through flipped classroom learning. In contrast, other students in the control groups continued with the traditional face-to-face classroom learning. It was proved that the students were motivated with strong learning interests, and their problem-solving ability had been improved tremendously through flipped classroom [43].
However, students nowadays have their preferences, and they can express their feelings and opinions towards the teaching approaches. Following both traditional and technology-assisted flipped classroom approaches, the first-year psychology students showed different attitudes towards the teachers' efforts of using the specific teaching approach in terms of learning materials and activities. Applying technology to teaching process may have motivated students' interest and satisfaction, but it still needs more to enhance their learning process [44].
B. Scaffolding higher-order thinking skills activities in learning approaches
Scaffolding is used in the classroom to support effective teaching and learning. As teachers apply scaffolding techniques in the class based on students' learning problems, teaching instructions are adjusted accordingly.
An open IBL course of "Integrated Biomedicine" in the Bachelor of Human Biology and Bachelor of Medicine program offered creative solutions to deal with learning problems in the class. By using stimulatory techniques, a workshop was set up to promote the development of creativity among students [45]. A participatory learning method with a traditional but a competitive assessment had been used, in which students have been challenged with the multiple-choice questions from their peers. A web-based system named Peerwise gave support to students in creating of an annotate question repository. And students not only needed to answer these questions, but also did the critical thinking and made a discussion. Therefore, while fostering higher-order thinking skills, the students performed well and rated highly [46].
A Strategic Instruction Model has been used to deal with higher-order thinking and reasoning in the SIM library, the Content Enhancement Routines and a comprehensive reading program, which is proved to have a positive effect on high school students with learning disabilities [47]. Using the DSR approach with three design principles and seven design characteristics from the meta-requirements, an IT tool has been developed to provide students with formative feedback to better their performance in large-scale lectures. Students received more points with the tool in the final exam have more interest in learning, skill development, and understanding towards the content of the lecture [48].
As one of the active learning pedagogies, the inquiry-based learning pedagogical approach was designed and carried out to enhance higher order cognitive skills among
students. Under the guidance of Bloom's taxonomy, a facilitation approach was adopted, in which students' performance was evaluated, and their ability to design new experiments, to apply the practical utility of the module to real-life were all involved [49]. To assist students who had difficulty in dealing with complex instructional materials, a constructivist approach to study the process of the central domain ideas, rules, and connections were constructed. Activities of drawing diagrams or mind maps are adopted in writing classes, guiding students to organize their thoughts as well as relate the concepts to their real-life situations. [50].
Conclusion
The study carries out both quantitative and qualitative analysis regarding publications in higher-order thinking skills in the Scopus database from 2011 to 2020. The quantitative analysis on higher-order thinking skills through bibliometric study has revealed the recent subfield of blende learning. Thus, the dataset is further refined by blended learning, considering publication year, sources, countries and organizations, keywords and topics. The annual output of publications on higher-order thinking in blended learning have risen in the past ten years, showing it is more likely to keep developing in the near future. Through analyzing the international collaboration, researcher easily find the potential research collaborations. In regard to the countries-related publications on higher-order thinking skills in blended learning, Korea is the more productive one in collaborating with other countries between from 2011 to 2020.
Studying the development of the documents on higher-order thinking skills in blended learning research can make sure the highly- targeted sources in the field. By conducting source growth analysis, Journal of Physics has attracted most researchers to publish their papers on higher order thinking skills in blended learning. By counting the occurrence of keywords during these ten-year periods, the hotspot in this research is obvious. Meanwhile, analyzing occurrence of the author's keywords indicates that the publications on higherorder thinking in blended learning are mainly concerned with education, students, thinking, online, collaborative learning, Internet, approach, flipped classroom, creativity, and learning skills. By the factorial analysis of keywords index as well as the analysis of highly cited publications, the similar themes concentrated on the learning approaches for higher-order thinking skill in flipped classroom emerged.
This paper explains how research conducted on higher-order thinking skills in blended learning areas have changed over the past 10 years. The paper plays a role in understanding the fostering higher-order thinking skills in different approaches and indicating further developments in the field. Research on fostering higher order thinking skills is still under the way. It is believed that plenty of researchers will follow the trend of doing higher-order thinking skills research in blended learning with more publications in the near future.
_Funding
This work was supported by Social Science Foundation of Hunan Province (19YBA119)
REFERENCES_
1. Saputri, A. C., Sajidan, Rinanto, Y., Afandi, & Prasetyanti, N. M. (2019). Improving Students' Critical Thinking Skills in Cell-Metabolism Learning Using Stimulating Higher-order thinking skills Model. International Journal of Instruction, 12(1), 327-342. doi: 10.29333/iji.2019.12122a
2. Bloom, B.S. (Ed.). Engelhart, M.D., Furst, E.J., Hill, W.H. & Krathwohl, D.R. (1956). Taxonomy of Educational Objectives, Handbook I: The Cognitive Domain. New York: David McKay Co Inc.
3. Anderson, L.W., Krathwohl, D.R., Airasian, P.W., Cruikshank, K.A., Mayer, R.E., Pintrich, P.R., Raths, J. & Wittrock, M.C. (2001). A Taxonomy for Learning, Teaching, and Assessing: A revision of Bloom's Taxonomy of Educational Objectives. New York, Pearson, Allyn & Bacon.
4. Sukatiman, Akhyar, M., Siswandari, & Roemintoyo. (2020). Enhancing Higher-Order Thinking Skills in Vocational Education through Scaffolding-Problem Based Learning. Open Engineering, 10(1), 612-619. doi: 10.1515/eng-2020-0070
5. Brookhart, S. (2010). How to Assess Higher-order thinking skills in Your Classroom, ASCD.
6. Huang, G., Lai, C., Liang, J., Chu, H. & Tsai, C. (2017). A long-term experiment to investigate the relationship between high school students' perceptions of mobile learning and peer interaction and higher-order thinking tendencies. Educational Technology Research and Development, 66(1), 75-93.
7. Lewis, A & Smith, D. (1993) Defining Higher-order thinking. Theory into Practice, 32 (3), 131-137, doi: 10.1080/00405849309543588
8. Alkhatib, O. J. (2019). A Framework for Implementing Higher-Order Thinking Skills (Problem-Solving, Critical Thinking, Creative Thinking, and Decision-Making) in Engineering & Humanities. 2019 Advances in Science and Engineering Technology International Conferences (ASET), pp. 1-8, doi: 10.1109/ICASET.2019.8714232
9. Yazar Soyadi, B. B. (2015). Creative and Critical Thinking Skills in Problem-based Learning Environments. Journal of Gifted Education and Creativity, 2(2), 71-71. doi: 10.18200/jgedc.2015214253
10. Lo, C. O., & Feng, L.-C. (2020). Teaching higher-order thinking skills to gifted students: A meta-analysis. Gifted Education International, 36(2), 196-217. Available at: https://lkyjw.lnist.edu.cn/fujian/2018/2018122406.pdf
11. Brika, S. K. M., Algamdi, A., Chergui, K., Musa, A. A. & Zouaghi, R. (2021). Quality of Higher Education: A Bibliometric Review Study. Frontiers in Education, 6. doi: 10.3389/feduc.2021.666087
12. Mokhtari, H., Barkhan, S., Haseli, D. & Saberi, M. K. (2020). A bibliometric analysis and visualization of the Journal of Documentation: 1945-2018. Journal of Documentation, 77(1), 69-92. doi: 10.1108/jd-08-2019-0165
13. Ebrahim, S.A., Ashtari, A., Pedram, M.Z. & Ebrahim, N.A. (2019). Publication trends in drug delivery and magnetic nanoparticles. Nanoscale Research Letters, 14, 1, 164.
14. Manzoor, B., Othman, I.& Pomares, J. C. (2021). Digital Technologies in the Architecture, Engineering and Construction (AEC) Industry-A Bibliometric-Qualitative Literature Review of Research Activities. International journal of environmental research and public health, 18(11), 6135. doi: 10.3390/ijerph18116135
15. He, L.,& Fang, H., Wang, X., Wang, Y., Ge, H., Li, C., Chen, C., Wan, Y. & He, H. (2020). The 100 most-cited articles in urological surgery: A bibliometric analysis. International Journal of Surgery, 75, 74-79.
16. Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019). Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories. SAGE Open, 9(1). doi: 10.1177/2158244019829575
17. Chadegani, A.A., Salehi, H., & Yunus, M.M., Farhadi, H., Fooladi, M., Farhadi, M., & Ebrahim, N.A. (2013). A comparison between two main academic literature collections: Web of science and Scopus databases. Asian Social Science, 9 (5), 18-26.
18. Archambault, E., Campbell, D., Gingras, Y., & Lariviere, V. (2009). Comparing bibliometric statistics obtained from the Web of Science and Scopus. Journal of the American Society for Information Science and Technology, 60(7), 13201326. doi: 10.1002/asi.21062
19. Agarwal A. et al. (2016). Bibliometrics: Tracking research impact by selecting the appropriate metrics. Asian Journal of Andrology, 18 (2), 296.
20. van Eck, N.J. & Waltman, L. (2010). Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics, 84 (2), 523-538.
21. van Eck, N. J., & Waltman, L. (2010). Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics, 84(2), 523-538. doi: 10.1007/s11192-009-0146-3
22. Dervi?, H. (2020). Bibliometric Analysis using Bibliometrix an R Package. Journal of Scientometric Research, 8(3), 156-160. doi: 10.5530/jscires.8.3.32
23. Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Moher, D. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, 372, 71. doi: 10.1136/bmj.n71
24. Weay, A.L., Masood, M., & Abdullah, S.H. (2016). Systematic Review of Revised Bloom Taxonomy, SOLO Taxonomy and Webb's Depth of Knowledge (DOK) in Assessing Students' Historical Understanding in Learning History. Malaysian Journal Of Higher Order Thinking Skills In Education, 1-27.
25. Omara, N., Harisa, S.S., Hassana, R., Arshada, H., Rahmata, M., Ainun Zainala, N.F., & Zulkifli, R. (2012). Automated analysis of exam questions according to Bloom's taxonomy. Procedia - Social and Behavioral Sciences, 59, 297-300.
26. King, R. B., McInerney, D. M., & Watkins, D.A. (2012). How you think about your intelligence determines how you feel in school: the role of theories of intelligence on academic emotions. Learn. Individ. Differ., 22, 814-819.
27. Brookhart, S. (2010). How to Assess Higher-Order Thinking Skills in Your Classroom. Alexandria, VA: ASCD.
28. Hwang, G. J., Lai, C. L., Liang, J. C., Chu, H. C., & Tsai, C. C. (2018). A long-term experiment to investigate the relationships between high school students' perceptions of mobile learning and peer interaction and higher-order thinking tendencies. Educational Technology Research and Development, 66(1), 75-93. doi: 10.1007/s11423-017-9540-3
29. Lu, K., Harrison, H. H., Shi, Y., Wang, X. (2021). Examing the key influencing factors on college students' higher order thinking skills in the smart classroom environment. International Journal of Educational Technology in Higher Education,18 (1). doi: 10.1186/s41239-020-00238-7
30. Lin, X., Hmelo, C., Kinzer, C. K., & Secules, T. J. (1999). Designing technology to support reflection. Educational Technology Research and Development, 47, 43-62. doi: 10.1007/BF02299633
31. Paul, R., Elder, L., & Foundation for Critical Thinking. (2003). The miniature guide to critical thinking: Concepts & tools. Dillon Beach, Calif: Foundation for Critical Thinking.
32. Saputri, A. C., Sajidan, Rinanto, Y., Afandi, & Prasetyanti, N. M. (2019). Improving Students' Critical Thinking Skills in Cell-Metabolism Learning Using Stimulating Higher Order Thinking Skills Model. International Journal of Instruction, 12(1), 327-342. doi: 10.29333/iji.2019.12122a
33. Bruer, J.T. (1993). Schools for Thought. Cambridge, MA: MIT Press
34. Churches, A. (2008). Bloom's Taxonomy Blooms digitally. Available at: https://www.techlearning.com/news/ bloom39s-taxonomy-blooms-digitally (accessed 10 September 2021)
35. Vavilina, S. H. (2020). Using Bloom's Revised Taxonomy to Develop Digital Literacies in the ESP Classroom. Information Technologies and Learning Tools, 79(5), 168-183. doi: 10.33407/itlt.v79i5.3355
36. Sneed, O. (2016). Integrating Technology with Bloom's Taxonomy. Available at: https://teachonline.asu.edu/2016/05/ integrating-technology-blooms-taxonomy/ (accessed 10 September 2021)
37. Lin, C. D. (2017). The Research of Core Competencies and Values for Students in China. Studies of Psychology and Behavior, 5(12), 145-154.
38. Lee, K. Y., & Lai, Y. C. (2017). Facilitating higher-order thinking with the flipped classroom model: a student teacher's experience in a Hong Kong secondary school. Research and Practice in Technology Enhanced Learning, 12(1), 8. doi: 10.1186/s41039-017-0048-6
39. Joyce Hwee Ling, K. (2019). Four Pedagogical Dimensions for Understanding Flipped Classroom Practices in Higher Education: A Systematic Review. Educational Sciences: Theory & Practice, 19(4), 14-33. doi: 10.12738/estp.2019.4.002
40. Rodríguez, G., Díez, J., Pérez, N., Baños, J. E., & Carrió, M. (2019). Flipped classroom: Fostering creative skills in undergraduate students of health sciences. Thinking Skills and Creativity, 33. doi: 10.1016/j.tsc.2019.100575
41. Kiat, P.N.& Kwong, Y.T. (2014). The flipped classroom experience. IEEE 27th Conference on Software Engineering Education and Training (CSEE&T), 39-43. doi: 10.1109/CSEET.2014.6816779
42. Lin, Y.-T. (2019). Impacts of a flipped classroom with a smart learning diagnosis system on students' learning performance, perception, and problem-solving ability in a software engineering course. Computers in Human Behavior, 95. doi: 10.1016/j.chb
43. Limniou, M. Schermbrucker, I., & Lyons, M. (2017). Traditional and flipped classroom approaches delivered by two different teachers: the student perspective. Education and Information Technologies, 23(2), 797-817. doi: 10.1007/ s10639-017-9636-8
44. Rodriguez, G., Perez, N., Nunez, G., Banos, J. E., & Carrio, M. (2019). Developing creative and research skills through an open and interprofessional inquiry-based learning course. BMC Medical Education, 19(1), 134. doi: 10.1186/ s12909-019-1563-5
45. Bottomley, S., & Denny, P. (2011). A participatory learning approach to biochemistry using student authored and evaluated multiple-choice questions. Biochemistry and Molecular Biology Education, 39(5), 352-361. doi: 10.1002/ bmb.20526
46. Hock, M. F., Bulgren, J. A., & Brasseur-Hock, I. F. (2017). The Strategic Instruction Model: The Less Addressed Aspects of Effective Instruction for High School Students with Learning Disabilities. Learning Disabilities Research & Practice, 32(3), 166-179. doi: 10.1111/ldrp.12139
47. Eryilmaz, E., Chiu, M. M., Thoms, B., Mary, J., & Kim, R. (2014). Design and evaluation of instructor-based and peer-oriented attention guidance functionalities in an open source anchored discussion system. Computers & Education, 71, 303-321. doi: 10.1016/j.compedu.2013.08.009
48. Madhuri, G. V., Kantamreddi, V. S. S. N., & Prakash Goteti, L. N. S. (2012). Promoting higher-order thinking skills using inquiry-based learning. European Journal of Engineering Education, 37(2), 117-123. doi: 10.1080/03043797.2012.661701
49. Singh, C. K. S., Gopal, R., Tek, O. E., Masa Singh, T. S., Mostafa, N. A., & Ambar Singh R. K. (2020). ESL teachers' strategies to foster higher-order thinking skills to teach writing. Malaysian Journal of Learning and Instruction, 17(2), 195-226.
Information about the authors
Xiangjun He
(Malaysia, Perak) PhD student of the Department of English Language and Literature Universiti Pendidikan Sultan Idris Associate professor of School of Foreign Languages Hunan University of Technology and Business E-mail: 605488050@qq.com ORCID ID: 0000-0001-8622-2162
Charanjit Kaur Swaran Singh
(corresponding author) (Malaysia, Perak) PhD in TESL, Associate Professor of the Department of English Language and Literature Universiti Pendidikan Sultan Idris E-mail: charanjit@fbk.upsi.edu.my ORCID ID: 0000-0003-3630-6215
Nader Ale Ebrahim
(Iran, Tehran; Malaysia, Kuala Lumpur) Adjunct Lecturer, Research and Technology Department Alzahra University Senior Lecturer, Office of the Deputy Vice-Chancellor (Research & Innovation) University of Malaya E-mail: aleebrahim@gmail.com ORCID ID: 0000-0001-7091-4439
