CC BY
18
Original scientific paper UDC:
378.147.091.33-027.22
Received: November 10, 2023. 159.9.072-057.875
Revised: March 12, 2024. d 10.23947/2334-8496-2024-12-1-41-55
Accepted: March 20, 2024.
Check for updates
Metacognitive Strategies for Mathematical Modeling with Engineering Groups of Students: Adaptation and Validation of a
Questionnaire
Noemi Carcamo-Mansilla1 , Maria D. Aravena-Diaz1"
1Faculty of Basic Sciences, Catholic University of Maule, Talca, Chile; Mathematical and Statistical Education Research
Center, Catholic University of Maule, Talca, Chile e-mail: [email protected], [email protected]
Abstract: A sequential exploratory mixed-methods study is implemented to develop an instrument that allows for the evaluation of the metacognitive strategies used by engineering groups of students when solving mathematical modeling problems. The findings of the qualitative study guided by observations and interviews reveal the use of metacognitive strategies of 'planning', 'monitoring and, if necessary, regulation', and 'evaluation'. In this article, we present the final categories of the qualitative analysis and discuss how these data were shaped into a theoretical construct and items of an instrument to measure metacognitive strategies. The psychometric properties of the instrument are analyzed, and it is argued that it has a similar interpretation among males and females, as there are no significant differences in these results. The development of the present study demonstrates how the qualitative method can support the adaptation of an instrument to measure metacognitive strategies, thus contributing to validity and applicability.
Keywords: modeling, metacognitive strategies, engineering education, questionnaire, gender invariance.
Introduction
Several authors suggest that the future of work is uncertain, but this uncertainty does not refer to the lack of employment, but rather to the skills that new jobs will demand (Dirksen, 2019; Weller et al., 2019; World Economic Forum, 2020). The case of engineering is particular, as professionals in this field are expected to be able to handle large volumes of data, work in teams with machines, possess communication skills, solve complex and open-ended problems, among other skills (Richert et al., 2016; Humberto and Rojas, 2017). The skills to solve complex and open-ended problems underlies much of the standards for 21st-century engineering education (Kent and Noss, 2003; International Engineering Alliance, 2014; ABET, 2017; Alpers, 2021). However, the literature has suggested that the deployment of metacognitive skills is required to successfully solve this type of task (Blum, 2011; Borromeo, 2006; MaaR, 2006; Stillman, 2011; Vorholter et al., 2019). Therefore, metacognition influences the way complex and open problems are approached and modeled in engineering (Wengrowicz et al., 2018; Hidayat, Zulnaidi, et al., 2018; Wedelin et al., 2015; Yildirim, 2010).
Modeling tasks concentrate essential features of common engineering professional environments' problems since they are complex, open-ended, and have few structure (Jonassen et al., 2006; Gainsburg, 2013; Lyon and Magana, 2020). When students in engineering training solve this type of tasks, they understand the connection between engineering contexts and mathematics (Aravena et al., 2022; Soon et al., 2011). Therefore, mathematical modeling in engineering is relevant due to its role in the connection between mathematics and industry problems (Li, 2013; Palmer et al., 2013). However, addressing modeling tasks is challenging because they are complex, open-ended, and it is not evident where to commence and where one is expected to arrive (Kaiser and Schwarz, 2010). Usually, students face obstacles in successfully solving modeling problems due to their own beliefs, limited self-regulation, inappropriate attitudes, and expectations (Wedelin et al., 2015). Incorporating this type of tasks in the classroom requires careful consideration of metacognition due to its significant impact.
'Corresponding author: [email protected]
fTS (?) I © 2024 by the authors. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.ora/licenses/bv/4.0/).
Metacognition plays an essential role in solving complex engineering problems as it promotes the understanding of mathematical concepts, encourages organizational behavior, and enhances teamwork. Metacognition enables the evaluation of local and global effects in engineering problem-solving, the organization of resolution strategies, and the effective monitoring of the calculation process (Hegedus, 2001). Furthermore, it has been found that metacognition positively influences the development of mathematical modeling skills among engineering students, fostering deliberate behaviors and a deeper understanding of the theoretical implications of application (Tzohar-Rozen and Kramarski, 2014; Wedelin et al., 2015; Hidayat et al., 2018). Collaboration in the resolution of modeling tasks also stimulates the use of metacognitive strategies in small groups, allowing efficient monitoring of progress, assessment of viability, and critical analysis of innovative ideas and results (Newell et al., 2004; Cardella, 2008; Gainsburg, 2013). Together, these findings highlight the importance of metacognition in the context of mathematical modeling in engineering.
However, at this educational level, there are few instruments for measuring metacognitive strategies in mathematical modeling problems (Yildirim, 2010; Czocher, 2018; Hidayat et al., 2018; Hidayat, Zulnaidi, et al., 2018), especially for measuring metacognitive strategies with engineering groups of students. The study contributes to the research field by addressing the need to construct instruments as didactic tools to measure the metacognitive strategies used by engineering students in mathematical modeling processes. In this article, we present the development of the metacognitive strategies instrument guided by a mixed methods study, which was conducted over an academic semester through observations, interviews, and the application of a questionnaire.
To address the study, two research questions are sequentially answered. The first question was: How do the metacognitive strategies of 'planning,' 'monitoring and, if necessary, regulation,' and 'evaluation' in the domain of mathematical modeling documented in the literature relate to the observed metacognitive strategies in engineering groups of students in a multiple case study? Based on the results of the qualitative research and theoretical aspects, an instrument was adapted and validated to measure metacognitive strategies in mathematical modeling among engineering groups of students. Based on the development of the instrument, the second research question was: What are the psychometric properties of the instrument, such as content validity, reliability, and measurement invariance? To answer this question, a quantitative correlational research design is utilized. The gender variable is used to investigate measurement invariance.
The stereotypes that exist regarding gender differences (Correl, 2004; Cheryan and Plaut, 2010; Cheryan, 2012) could play a significant role in shaping how engineering students employ their metacognitive strategies to solve mathematical modeling problems. Beliefs based on gender stereotypes have shaped the interest and performance in engineering, science, technology, or mathematics careers (Charles and Bradley, 2009; Noel-Levitz, 2014). This has led to, on the one hand, women and other minorities being underrepresented in engineering careers (Wang and Degol, 2017; National Center for Science and Engineering Statistics, 2023), that is to say, there is low entry into such careers. On the other hand, some women who enter engineering do not persist in their trajectory in response to negative feedback associated with performance in introductory mathematics or science courses, cultural stereotypes, limited peer networks, or discrimination (Riegle-Crumb, 2006; Cech et al., 2011; Mann and DiPetre, 2013; Sanabria and Penner, 2017). This situation generates a global-scale problem, as industries and the economic sector of countries require the presence of men, women, and diverse individuals to develop innovative and creative solutions to complex problems, drive the economy, and achieve sustainable development (Woetzel et al., 2015; Smith-Doerr et al., 2017).
Inquiry into gender disparities regarding metacognitive skills in complex problem-solving has been the subject of study for a long period, revealing diverse results concerning the gender effect on self-efficacy, self-regulation, and metacognitive skills (Ciascai and Lavinia, 2011; Hong et al., 2021). It has been demonstrated that women tend to use self-regulation strategies more effectively, adjusting their efforts according to learning tasks and showing superior performance compared to men when solving complex problems (Bidjerano, 2005). However, there are studies suggesting the absence of gender differences in this field (Fitzpatrick, 1994; Bembenutty, 2007; Aravena-Diaz et al., in press), highlighting the lack of clarity in the relationship between gender and the control of one's own learning processes, that is, metacognition.
Metacognitive strategies in mathematical modeling
A first approach to the analysis of metacognitive strategies in contextualized problems in real-world environments is carried out by Stillman and Galbraith (1998), who take into account the cognitive-metacognitive framework of Garofalo and Lester (1985), to categorize the knowledge of the metacognitive strategy for understanding the problem, organizing information, developing and executing plans, monitoring progress, and verifying the final result. Later, Stillman (2011) highlights the importance of metacognitive activity in the development of mathematical modeling processes, considering the presence of metacognition in each transition between the stages of the modeling cycle. Furthermore, the work of Schukajlow and Krug (2013) reveals the positive influence of metacognitive planning and monitoring strategies on modeling problems that require assuming missing data. Continuing the line of research, Vorhölter (2018, 2019) implemented a project aimed at stimulating ninth and tenth-grade students in this domain, from which he developed an analytical framework for analyzing metacognition in the context of mathematical modeling.
According to Vorhölter et al. (2019), declarative metacognition involves understanding the characteristics of individuals, strategies, and tasks related to modeling. In contrast, procedural metacognitive strategies focus on 'planning', 'monitoring' and, if necessary, regulation', as well as the 'evaluation' of actions and performance in the process of solving mathematical modeling problems. Metacognitive strategies play an essential role in the successful development of modeling tasks. The metacognitive strategies for 'planning' allow for an understanding of the task's objectives, the prediction of outcomes, and the exploration of various resolution approaches. Decision-making regarding which approach to employ for resolution is also considered part of metacognition. "Monitoring" involves the constant review of one's own cognitive processes during the task, posing questions to verify the accuracy of the steps being followed. 'Regulation' is activated when students encounter obstacles and seek support. 'Evaluation' analyzes the processes undertaken and the team's performance, considering the strengths, weaknesses, and team dynamics. These classifications in dimensions of metacognitive strategies are applicable to both individual and group metacognition (Vorhölter, 2019; Vorhölter and Krüger, 2021).
Materials and Methods
The design of this research is of a mixed, sequential exploratory type (Creswell and Plano, 2018), in which the findings from four months of observation and interviews were integrated into an instrument to measure metacognitive strategies (Tashakkori and Teddlie, 2003). This research method allowed for a profound understanding of student's metacognitive strategies and assisted in formulating items for the instrument. This design is developed in two phases: the first phase identifies and classifies the metacognitive strategies of the engineering groups of students. In the second phase, the instrument is adapted and validated based on the categories established in the first phase and a review of the existing literature. The development of the instrument allowed for the application of the study to a larger and more diverse sample, thus extending the scope of the results. For both phases, the analytical framework of specific metacognitive strategies in the mathematical modeling process (Vorhölter et al., 2019) was employed, along with studies that reveal metacognitive aspects of engineering students.
Qualitative component
In the qualitative phase, research techniques guided by multiple case studies were used, including non-participant observations and interviews (Yin, 2014). The observations were carried out while the students were participating in a mathematical modeling program in a one-variable calculus course for engineering. In this initial research phase, a qualitative approach was adopted with the purpose of exploring the processes of mathematical modeling in engineering students to deeply understand the metacognitive strategies that may contribute to the development of this type of problems (Creswell, 2009).
Participants
In the phase that adopts a qualitative design, four engineering student cases with experience in mathematical modeling problem solving are selected to observe the metacognitive behavior of students
encompassing greater diversity (Simons, 2013). The cases consist of four groups, each composed of three students who had participated in a mathematical modeling program within the context of an engineering calculus course, collaborating on modeling problems. For a detailed description, see Cárcamo Mansilla et al., (in press). To distinguish the groups in description of Table 1, we will use the letter G1, G2, G3 and G4 and the students with the letters s1, s2, s3 respectively.
Data collection
Non-participant observations were made of four groups of students as they worked on a sequence of mathematical modeling problems in the classroom. The sessions were recorded using twelve video recordings, and interviews were conducted with the groups after solving the modeling tasks to record retrospective knowledge of the metacognitive strategies they had used in their own process of mathematical modeling. The interviews were organized around metacognitive strategies for 'planning,' 'monitoring and, if necessary, regulation,' and 'evaluation,' based on the work of Vorholter et al. (2019). The interview script was validated by experts according to the criteria of Escobar-Pérez and Cuervo-Martinez (2008).
Data analysis
A deductive-inductive content analysis (Mayring, 2014; Glaser-Zikuda et al., 2020) is used to categorize metacognitive strategies. For this purpose, a categorization was carried out considering codes based on the literature, taking into account the metacognitive strategies of 'planning', 'monitoring and if necessary regulation', and 'evaluation' (ME1 to ME3) based on the theoretical framework of Vorholter et al. (2019). Besides the literature-based codes, the research team allowed for the emergence of additional phenomena (Kuckartz, 2019; Kohlbacher, 2006) related to the metacognitive strategies of engineering students, such as the necessity to make assumptions, as it prompted engineering students to comprehensively address the context of the problem. Additionally, the organization of groups, the definition of roles, and the allocation of time fostered autonomy. Such skills are crucial for future engineers because they will face new challenges ahead (Newell et al., 2004; Penagos, 2011).
Qualitative results
A set of topics organized into categories and subcategories emerged from the observations that were contrasted with the interviews conducted, and are summarized in Table 1. These topics were used to form the final constructs of the instrument to measure metacognitive strategies, which are discussed later.
Table 1. Qualitative categories and subcategories
Category/ subcategories
Planning metacognitive strategies (ME1)
• They discuss the information among all [p1]
• Establish demands and objectives among everyone [p2]
• Discuss solution strategies [p3]
• Realize the need to establish assumptions [p4]
• Consider the resources available to solve the task [p5]
Monitoring and regulation metacognitive strategies (ME2)
• Monitor the progress of the modeling process [m1]
• Validates the modeling process [m2]
• Question the evolving mathematical model [m3]
• Discuss successes and difficulties in the process [r1]
• Establish a course of action at the local level [r2]
• Discuss action plans at a global level [r3]
• Select a new solution plan [r4]
Evaluation metacognitive strategies (ME3)
• Evaluate the effectiveness of resources [e1]
• Evaluate the effectiveness of the process [e2]
• Evaluate the effectiveness of the strategies [e3]
Description
It reflects the active participation of students in the discussion and the joint establishment of demands, objectives, and solution strategies, as well as the awareness of the importance of making assumptions and considering the available resources to effectively address the tasks.
It refers to the constant awareness and reflection on progress and difficulties in the modeling process, where students demonstrate their ability to monitor, question, and adjust evolving mathematical models.
They demonstrate their skills to analyze the effectiveness of available resources, as well as to critically assess the modeling process itself, considering approaches, methods, successes, and challenges. Furthermore, the evaluation of the strategies employed highlights their skill in identifying those that prove effective and applicable in future contexts._
Several of these categories of metacognitive strategies have been previously studied, specifically in the contexts of high school students (Schukajlow and Krug, 2013; Vorhölter, 2017, 2018; Vorhölter, 2019), but have not been explored with engineering groups of students.
Development of the instrument to measure metacognitive strategies
The qualitative data analysis phase reported categories and subcategories of metacognitive strategies, serving as constructs for the instrument to measure metacognitive strategies in mathematical modeling among groups of engineering students. These constructs were operationalized using the dimensions from the analytical framework of Vorhölter et al. (2019) and the transcriptions of video episodes and retrospective interviews to create items that formed part of the assessment tool. For instance, if a phrase appeared in multiple video episodes and interviews, it was used to create an item addressing this theme in the assessment instrument and added to the corresponding dimension. Table 2 presents examples of qualitative data transformed into assessment instrument items.
Table 2. Item creation from qualitative data
Category/ subcategories Survey Item
Qualitative Data
Video episodes transcription:
"What I want is for the four of us to do it to see if we are doing it equally and well" (G4_S1)
"Let's all four do the data for the cylinder and if they give us the same it's because the cylinder is good" (G2_S2) Interview transcription: "We try to see everything together and then compare and follow the same path" (G4_S2)
Planning metacognitive strategies (ME1)
• Discuss solution strategies [p3]
i 14:... We agreed on the same approach to solving the problem individually, to then compare and review results or calculations together
Video episodes transcription: "Wait, let's check again" (G3_S1)
"With these numbers, what we are getting is the capacity that we are going to have inside, do you understand me?" (G1_S3) Interview transcription:
"I thank my colleagues because sometimes I miss a sign." (G4_ S1)
Video episodes transcription: "Yes, if you look at the photo I sent, ehm, it actually has an interval of thirteen points, it has an interval of thirteen point sixty-five" (G4_S3)
"Why do you give us 10 bottles at the base, it seems that it is bad?" (G3_S1)
"Did you see the measurements? It's bad, we have to check it" (G1_S2)
Interview transcription:
"Then we saw that it should not be easy but it is important to try to apply mathematics to something that one recognizes" (G4_S1)
"At one point, when we calculated the number of bottles it didn't fit us and we realized it was wrong and we had to calculate everything again" (G3 S3)_
Monitoring and, if necessary, regulation metacognitive strategies
(ME2)
• Monitor the progress of the modeling process [m1]
i 10:... The groups ensures that everyone understand individually in order to proceed with the resolution of the issue
Monitoring and, if necessary, regulation metacognitive strategies
(ME2)
• Select a new solution plan [r3]
i2: ... In some occasions, we notice that it is necessary to change the work plan (due to confusion or errors)
Additionally, theoretical items were incorporated into each dimension of the instrument. The theoretical references in specialized literature served as a guide. The scale developed by Rakoczy et al. (2005) was used and translated to adapt the instrument, with reported reliability exceeding a Cronbach's alpha of .70 in each of its dimensions. The Schukajlow and Krug (2013) scale is also employed, who reported reliability levels of .74 and .84 for the planning and monitoring dimensions, respectively. Finally, the items are supplemented with the findings of those who have measured metacognitive strategies in the specific domain of mathematical modeling (Vorholter, 2017, 2018, 2019).
Before administering the instrument, an expert validity test was conducted. Specialists in mathematics education and cognitive psychology evaluated the instrument by reviewing the used scale and items obtaining a score of .9 and guaranteeing content validity (Tristan-Lopez, 2008). Then, a pilot study is conducted to carry out a preliminary analysis of the instrument with a sample of 30 engineering students (Johanson and Brooks, 2010) who had experience in modeling processes. The overall reliability of the scale is analyzed through the Cronbach's alpha coefficient, which reaches a value of 0.83, surpassing the minimum acceptable internal consistency criterion (Allen et al., 2008).
Then, an instrument with 23 items and three subconstructs is obtained (Appendix A): metacognitive strategies for 'planning' (8 items), 'monitoring and, if necessary, regulating' (12 items) and 'evaluating' (3 items). To measure the metacognitive strategies with the instrument, a 5-point Likert scale is used (1 = totally disagree, 5 = totally agree).
Quantitative component
After the creation of the instrument, the quantitative phase of the cross-sectional study is conducted. This phase should be treated as a follow-up to the initial qualitative phase, as suggested by the sequential exploratory research design (Creswell and Plano, 2018). This phase has been designed to analyze the psychometric properties of the metacognitive strategies assessment instrument and its invariance across male and female.
Participants
According to the guidelines outlined by Creswell and Plano (2018) for mixed methods, this phase involved different participants from the same student population. Engineering students with experiences in mathematical modeling problem solving were selected. To recognize such expertise, it is considered that they have participated in a modeling program or courses such as calculus, linear algebra, structural analysis, differential equations, among others, where students confront problems in the field of engineering, applying knowledge of mathematics and engineering, with critical thinking and analytical capacity. A total of 104 students (random selection) participated, who had experience in mathematical modeling problem solving acquired through various courses, including 71 men and 33 women. The age ranged from 18 to 47 years (Sd = 11.89). They belonged to various engineering fields. The statistical power of the sample was calculated to be .80, considering a confidence level of .05; a root mean square error of approximation (RMSEA) index approximation of 0.080, and 227 degrees of freedom (Preacher and Coffman, 2006).
Data collection
The instrument was administered in 2022, after obtaining approval from the authorities of the institutions involved in the study and informed consent from the participants. The students took approximately 20 minutes to complete the instrument. Once the data were collected through the application of the instrument, they were transcribed in Excel format, and a spreadsheet was prepared for further analysis in the statistical programs M-Plus (Muthen and Muthen, 2007) and R Studio, respectively.
Data analysis
A descriptive analysis of the items was carried out, considering the mean, standard deviation, and skewness in order to assess the performance of each questionnaire item, ensuring that none of them exhibit an extreme mean or an excessively low standard deviation. Confirmatory factor analysis (CFA) procedures were used to assess the construct validity of the instrument (Ferrando & Anguiano-Carrasco, 2010). The internal consistency values were measured using ordinal data alpha (Oliden and Zumbo,
2008) to estimate the reliability of the scale's dimensions. To identify reliability by factor, the criterion of Ferrando and Anguiano-Carrasco (2010) is followed, that is, if there is a factor loading less than 0.3 in magnitude, the item's inclusion is evaluated.
The measurement invariance between male and female students was also evaluated. Due to the lack of normality assumption in the variables, confirmed by the Kolmogorov-Smirnov test (p < 0.05), the non-parametric method of the Mann-Whitney U test was used to compare two independent groups (Flores-Ruiz, 2017).
Results
Quantitative results
The mean scores, variance, and skewness were obtained for each of the items. As shown in Table 3, participants exhibit a relatively high level of metacognitive strategies in the specific domain of modeling, with an average of 4.22 on a scale of 1 to 5. The items showed a slight negative skewness within acceptable ranges (Ferrando and Anguiano-Carrasco (2010)) for the development of a CFA (Confirmatory Factor Analysis). The skewness indicates that the responses for each item are clustered around the mean, consistent with the high level of reported metacognitive strategies. The standard deviations demonstrate that all items were able to discriminate among subjects.
The information presented in Table 3 does not justify the elimination of any item due to low discrimination capacity, a high grouping of responses in very low or very high scores, or asymmetries outside acceptable ranges. This is confirmed by the high significant correlations among the items and the corrected total score, which ranged from 0.30 to 0.70, which has been acceptable in studies in the educational field (Assis Gomes et al., 2017). With the factors composed of 23 items, organized according to the defined theoretical perspective, a confirmatory factor analysis for categorical variables was conducted. Upon evaluating the validity of the proposed construct, the following statistics are observed. RMSEA, CFI, TLI: The factorial model exhibited a good fit to the data (x2 (N=108, df=227) = 289.426, p=.003; CFI=.947; TLI=.940; RMSEA=.051 (CI90%=.031, .068).
Table 3. Descriptive statistics of items from the metacognitive strategies questionnaire and factorial loads
resulting from the confirmatory factor analysis.
N Items Mean SD Skewness Factor loading
Metacognitive strategies for planning
il ... We ensure to have clarity on what the problem is about 4.61 0.58 -1.16 0.38
i19 ... We realize that we can establish assumptions (hypotheses) to solve the 4.16 0.95 -1.15 0.80
problem
i15 ... First, we plan how to deal with the problem 4.09 0.98 -1.15 0.70
i13 ... We make sure to focus on the objective requested in the problem 4.63 0.58 -1.30 0.46
i18 ... We realize that it is necessary to organize time, roles, and responsibilities to 4.29 0.97 -1.36 0.72
respond to what the professor has requested
i7 ... We discussed the steps that we must take to solve the task (strategies) 4.35 0.81 -1.25 0.66
i5 ... We discussed different ways to solve the problem with the aim to choose 4.36 0.91 -1.45 0.58
then the best one
i14 ... We agreed on the same approach to solving the problem individually, to then 3.79 1.25 -0.72 0.50
compare and review results or calculations together
Metacognitive strategies for monitoring and, if necessary, regulation
i12 ... Throughout the problem solving process, we continuously check whether we 4.44 0.72 -1.19
are on the right track 0.55
i10 ... The groups ensures that everyone understand individually in order to pro- 3.94 1.13 -0.89 0.70
ceed with the resolution of the issue
i4 ... Every now and then we monitor the time we have left to finish solving the 4.37 0.90 -1.57 0.30
problem
i9 ... We questioning the evolving mathematical model 4.32 0.85 -1.22 0.75
i6 ... We questioning whether mathematical procedures (equation solving, 4.36 0.93 -1.55 0.63
variable manipulation, matrix resolution, application of theorems) are well-
developed
i3 ... We endeavor to try to clarify if the solution is reasonable according to the 4.54 0.64 -1.04 0.61
real problem
i8 ... When difficulties arose, we discussed them together as a groups 4.38 0.84 -1.19 0.62
i11 ... We realized the need to ask the teacher for support when we encountered 4.49 0.79 -1.82 0.56
difficulties
i20 ... In some occasions during the resolution process, we realized that we 4.33 0.70 -1.40 0.67
needed to change the resolution strategy
i2 ... In some occasions, we notice that it is necessary to change the work plan 4.21 0.88 -0.85 0.46
(due to confusion or errors)
i23 ... When we notice that we are facing difficulties in reaching a solution to the 4.32 0.84 -1.64 0.73
problem, we undertake a new plan
i17 ... When there are incorrect solutions, we review the resolution process, and if 4.41 0.80 -1.34 0.63
deemed necessary, we propose a new work plan
Metacognitive strategies for evaluation 3.47 1.30 -0.43 0.78
i16 ... At the final of process, we collectively reflect on the quality of our perfor-
mance, the fulfillment of roles we undertook, time management, and agree-
ments
i21 ... At the final of process, we reflect on the entire problem-solving process to 3.96 1.00 -1.03 0.81
enhance our performance in future opportunities, considering successes, er-
rors, strategies, achievements, and obstacles.
i22 ... At the final of process, we discuss whether there was alternative approach 3.63 1.30 -0.70 0.72
that could have led to the solution.
TOTAL 4.23 0.89 -1.19 NA
The Mann-Whitney U test results indicated that there are no statistically significant differences associated with gender in the metacognitive strategies of (1) 'planning,' (2) 'monitoring and, if necessary, regulation,' and (3) 'evaluation' (p > 0.05).
Figures 1a and 1b of metacognitive strategies for planning related to (i13) task objective and (i1)
clarity of the problem, did not reveal students in disagreement. However, 7% of male students expressed indifference compared to females who predominantly chose agreement or complete agreement. Nevertheless, these differences are not statistically significant (i13: U = 1320, p = 0.20; i1: U = 1072.5, p = 0.41). Only the strategy related to (i14) agreeing on the same way to solve the problem and then comparing together showed higher disagreements, which were not statistically significant, and a lower frequency of usage (U = 1171.5, p = 1).
¡7 4.34 (0.84 1.4% 1.4% 11.3% 33.8% 52.1%
¡5 4.38 (0.83 0.0% 4.2% 9.9% 29.6% 56.3%
¡19 4.10 (0.97 1.4% 5.6% 16.9% 33.8% 42.3%
¡18 4.25 (1.04 2.8% 4.2% 14.1% 22.5% 56.3%
¡15 4.14(1.05 2.8% 5.6% 14.1% 29.6% 47.9%
¡14 3.77 (1.29 7.0% 11.3% 19.7% 21.1% 40.8%
¡13 4.66 (0.61 0.0% 0.0% 7.0% 19.7% 73.2%
¡1 4.56 (0.63 0.0% 0.0% 7.0% 29.6% 63.4%
Ker«
:
Mean (SD) 1
Figure 1a. The percentage for likert scale 'planning', mean and standard deviation of the student male
¡7 4.36 (0.74 0.0% 00% 15.2% 33.3% 51.5%
¡5 4.30 (1.07 3.0% 6.1% 9.1% 21.2% 60.6%
¡19 4.30 (0.88 3.0% 0.0% 9.1% 39.4% 48.5%
¡18 4.36 (0.82 0.0% 3.0% 12.1% 30.3% 54.5%
¡15 4 00 (0.87 3.0% 0.0% 18.2% 51.5% 27.3%
¡14 3.82 (1.18 6.1% 6.1% 24.2% 27.3% 36.4%
¡13 4.58 (0.50 0.0% 0.0% 0.0% 42.4% 57.6%
¡1 4.70 (0.47 0.0% 0.0% 0.0% 30.3% 69.7%
Mean (SD) 1
Figure 1b. The percentage for likert scale 'planning', mean and standard deviation of the student female
While female students use metacognitive strategies at a slightly higher level, there were no significant differences between the groups regarding (i7) discussing steps (U = 1178.5, p = 0.96), (i19) realizing the need to establish assumptions (U = 1036.5, p = 0.31), and (i18) discussing time organization, roles, or other available resources to solve the task (U = 1151.5, p = 0.88). Likewise, no significant differences were found in the use of metacognitive strategies for (i5) discussing different ways to solve a problem to choose the best one (U = 1157, p = 0.91) and for (i15) planning first (U = 1339.5, p = 0.21); even though men have slightly higher scores than women.
Figures 2a and 2b of metacognitive monitoring and, if necessary, regulation strategies show that they are, on average, higher than the total scale. Students predominantly tend to agree or strongly agree with the statements.
¡9 4.23 (0.68 1.4% 1.4% 16.9% 33.6% 46.5% ¡9 4.52 (0.76) 0.0% 3.0% 6.1% 27.3% 63.6%
¡8 4.34 (0.84 0.0% 4.2% 11.3% 31.0% 53.5% ¡3 4.45 (0.83) 0.0% 3.0% 12.1% 21.2% 63.6%
¡6 4.25 (0.98 £.8% 2.8% 12.7% 29.6% 52.1% ¡6 4.58 (0.79) 0.0% 3.0% 9.1% 15.2% 72.7%
¡4 4.48 (0.83 1.4% 1.4% 8.5% 25.4% 63.4% ¡4 4.12 (1.02) 3.0% 3.0% 18.2% 30.3% 45.5%
¡3 4.58 (0.62 0.0% 0.0% 7.0% 28.2% 64.8% Percent _ 100 ¡3 4.45 (0 67) 0.0% 0.0% 9.1% 36.4% 54.5%
¡23 4.23 (0.93 2.8% 2.8% 8.5% 40.8% 45.1% ■ 75 ¡23 4.52 (0.57) 0.0% 0.0% 3.0% 42.4% 54.5%
¡20 4.35 (0.78 1.4% 1.4% 5.6% 43.7% 47.9% 50 25 0 ¡20 4.27 (0.76) 0.0% 3.0% 9.1% 45.5% 42.4%
¡2 4.24 (0.87 0.0% 4.2% 16.5% 32.4% 47.9% ¡2 4.15 (0.91) 0.0% 6.1% 15.2% 38.4% 42.4%
¡17 4.46 (0.81 0.0% 4.2% 7.0% 26.8% 62.0% ¡17 4.30 (0.77) 0.0% 3.0% 9.1% 42.4% 45.5%
¡12 4.38 (0.76 0.0% 2.8% 8.5% 36.6% 52.1% ¡12 4.58 (0.61) 0,0% 0.0% 6,1% 30.3% 63.6%
¡11 4.48 (0.84 1.4% 2.8% 5.6% 26.8% 63.4% ¡11 4.52 (0.67) 0.0% 0.0% 9.1% 30.3% 60.6%
i 10 3.86 (1.19 5.6% 8.5% 18.3% 29.6% 38.0% ¡10 4.12 (0.99) 0.0% 9.1% 15.2% 30.3% 45.5%
Vlean (SD) 1 2 3 4 5 Mean (SD 1 1 2 3 4 5
Percent
100
75
50
25
0
Figure 2a. The percentage for likert scale 'monitoring and if necessary regulation', mean and standard deviation of the student male.
Figure 2b. The percentage for likert scale 'monitoring and if necessary regulation', mean and standard deviation of the student female.
The strategy related to ensuring that (i10) everyone understands individually before continuing with the problem resolution shows a slightly higher frequency of use in women (M = 4.12) compared to men (M = 3.86). However, there were no statistically significant differences (U = 1042, p = 0.34). The strategy (i3) of trying to clarify if the solution is reasonable according to the real problem concentrates more than 90% of preferences on agreement or strong agreement, with a slight advantage towards men (M = 4.58) compared to women (M = 4.45), which is not statistically significant (U = 1291.5, p = 0.33). Similarly, metacognitive strategies associated with (i9) questioning the evolving mathematical model (U = 947, p = 0.08), (i23) undertaking a new plan in the face of difficulties (U = 1003, p = 0.19), (i12) checking if one is on the right track (U = 1020.5, p = 0.24), and (i11) ask the teacher for support when we encountered difficulties (U = 1193, p = 0.86) concentrate 90% of responses in the agreement or strong agreement preference, with a slight advantage towards women that is not significant.
Figures 3a and 3b indicate a moderately lower average compared to the overall scale average, suggesting that students use metacognitive evaluation strategies less frequently.
Figure 3a. The percentage for likert scale 'evaluation', Figure 3b. The percentage for likert scale 'evaluation', mean and standard deviation of the student male. mean and standard deviation of the student female.
While the (i16) team reflection on performance, roles, time, and agreements strategy is slightly higher in women (m = 3.52) compared to men (M = 3.45), there are no statistically significant differences (U = 1140.5, p = 0.83). Even though male students more frequently use metacognitive evaluation strategies for (i21) improving performance in a future opportunity (M = 3.99) compared to women (M = 3.91) and (i22) analyzing other solution approach (M = 3.66) compared to women (M = 3.58), there are no statistically significant differences (i21: U = 1178.5, p = 0.96; i22: U = 1242, p = 0.61).
Discussions
The qualitative results reveal the presence of specific metacognitive strategies in engineering students, divided into planning categories (ME1) involving active discussion and joint decision-making on objectives, strategies, and available resources; monitoring and if necessary regulation categories (ME2) involving constant reflection and adjustment of the modeling process based on progress and encountered difficulties; and evaluation categories (ME3) highlighting students' ability to critically analyze the modeling process, resources used, and the effectiveness of employed strategies. Regarding the first research question, it can be concluded that the observed metacognitive strategies in engineering students align with existing literature based on the analytical framework of metacognitive strategies in the domain of mathematical modeling (Vorhölter et al., 2019).
Furthermore, the quantitative results support these qualitative findings. There is evidence of a generally high level of use of all metacognitive strategies. Previous research has highlighted the crucial role of metacognition in solving complex mathematical modeling problems among engineering students (Li, 2013; Palmer et al., 2013; Hidayat et al., 2018). Team collaboration has also been identified as a key factor that stimulates the effective use of metacognitive strategies, reinforcing the importance of social interactions in the development of metacognitive competence (Gainsburg, 2006; Newell et al., 2004; Cardella, 2008).
Regarding the second research question, it can be concluded that the instrument exhibits adequate psychometric properties. The analytical framework aligns appropriately with the observed data, as indicated by the results of the construct validity test. Additionally, it can be concluded that there were no statistically significant differences associated with gender male or female in terms of the metacognitive
strategies used in mathematical modeling processes. Therefore, the instrument is invariant across gender groups. The existing literature has shown various results in terms of gender disparities in the metacognitive dimension, especially in the context of engineering careers and the use of metacognitive strategies to address complex problems (Ciascai and Lavinia, 2011; Hong et al., 2021; Aravena-Diaz et al., in press). However, our findings coincide with the results obtained by Aravena-Diaz et al. (in press), indicating that no significant gender differences are observed when students face mathematical modeling tasks, especially in relation to the metacognitive strategies they use.
Because addressing modeling challenges is generally not the task of students alone, but is done in small groups, it is crucial not only to consider individual metacognitive strategies, but also to pay special attention to group metacognitive strategies (Vorhölter and Krüger, 2021). For this reason, an instrument has been developed to measure metacognitive strategies at the group level in this study.
Conclusions
Based on the qualitative and quantitative results obtained in this study on metacognitive strategies in the context of mathematical modeling among engineering students, clear patterns of behavior have been identified. Qualitative findings revealed the presence of three main categories of metacognitive strategies, namely planning strategies (ME1), monitoring and if necessary regulation strategies (ME2), and evaluation strategies (ME3). These categories reflect students' active involvement in the modeling process, highlighting their ability to collaborate, critically reflect, and effectively evaluate their own performance and team progress.
It is worth noting that this study provides additional evidence on the issue of gender disparities in the use of metacognitive strategies in the context of mathematical modeling, demonstrating that there were no statistically significant differences based on the gender variable. These findings contradict some previous research suggesting possible gender differences in the application of metacognitive strategies. The refutation is grounded in the strengths of mathematical modeling as an educational innovation in engineering training since, at the metacognitive level, it does not reproduce gender differences.
In conclusion, this study provides a deeper understanding of the dynamics of metacognitive strategies among groups of engineering students in the context of mathematical modeling and offers an instrument to measure metacognitive strategies that can be useful for future research exploring in-depth interactions between metacognitive strategies, mathematical performance, and other relevant factors in the field of mathematics and engineering education.
Acknowledgements
This research was supported by the Mathematical and Statistical Education Research Center (CIEMAE, for its name in Spanish), Catholic University of Maule, Talca, Chile.
The study has been approved by the Scientific Ethics Committee of the Catholic University of Maule in Act N°41/2021. All study participants expressed their willingness to participate freely, consensually and voluntarily.
Conflict of interests
The authors declare no conflict of interest.
Author Contributions
Conceptualization, N.C.M., M.A.D.; Investigation, N.C.M., M.A.D.; Methodology, N.C.M., M.A.D.; Software, N.C.M.; Formal analysis, N.C.M., M.A.D.; Visualization, N.C.M.; Writing—original draft preparation, N.C.M.; Writing—review and editing, M.A.D.. All authors have read and agreed to the published version of the manuscript.
Appendix A
Items included in the Metacognitive Strategies Questionnaire for Mathematical Modeling
When I solve problems that come from the real world...
11 ... We ensure to have clarity on what the problem is about
12 ... In some occasions, we notice that it is necessary to change the work plan (due to confusion or errors)
13 ... We endeavor to try to clarify if the solution is reasonable according to the real problem
14 ... Every now and then we monitor the time we have left to finish solving the problem
15 ... We discussed different ways to solve the problem with the aim to choose then the best one
16 ... We questioning whether mathematical procedures (equation solving, variable manipulation, matrix resolution, application of theorems) are well-developed
17 ... We discussed the steps that we must take to solve the task (strategies)
18 ... When difficulties arose, we discussed them together as a groups
19 ... We questioning the evolving mathematical model
110 ... The groups ensures that everyone understand individually in order to proceed with the resolution of the issue
111 ... We realized the need to ask the teacher for support when we encountered difficulties
112 ... Throughout the problem solving process, we continuously check whether we are on the right track
113 ... We make sure to focus on the objective requested in the problem
114 ... We agreed on the same approach to solving the problem individually, to then compare and review results or calculations together
115 ... First, we plan how to deal with the problem
116 ... At the final of process, we collectively reflect on the quality of our performance, the fulfillment of roles we undertook, time management, and agreements
117 ... When there are incorrect solutions, we review the resolution process, and if deemed necessary, we propose a new work plan
118 ... We realize that it is necessary to organize time, roles, and responsibilities to respond to what the professor has requested
119 ... We realize that we can establish assumptions (hypotheses) to solve the problem
120 ... In some occasions during the resolution process, we realized that we needed to change the resolution strategy
121 ... At the final of process, we reflect on the entire problem-solving process to enhance our performance in future opportunities, considering successes, errors, strategies, achievements, and obstacles.
122 ... At the final of process, we discuss whether there was alternative approach that could have led to the solution.
123 ... When we notice that we are facing difficulties in reaching a solution to the problem, we undertake a new plan
References
ABET. (2017). Engineering Accreditation Commission Criteria for Accrediting. Engineerings programs. ABET. http://www.abet.org Allen, K., Reed-Rhoads, T., Terry, R. A., Murphy, T. J., & Stone, A. D. (2008). Coefficient alpha: An engineer's interpretation of
test reliability. Journal of Engineering Education, 97, 87-94. https://doi.org/10.1002/j.2168-9830.2008.tb00956.x Alpers, B. (2021). Making Sense of Engineering Workplace Mathematics to Inform Engineering Mathematics Education. A Report for the Mathematics Interest Group. European Society for Engineering Education (SEFI). https://www.sefi.be/ wp-content/uploads/2021/04/Workplace_Mathematics_SEFI_final.pdf Aravena Díaz, M. D., Díaz Levicoy, D., Rodríguez Alveal, F., & Cárcamo Mansilla, N. (2022). Case study and mathematical modeling in the training of engineers. Characterization of STEM skills. Ingeniare. Revista chilena de ingeniería, 30(1), 37-56. http://dx.doi.org/10.4067/S0718-33052022000100037. Aravena-Díaz, M. D., Sanhueza Henríquez, S., Rodriguez Gallardo, M., & Cárcamo Mansilla, N. (in press). Mathematical modeling to reduce gender gaps in STEM: characterization of STEM skills in high school students. In V. Geiger, G. Kaiser & H. Siller (Eds.), Researching Mathematical Modelling Education in Disruptive Times, International Perspectives on the Teaching and Learning of Mathematical Modelling.
Assis Gomes, C., Almeida, L. S., & Núñez, J. C. (2017). Rationale and Applicability of Exploratory Structural Equation Modeling (ESEM) in psychoeducational contexts. Psicothema, 29(3), 396-401. https://doi.org/10.7334/psicothema2016.369
Bembenutty, H. (2007). Self-Regulation of Learning and Academic Delay of Gratification: Gender and ethnic differences among college students. Journal of Advanced Academics, 18(4), 586-616. https://doi.org/10.4219/jaa-2007-553
Bidjerano, T. (2005). Gender differences in self-regulated learning [Paper presentation]. Annual Meeting of the Northeastern Educational Research Association, Kerhonkson, New York. https://eric.ed.gov/?id=ED490777
Blum, W. (2011). Can Modelling Be Taught and Learnt? Some Answers from Empirical Research. In G. Kaiser, W. Blum, R. Borromeo Ferri, & G. Stillman (Eds.), Trends in Teaching and Learning of Mathematical Modelling. International Perspectives on the Teaching and Learning of Mathematical Modelling (pp. 15-30). https://doi.org/10.1007/978-94-007-0910-2_3
Borromeo, R. (2006). Theoretical and empirical differentiations of phases in the modelling process. ZDM - International Journal on Mathematics Education, 38(2), 86-95. https://doi.org/10.1007/BF02655883
Cardella, M. E. (2008). Which mathematics should we teach engineering students? An empirically grounded case for a broad notion of mathematical thinking. Teaching Mathematics and Its Applications, 27(3), 150-159. https://doi.org/10.1093/ teamat/hrn008
Cárcamo Mansilla, N., Aravena-Díaz, M. D., & Berres, S. (in press). Metacognitive Strategies in Mathematical Modelling with Groups of Engineering Students. In V. Geiger, G. Kaiser & H. Siller (Eds.), Researching Mathematical Modelling Education in Disruptive Times, International Perspectives on the Teaching and Learning of Mathematical Modelling. Cech, E., Rubineau, B., Silbey, S., & Seron, C. (2011). Professional role confidence and gendered persistence in engineering.
American sociological review, 76(5), 641-666. https://doi.org/10.1177/0003122411420815 Charles, M., & Bradley, K. (2009). Indulging our gendered selves? Sex segregation by field of study in 44 countries. American
journal of sociology, 114(4), 924-976. https://doi.org/10.1086/595942 Cheryan, S. (2012). Understanding the paradox in math-related fields: Why do some gender gaps remain while others do not?.
Sex roles, 66, 184-190. https://doi.org/10.1007/s11199-011-0060-z Ciascai, L., & Lavinia, H. (2011). Gender differences in metacognitive skills. A study of the 8th grade pupils in Romania.
Procedia - Social and Behavioral Sciences, 29, 396-401. https://doi.org/10.1016/j.sbspro.2011.11.255 Correll, S. J. (2004). Constraints into preferences: Gender, status, and emerging career aspirations. American sociological review, 69(1), 93-113.
Creswell, J. (2009). Research Design Qualitative, Quantitative and Mixed Methods Approaches. SAGE Publications.
Creswell, J., & Plano, V. (2018). Designing and Conducting Mixed Methods Research. SAGE Publications.
Dirksen, U. (2019). Trabajo del fututo y futuro del trabajo. Nueva Sociedad, 279. https://static.nuso.org/media/articles/
downloadsZ3.TC_Dirksen_279.pdf Escobar-Pérez, J., & Cuervo-Martínez, Á. (2008). Validez de contenido y juicio de expertos: Una aproximación a su utilización. Avances En Medición, 6(1), 27-36. http://www.scielo.org.mx/scielo.php?script=sci_arttext&pid =S1665-61802017000300042
Ferrando, P. J., & Anguiano-Carrasco, C. (2010). El análisis factorial como técnica de investigación en psicología. Papeles del
psicólogo, 31(1), 18-33. https://www.papelesdelpsicologo.es/pdf/1793.pdf Fitzpatrick, C. (1994). Adolescent Mathematical Problem Solving: The Role of Metacognition, Strategies and Beliefs [paper presentation]. Annual Meeting of the American Educational Research Association. New Orleans. https://files.eric. ed.gov/fulltext/ED374969.pdf
Flores-Ruiz, E., Miranda-Novales, M. G., & Villasís-Keever, M. A. (2017). The research protocol VI: How to choose the appropriate statistical test. Inferential statistics. Revista Alergia México, 64(3), 364-370. https://doi.org/10.29262/ram. v64i3.304
Gainsburg, J. (2013). Learning to Model in Engineering. Mathematical Thinking and Learning, 15(4), 259-290. https://doi.org/
10.1080/10986065.2013.830947 Garofalo, J., & Lester, F. (1985). Metacognition, Cognitive Monitoring, and Mathematical Performance. Journal for Research in
Mathematics Education, 16(3), 163. https://doi.org/10.2307/748391 Glaser-Zikuda, M., Hagenauer, G., & Stephan, M. (2020). The potential of qualitative content analysis for empirical educational
research. Forum Qualitative Sozialforschung, 21(1), 35-61. https://doi.org/10.17169/fqs-2H.3443 Hegedus, S. J. (2001). Problem Solving in Integral Calculus: One Role of Metacognitive Thinking. In R. Speiser, C. A. Maher, & C. N. Walter (Eds.), Proceedings of the 23rd Annual Meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education. (pp. 491-500). ERIC Clearinghouse for Science, Mathematics and Environmental Education. https://files.eric.ed.gov/fulltext/ED476613.pdf Hidayat, R., Zamri, S., & Zulnaidi, H. (2018). Does Mastery of Goal Components Mediate the Relationship between Metacognition and Mathematical Modelling Competency?. Educational Sciences: Theory & Practice, 18(3). https://doi. org/10.12738/estp.2018.3.0108 Hidayat, R., Zulnaidi, H., & Zamri, S. (2018). Roles of metacognition and achievement goals in mathematical modeling
competency: A structural equation modeling analysis. PLoS ONE, 13(11). https://doi.org/10.1371/journal.pone.0206211 Hong, E., O'Neil, H. F., Jr., & Feldon, D. (2005). Gender effects on mathematics achievement: Mediating role of state and trait self-regulation. In A. M. Gallagher, & J. C. Kaufman (Eds.), Gender differences in mathematics (pp. 264-293). New York, NY: Cambridge University Press. http://ndl.ethernet.edu.et/bitstream/123456789/16205/1/4.pdf#page=281
Humberto, J., & Rojas, C. (2017). La Cuarta Revolución Industrial o Industria 4.0 y su Impacto en la Educación Superior en Ingeniería en Latinoamérica y el Caribe. 15 Th LACCEI International Multi-Conference for Engineering, Education, and Technology. http://www.laccei.org/LACCEI2017-BocaRaton/work_in_progress/WP386.pdf International Engineering Alliance. (2014). 25 years of the Washington Accord. International Engineering Alliance. http://www. ieagreements.org/25_years/
Johanson, G. A., & Brooks, G. P. (2010). Initial scale development: Sample size for pilot studies. Educational and Psychological
Measurement, 70(3), 394-400. https://doi.org/10.1177/0013164409355692 Jonassen, D., Strobel, J., & Lee, C. (2006). Everyday Problem Solving in Engineering: Lessons for Engineering Educators.
Journal of Engineering Education, 9(2), 139-151. https://doi.org/https://doi.org/10.1002/j.2168-9830.2006.tb00885.x Kaiser, G., & Schwarz, B. (2010). Authentic Modelling Problems in Mathematics Education-Examples and Experiences. J Math
Didakt, 31, 51-76. https://doi.org/10.1007/s13138-010-0001-3 Kent, P., & Noss, R. (2003). Mathematics in the University Education of Engineers A Report to the Ove Arup Foundation. The
Ove Arup Foundation. https://www.ovearupfoundation.org/library/media-reports Kohlbacher, F. (2006). The use of qualitative content analysis in case study research. Forum Qualitative Sozialforschung, 7(1).
https://doi.org/10.17169/fqs-7.175 Kuckartz, U. (2019). Qualitative Text Analysis: A Systematic Approach. In: Kaiser, G., Presmeg, N. (Eds.) Compendium for Early Career Researchers in Mathematics Education. ICME-13 Monographs. Springer, Cham. https://doi.org/10.1007/978-3-030-15636-7_8
Li, T. (2013). Mathematical Modeling Education is the Most Important Educational Interface Between Mathematics and Industry. In A. Damlamian, J. Rodrigues, & R. Sträßer (Eds.), New ICMI Study Series (pp. 51-58). Springer. https://doi. org/10.1007/978-3-319-02270-3_5 Lyon. J. A., & Magana, A. J. (2020). A Review of Mathematical Modeling in Engineering Education. International Journal of
Engineering Education, 36(1), 101-116. https://www.ijee.ie/1atestissues/Vol36-1A/09_ijee3860.pdf Mann, A., & DiPrete, T. A. (2013). Trends in gender segregation in the choice of science and engineering majors. Social science
research, 42(6), 1519-1541. https://doi.org/10.1016Zj.ssresearch.2013.07.002 Maaß, K. (2006). What are modelling competencies?, ZDM, 38(2), 113-142. https://doi.org/10.1007/BF02655885 Mayring, P. (2014). Qualitative Content Analysis. Theoretical Foundation, Basic Procedures and Software Solution. https://
www.ssoar.info/ssoar/handle/document/39517 Muthén, L. K., & Muthén, B. O. (2017). Mplus User's Guide. Muthén & Muthén.
National Center for Science and Engineering Statistics (NCSES). 2023. Diversity and STEM: Women, Minorities, and Persons with Disabilities 2023. National Science Foundation. https://ncses.nsf.gov/wmpd
Newell, J., Dahm, K., Harvey, R., & Newell, H. (2004). Development metacognitive engineering teams. Chemical Engineering Education, 38(4), 106-129. https://doi.org/10.4018/978-1-5225-2212-6.ch006
Oliden, P. E., & Zumbo, B. D. (2008). Coeficientes de fiabilidad para escalas de respuesta categórica ordenada. Psicothema, 20(4), 896-901. http://www.psicothema.com/psicothema.asp?id=3572
Palmer, A., Amat, S., Busquier, S., Romero, P., & Tejada, J. (2013). Mathematics for Engineering and Engineering for Mathematics. In A. Damlamian, J. Rodrigues, & R. Sträßer (Eds.), New ICMI Study Series (pp. 185-198). Springer. https://doi.org/10.1007/978-3-319-02270-3_17 Penagos, H. P. (2011). How can metacognition be developed through problem-solving in higher education?. Ingeniería e
Investigación, 31(1), 213-223. https://repositorio.unal.edu.co/handle/unal/33495 Preacher, K., & Coffman, D. (2006). Computing power and minimum sample size for RMSEA. http://quantpsy.org/ Rakoczy, K., Buff, A., & Lipowsky, F. (2005). Dokumentation der Erhebungs-und Auswertungsinstrumente zur schweizerischdeutschen Videostudie." Unterrichtsqualität, Lernverhalten und mathematisches Verständnis". 1. Frankfurt: Main: GFPF ua.
Richert, A., Shehadeh, M., Willicks, F., & Jeschke, S. (2016). Digital Transformation of Engineering Education - Empirical Insights from Virtual Worlds and Human-Robot-Collaboration. International Journal of Engineering Pedagogy (IJEP), 6(4), 23. https://doi.org/10.3991/ijep.v6i4.6023
Riegle-Crumb, C. (2006). The path through math: Course sequences and academic performance at the intersection of race-ethnicity and gender. American Journal of Education, 113(1), 101-122. https://doi.org/10.1086/506495
Schoenfeld, A. (1992). Learning to think mathematically: Problem solving, metacognition, and sense making in mathematics. In D. Grouws (Ed.), Handbook for Research on Mathematics Teaching and Learning (pp. 334-370). New York: MacMillan. http://hplengr.engr.wisc.edu/Math_Schoenfeld.pdf
Schukajlow, S., & Krug, A. (2013). Planning, monitoring and multiple solutions while solving modeling problems. In A. M. Lindmeier & A. Heinze (Eds.), Proceedings of the 37th Conference of the International 4 -177 Group for the Psychology of Mathematics Education (pp. 177-184). Kiel, Germany: PME. https://www.researchgate.net/publication/274257874_ PLANNING_MONITORING_AND_MULTIPLE_SOLUTIONS_WHILE_SOLVING_MODELLING_PROBLEMS Sanabria, T., & Penner, A. (2017). Weeded out? Gendered responses to failing calculus. Social Sciences, 6(2), 47. https://doi.
org/10.3390/socsci6020047 Simons, H. (2013). El estudio de caso: Teoría y práctica. Ediciones Morata, S. L. Madrid
Smith-Doerr, L., Alegria, S. N., & Sacco, T. (2017). How diversity matters in the US science and engineering workforce: A critical review considering integration in teams, fields, and organizational contexts. Engaging Science, Technology, and Society, 3, 139-153. https://doi.org/10.17351/ests2017.142 Soon, W., Lioe, L. T., & McInnes, B. (2011). Understanding the difficulties faced by engineering undergraduates in learning mathematical modelling. International Journal of Mathematical Education in Science and Technology, 42(8), 10231039. https://doi.org/10.1080/0020739X.2011.573867 Stillman, G. (2011). Applying Metacognitive Knowledge and Strategies in Applications and Modelling Tasks at Secondary School. In G. Kaiser, W. Blum, R. Borromeo Ferri, & G. Stillman (Eds.), Trends in Teaching and Learning of Mathematical Modelling. International Perspectives on the Teaching and Learning of Mathematical Modelling (pp. 165-180). https:// doi.org/10.1007/978-94-007-0910-2_18
Stillman, G., & Galbraith, P. (1998). Applying mathematics with real world connections: metacognitive characteristics of secondary students. Educational Studies in Mathematics, 36(2), 157-194. https://doi.org/10.1023/A:1003246329257
Tashakkori, A., & Teddlie, C. (2003). Handbook of mixed methods in social and behavioral research. Thousand Oaks, CA: Sage. https://doi.org/10.4135/9781506335193
Tristán-López, a. (2008). Modificación al modelo de Lawshe para el dictamen cuantitativo de la validez de contenido de un instrumento objetivo. Avances En Medición, 6, 37-48. https://dialnet.unirioja.es/servlet/articulo?codigo=2981185
Tzohar-Rozen, M., & Kramarski, B. (2014). Metacognition, Motivation and Emotions: Contribution of Self-Regulated Learning to Solving Mathematical Problems. Global Education Review, 1(4), 76-95. http://ger.mercy.edu/index.php/ger/article/ view/63
Vorhölter, K. (2017). Measuring Metacognitive Modelling Competencies. In G Stillman, W. Blum, & G. Kaiser (Eds.), Mathematical Modelling and Applications. International Perspectives on the Teaching and Learning of Mathematical Modelling (pp. 175-185). Springer, Cham Vorhölter, K. (2018). Conceptualization and measuring of metacognitive modelling competencies: empirical verification of
theoretical assumptions. ZDM Mathematics Education, 50(1), 343-354. https://doi.org/10.1007/s11858-017-0909-x Vorhölter, K. (2019). Enhancing metacognitive group strategies for modelling. ZDM Mathematics Education, 51, 703-716. https://doi.org/10.1007/s11858-019-01055-7
Vorhölter, K., Krüger, A., & Wendt, L. (2019). Chapter 2: Metacognition in Mathematical Modeling - An Overview. In S. Chamberlin & B. Sriraman (Eds.), Affect in Mathematical Modeling. Advances in Mathematics Education (pp. 29-51). Springer. https://doi.org/10.1007/978-3-030-04432-9_3 Vorhölter, K., & Krüger, A. (2021). Metacognitive strategies in modeling: Comparison of the results achieved with the help of
different methods. Quadrante, 30(1), 178-197. https://doi.org/10.48489/quadrante.23653 Wang, M. T., & Degol, J. L. (2017). Gender gap in science, technology, engineering, and mathematics (STEM): Current knowledge, implications for practice, policy, and future directions. Educational psychology review, 29, 119-140. https:// doi.org/10.1007/s10648-015-9355-x Wedelin, D., Adawi, T., Jahan, T., & Andersson, S. (2015). Investigating and developing engineering students' mathematical modelling and problem-solving skills. European Journal of Engineering Education, 40(5), 557-572. https://doi.org/10. 1080/03043797.2014.987648
Weller, J., Gontero, S., & Campbell, S. (2019). Cambio tecnológico y empleo: una perspectiva latinoamericana. Riesgos de la sustitución tecnológica del trabajo humano y desafíos de la generación de nuevos puestos de trabajo. Macroeconomía del Desarrollo, N° 201 (LC/TS.2019/37), Santiago, Comisión Económica para América Latina y el Caribe (CEPAL). www.cepal.org/apps
Wengrowicz, N., Dori, Y. J., & Dori, D. (2018). Metacognition and Meta-assessment in Engineering Education. In Y. J. Dori, Z. R. Mevarech, & D. R. Baker (Eds.), Cognition, Metacognition, and Culture in STEM Education. Innovations in Science Education and Technology (pp. 191-216). Springer, Cham. https://doi.org/10.1007/978-3-319-66659-4_9
World Economic Forum, (2020). The Future of Jobs Report 2020. https://www.weforum.org/reports/the-future-of-jobs-report-2020
Woetzel, J., Madgavkar, A., Ellingrud, K., Labaye, E., Devillard, S., Kutcher, E., Manyika, J., Dobbs, R., & Krishnan, M. (2015). The power of parity: How advancing women's equality can add $12 trillion to global growth. McKinsey Global Institute. https://www.mckinsey.com/featured-insights/employment-and-growth/how-advancing-womens-equality-can-add-12-trillion-to-global-growth
Yin, R. K. (2014). Case study research design and methods (5th ed.). Thousand Oaks, CA: Sage.
Yildirim, T. Pinar. (2010). Understanding the modelling skills shift in engineering: The impact of self-efficacy, epistemology, and metacognition [Master's thesis, University of Pittsburgh]. University of Pittsburgh ProQuest Dissertations Publishing. https://www.proquest.com/docview/858073953
