CC BY
15
DOI: 10.15514/ISPRAS-2023-35(1)-7
Students' Systems Thinking Competencies Level Identification through Concept Maps Assessment
1 J.R. Aguilar-Cisneros, ORCID: 0000-0003-3040-157X <jorge.aguilar@upaep.mx> 2 R. Valerdi, ORCID: 0000-0002-2746-0395 <rvalerdi@arizona.edu> 3 B.P. Sullivan, ORCID: 0000-0002-4646-7277 <b.p.sullivan@utwente.nl> 1 Universidad Popular Autónoma del Estado de Puebla (UPAEP), Puebla, Pue., México, 72410
2 University of Arizona, Tucson, AZ 85721, USA
3 University of Twente
p.o. box 217, 7500 AE Enschede, The Netherlands
Abstract. Systems Thinking Competencies have become extremely important and widely studied due to increasing systems complexity. Because of this, when they are taught, it is extremely useful to identify whether or not students own Systems Thinking Competencies in order to design a specific teaching strategy. This research applied an Adapted Holistic Scoring Method to assess Concept Maps developed by postgraduate and undergraduate engineering students in order to identify Systems Thinking Competencies. It had two phases. At the first one, Students showed an acceptable knowledge of cost estimation drivers, and a certain level of Systems Thinking Competencies. In the second phase, both cost estimation drivers and Systems Thinking Competencies showed an improvement. Mann-Whitney U-test was applied in order to identify if there were significant differences between Phase 1 and Phase 2. Confidence level of 95%, and a significance level of 0.05 was considered.
Keywords: system thinking; student's competencies; Adapted Holistic Scoring Method; Concept Maps; U-test
For citation: Aguilar-Cisneros J.R., Valerdi R., Sullivan B.P. Students' Systems Thinking Competencies Level Identification through Concept Maps Assessment. Trudy ISP RAN/Proc. ISP RAS, vol. 35, issue 1, 2023. pp. 101-112. DOI: 10.15514/ISPRAS-2023-35(1)-7
Определение уровня способности студентов к системному мышлению с помощью оценки концептуальных карт
1 Х.Р. Агилар-Сиснерос, ORCID: 0000-0003-3040-157X <jorge.aguilar@upaep.mx> 2 Р. Валерди, ORCID: 0000-0002-2746-0395 <rvalerdi@arizona.edu> 3 Б.П. Салливан, ORCID: 0000-0002-4646-7277 <b.p.sullivan@utwente.nl>
1 Народный автономный университет штата Пуэбла, Мексика, 72410, Пуэбла 2 Аризонский университет, США, AZ 85721, Тусон 3 Университет Твенте Нидерланды, 7500 AE Энсхеде
Аннотация. Способность к системному мышлению стала чрезвычайно важной и широко изучаемой из-за возрастающей сложности систем. Из-за этого при обучении студентов очень полезно определить, обладают ли они способностями к системному мышлению, чтобы разработать конкретную стратегию
обучения. В нашем исследовании применялся адаптированный целостный метод для оценки концептуальных карт, построенных аспирантами и студентами технических специальностей с целью выявления способностей к системному мышлению. Исследование состояло из двух фаз. На первой фазе студенты показали приемлемое знание факторов оценки затрат и некоторый уровень способности к системному мышлению. На втором этапе оба эти показателя были улучшены. Применялся U-критерий Манна-Уитни, чтобы определить, есть ли существенные различия между фазой 1 и фазой 2. Учитывались уровень достоверности 95% и уровень значимости 0,05.
Ключевые слова: системное мышление; способности студентов; адаптированный целостный метод оценки; концептуальные карты; U-критерий
Для цитирования: Агилар-Сиснерос Х.Р., Валерди Р., Салливан Б.П. Определение уровня способности студентов к системному мышлению с помощью оценки концептуальных карт. Труды ИСП РАН, том 35, вып. 1, 2023 г., стр. 101-112. DOI: 10.15514/ISPRAS-2023-35(1)-7
1. Introduction
Systems Thinking is a holistic approach to analyze, and solve complex problems and systems [1, 2]. It means looking for an acceptable solution among several potential solutions. These solutions will own relationships of correlation, some of them may involve cause and effect. Systems Thinking emphasizes the complexity of relationships, seeking out webs of causality rather than single, linear causes [3]. In this regard, this study focuses on, firstly, identifying how students change their mind, over time, about cost estimation drivers while they receive an academic course related to this topic. Secondly, how students represent their change of mind through a concept map, and finally how those Concept Maps shed light on Systems Thinking. Particularly, one Systems Thinking Competence defined by [4], was analyzed. The study was applied to undergraduate and postgraduate students. They were enrolled in a cost estimation course during the spring 2020, where, among several concepts, cost estimation drivers were studied. This paper took into account the COCOMO model [5].
After Students' Concept Maps (SCMs) were assessed in phase 2, the outcomes showed an increasing knowledge on estimation cost drivers regarding the first evaluation on phase 1. Additionally, Systems Thinking Competence #3 was identified in the same way at a higher level on phase 2 than phase 1.
2. Background
The term Systems Thinking is known as "the art and science of making reliable inferences about behavior by developing an increasingly deep understanding of an underlying structure" [6]. Systems Thinking has been applied in a wide number of areas [7-9]. Based on these cases, Systems Thinking has demonstrated an ability to manage the complexity of systems, technical and societal, by considering the future implications of decision making and their long-term consequences. Additionally, Systems Thinking can be applied to decision making, and it often involves understanding the complexity of the situation, to see causal relationships, identify dynamic relationships among variables, and so on. In order to identify relationships [10, 11], and complex relationships [12, 13], Conceptual Maps are often used for modeling.
2.1 STC and Skills
There is neither an agreement about competencies definition nor its interpretation. The interpretation ranges from a description of competency in terms of performance and competence in terms of skills acquired by training to a broad view that encompasses knowledge, understanding, skills, abilities and attitudes [14].
This paper uses the term competence instead of competency. Competence is defined as a set of skills acquired by training or teaching. Systems Thinking can be described as a dual ability to understand systems and analyze circumstances, questions, or problems from a systems perspective [15].
Systems can be divided into three aspects, function (utility), structure (form) and behavior (dynamics) [16]. When both terms, Competence and Systems Thinking, are put together. Systems Thinking Competence (STC) arises. STC can be defined as: "Aspect that implies skills, knowledge, attitudes and behavior applied to tasks or activities where Systems Thinking perspective is needed". [1] stated, Systems Thinking can help to develop higher-order thinking skills, such as critical thinking, in order to understand and address complex, interdisciplinary, real-world problems. In this sense, some critical thinking skills (competencies) were defined by [17].
This paper was focused on just one of the eight STC defined by [4]. The competence selected, among eight of them, was the number 3. Competence #3: Ability to see relationships, a system can be understood in the context of relationships.
The competence #3 (STC #3) was selected because this competence is most closely related to cost modeling and allows to isolate a single competence without worrying about the confounding or mediating effects of others.
2.2 Cost Estimation
There are several Effort Estimation Methods, among them, COSMIC [18], User Stories [19]. COCOMO model was used in this research, it estimates the amount of effort in person-months (PM). COCOMO's equations require effort multipliers (EM) and scale drivers/factors (SDF) as inputs. COCOMO defines five Scale Drivers Factors: PREC, FLEX, RESL, TEAM, and PMAT. Additionally, COCOMO defines several Effort Multipliers: RELY, DATA, CPLX, RUSE, DOCU, TIME, STOR, PVOL, ACAP, PCAP, PCON, APEX, LTEX, PLEX, TOOL, SITE, and SCED. Effort Multipliers and Scale Drive Factors were searched out on each SxCMs (Student x's Concept Map) in order to identify how many of them were used to build their SxCM. Depending on this, a rate was assigned to each SxCM [20].
2.3 CMs and Scoring
Concept Maps were developed by [21] at Cornell University, in order to understand changes in students' knowledge, mainly because CMs are graphical tools for organizing and representing conceptual understanding [22]. Additionally, CMs and Systems Thinking can be used together because they share common characteristics such as structure, dynamism and hierarchy, and some researchers indicate that increase in the number of concepts, connections and diversity in CMs are a reliable parameter for measuring students' systematic thinking [23, 24]. CMs have been used in a wide spectrum of areas due to its advantages [25-29].
Additionally, [26] did an evaluation of six scoring methods: 1) holistic, 2) holistic with master map, 3) relational, 4) relational with master map, 5) structural, and 6) structural with master map. To calculate similarity between subjects' Concept Maps, and the Master Concept Map (MCM), [30] a set of theoretic methods described by [20] can be used.
In general terms, there is a big quantity of research where Concept Maps have been used in order to identify Systems Thinking skills or competencies [31-35].
3. Research Methodology
This section describes the research methodologies' main elements and instruments used to gather and analyze data.
3.1 Research General Background
In [36], authors analyzed whether particular features (medium) of Concept Maps affect the assessment of student's Systems Thinking. They found that the medium rarely influenced the validity of Concept Maps for Systems Thinking. Furthermore, the authors suggest Concept Maps as an appropriate assessment of Systems Thinking.
However, for this research neither specific tool to build Concept Maps was requested nor specific instructions to build them was given. Additionally, for this study an Adapted Holistic Scoring Method (AHSM) was used together with a Master Map Methodology. The original holistic methodology was adapted because of the type of raw material gathered. There were two assessments. The first one was applied in January (Phase 1) and the second one in February (Phase 2), both of them in 2020. The assessments consisted, basically, about identifying how many Cost Estimation Drivers terms the students were able to use, and how many Scale Factors and Effort Multipliers they were able to specify when they developed their Concept Maps, additionally the level of Systems Thinking Competence #3 embedded into their Concept Maps was tried to identify. The second assessment analyzed if students used specific cost estimation drivers. System Thinking Competence #3 embedded into their Concept Maps was evaluated. Particularly, we identified if students have reached a better level.
3.2 Research Problem
Solving complex problems is one of the main activities in some industries, where, in the future, students could be hired. In this sense, identifying Systems Thinking Competencies owned by students can be useful. With this identification, training activities to strengthen them could be planned and managed. This research identified one Systems Thinking Competence owned by a group of students (graduated and ungraduated) The Systems Thinking Competence identified was STC #3.
3.3 Research Focus
This research was focused on identifying a STC owned by students, particularly we were focused on how students change their ability to see relationships after they received theory about cost estimation drivers and how they were able to represent it through a Concept Map. Applying the Adapted Holistic Scoring Method together with the assessment rubric, Concept Maps were assessed.
3.4 Research Aim and Research Questions
Research questions guidelines this study:
RQ1: How does ability to see relationships (STC #3) change over time as a result of learning cost modeling?
RQ2: How do students' mental models of the factors that impact project costs change over time?
3.5 Participants
This research was applied to a sample of participants consisting of undergraduate and postgraduate students. All of them were enrolled at different Systems and Industrial Engineering Department careers. The first study was applied to 61 students and the second to 45 students. The survey wasn't mandatory in order to avoid any kind of bias. The study was applied in the spring 2020.
3.6 Instrument and Procedures
The methodology included two phases. The first one (see Figure. 1) assess the degree of similarity between Students' CMs (SxCM) and the Master Concept Map (MCM). High levels of similarity indicate SxCMs own a considerable quantity of Cost Estimation Drivers in their Concept Maps. In order to do this assessment, the Adapted Holistic Scoring Method [30] was used (See Figure 1). According to [30] in order to assess CMs similarity, these have to be constructed using the same set of concepts. However, when SxCMs was requested for this research, and when MCMs were developed, both of them used different sets of concepts because our students did not receive a common set of concepts. Our students did not receive specific information about how to build CMs. During the first assessment, students did not know Cost Estimation Drivers. 104
Fig. 1. Adapted Holistic Scoring Method (AHSM)
These equations were used:
a) SXI = MCM n SXCM;
b) SXU = MCMterm. + I^=1SxCMterm.) - SXI;
c) S S = — •
d) Gx=SxSm*10;
e) EG = ExpertGrage; f FGx = (EG + Gx)/2.
3.7 Assessment Rubric
The assessment was done through a rubric (see Table 1). This rubric was used to evaluate each SxCM, and level of System Thinking Competence #3 (STC#3) was identified. The elements of STM#3 identified indicate whether students saw relationships between the cost estimation drivers (direct costs) and the costs around the project (indirect costs). Regarding relationships, three levels were defined: Low, Medium and High. A SxCMs got low level, if 0 or 1 related element with STC #3 was identified, when 0 elements were identified, it means SxCMs only contains Cost Drivers Estimation related with the cost estimation project itself (direct costs). A SxCMs got medium level, if two related elements with STC #3 were identified. Finally, a SxCMs got high level, if three or more related elements with STC #3 were identified.
Table 1. Rubric to assess SxCMs vs STC #3
SxCM Assessment STC #3 Identified Level
Low Medium High 1 2 3
Low. If 1 or Medium. If High. If 3
0 external 2 external or more
elements elements external
were were elements
identified identified were
identified
The methodology to assess the STC #3 in each SxCMs is shown in figure 3. Figures 11 and 12 represent the outcomes.
Apply Rubric to SXCM
Analyzing results
Creating graphics
Fig. 2. Methodology to identify STC #3 into SxCMs
3.8 Statistical Test
The Mann-Whitney U-test was used to compare SxCMs developed at the first phase vs SxCM developed at the second phase
Two Mann-Whitney tests were applied, one of them to assess Cost Estimation Drivers. The second test to assess STC#3's level reached.
The first Mann-Whitney test was applied to Cost Estimation Drivers. The Null Hypothesis (H0), and the alternative hypothesis (H1) were defined:
H0=NCEDi >= NCED2; Hi=NCEDi < NCED2.
NCEDi represents the number of Cost Estimation Drivers included by students in their CMs in the first phase study, and NCED2 represents the number of Cost Estimation Drivers included by students in their CMs in the second phase study. A confidence level of 95%, and a significance level of 0.05 (a = 0.05) were defined.
The size sample was 105, in the first phase study there were 60 students (one of them was eliminated) and in the second phase there were 45 students. It means, 105 Concept Maps were analyzed. The second Mann-Whitney test was applied to Systems Thinking Competence Level #3 reached by students. The Null Hypothesis (H0), and the alternative hypothesis (H1) were defined:
Ho=STCLi <= STCL2; Hi=STCLi > STCL2.
STCLi represents the level of Systems Thinking Competence #3 reached by students in their CMs in the first phase study, and STCL2 represents the level of Systems Thinking Competence #3 reached by students in their CMs in the second phase study. A confidence level of 95%, and a significance level of 0.05 (a = 0.05) were defined.
The size sample was 95. In the first phase study there were 54 students and in the second phase there were 41 students. Some students were eliminated because they didn't reach any level of STC#3. Some parameters have to be computed, U and Zu, according with the following equations:
,n1(n1 + 1)
U-i = nini +----Ri; (1)
n2(n2 + 1)
U2=nini+----R2; (2)
Zu= , , (3)
n1n1(n1 + n2 + 1)
12
4. Research Results
This section presents the outcomes regarding degree of similarity between Students' CMs (SxCM) and the Master CMs (MCM). Additionally, the results regarding the level of Systems Thinking Competence (STC #3) reached are shown.
4.1 AHSM and Rubric Results
The first assessment (phase one), maximum and minimum scores reached for SxCMs (see Table 2) were calculated. The SxCM's results showed low grades but it was due to, at this point of time, the
students had not received formal teaching about cost estimation drivers and they had a whole freedom to develop their own Concept Maps.
As it can be seen in Table 2, due to low standard deviation, most of the students got around 4.2 points and an average of 4.15
Table 2. Phase one outcomes (First assessment)
MCMvsS1CMtoS61CM
Average grade Max. grade Min. grade Mean Standard Deviation
4.15 6.42 2.69 4.18 0.98
Approximately one month later second assessment was applied, and the same task was requested. There was an increase of almost two points, from 6.42 to 8.06 (See Table 3)). Additionally, more specific cost estimation factors were used by students.
Table 3. Phase two outcomes (Second assessment)
MCMvsS1CMtoS45CM
Average Max. Min. Mean Standard
grade grade grade Deviation
4.49 8.06 3.21 4.41 0.88
Additionally, an analysis about Scale Drive Factors and Effort Multipliers was applied. 64.4 % of students included, at least, one SDF on their SxCMs (See Figure 3).
Scale Factors
- Identify Nü Weniify
Fig. 3. Scale drive factors
Moreover, EM were identified on SxCMs 73.3% of students included it on their SxCMs. It means almost all students increased their knowledge about specific elements that influence a project cost.
STC # 3 Assessment
Low Medium High
Fig. 4. Level of STC #3 identified. First assessment phase
Furthermore, the assessment rubric (See Table 1) was applied in order to identify aspects about Systems Thinking Competence #3 (STC#3). It was applied in the first and second assessment phase (See Figures 4, and 5).
Fig. 5. Level of STC #3 identified. Second assessment phase Furthermore, the assessment rubric (See Table 1) was applied in order to identify aspects about Systems Thinking Competence #3 (STC#3). It was applied in the first and second assessment phase (See Figures 4, and 5).
The S19CMs (student 19 of 61, first assessment phase), is shown in order to identify what kind of Student Concept Map was built (See Figure 6). This student got a final grade of 4.6 points in the first assessment phase. 4.6 means a similarity of 46% with Master Concept Map, in other words, the student 19 included 5 of 19 cost estimation drivers expected.
Fig. 6. S19CM (First assessment phase) The same student (Student 35 = student 19 in phase 1), but in the second assessment phase (Phase 2), got a final grade of 8.1. This grade, 8.1, means a similarity of 81% with Master Concept Map. In other words, the student 35 included 13 of 20 cost estimation drivers. There was an improvement of 3.5 points.
4.2 Mann-Whitney Results
The first Mann-Whitney U-Test applied to Estimation Cost Drivers Included by students in their Concept Maps at first and second study gives us the next results.
The Null Hypothesis (Ho), and the alternative hypothesis (Hi) were defined to the first MannWhitney test, where significant Estimation Cost Drivers included at first phase regarding Estimation Cost Drivers included at second phase study was computed.
Phase i and Phase 2 were related to NCEDi and NCED2, respectively. The sample size for the first phase and the second phase was ni =54, and n2 =4i. The median to Phase i, and Phase 2 was 3 and 5 respectively (See Figure 7).
Fig. 7. Medians Phase 1 and Phase 2
Additionally, Figure 8 shows frequency histogram of Estimation Cost Drivers included in students' Concept Maps. This graph represents how many Cost Estimation Drivers were included in the Students' Concept Maps at the first phase, and at the second phase.
Fig. 8. ECD included at CMs The rank of Phase 1 and Phase 2 was calculated: Ranki = 2641, and Rand2 = 2924. After these calculations, the U parameter was calculated for both Groups: U1 = 1889, and U2 = 811 (See equation (1), (2)). Hence U = 811.
Zu was computed because our sample was larger than 20. After applying the equation (3), 3.49 was the value obtained to Zu.
Since p-value>a (p-value=0.99, a=0.05), the null hypothesis H0=STCL1 <= STCL2 cannot be rejected. The cost estimation drivers identified by students in Phase 1 are assumed to be less than or equal to the cost estimation drivers identified by students in Phase 2. Additionally, p(x<=Z=0.00095), it means that the chance of error rejecting H0 is too high:0.999 (99.9%). However, when we estimate the common language effect size (U/(n1 * n2) = 0.30, this is the probability that a random cost estimation driver from Phase 1 is greater than a random cost estimation driver from Phase 2. Finally, H0 cannot be rejected.
The second Mann-Whitney test was applied to Systems Thinking Competence Level #3 reached by students. Phase 1 and Phase 2 were related with STCL1 and the second with STCL2. The sample size for the first phase and the second was n1 =54, and (n2 =41). The median to Phase 1, and Phase 2 was 2 and 3 respectively.
The U parameter was calculated for both phases. U1 = 1321.5, and U2 = 892.5. Hence U = 892.5 because the U2 with fewer scores is selected, in this case U2 was selected.
Zu was computed because our sample was larger than 20. After applying the equation, the value obtained to Zu was 1.61.
Since p-value>a (p-value=0.97, a=0.05), the null hypothesis H0=STCLi <= STCL2 cannot be rejected. The Systems Thinking Competence #3 reached by students at phase i are assumed to be less than or equal to the Systems Thinking Competence #3 reached by students at phase 2.
5. Discussion
Regarding RQi. There was just one month between phase one and phase two, despite those, a relevant and remarkable difference was detected. After students received a little training about cost estimation drivers, they were able to develop Concept Maps where more cost estimation drivers were included, and they were able to see more relationships, it means, they increased the level of Systems Thinking Competence #3 embedded into their Concept Maps.
Regarding RQ2. It is necessary to provide training or teaching where students get information about cost estimation drivers and this information can be internalized for each student in case they work alone and socialized in case they work in a team. During this time, students received information about what factors could impact project costs. The knowledge acquired allowed students to build Concept Maps with more cost estimation drivers included. These outcomes shown it is important include information about what Systems Thinking Competence is, and add information about the specific topic or area to be tackled.
A Mann-Whitney U-test was applied to Cost Estimation Drivers included in Concept Maps developed by students in both Phase i and Phase 2. The changes between these Phases is statistically significant. The null hypothesis H0=STCLi <= STCL2 cannot be rejected
An additional Mann-Whitney U-test was applied to Level of Systems Thinking Competence#3 reached by students when they develop their Concept Maps, in both Phase i and Phase 2, as a result, we can realize, the changes between Phase i and 2 were statistically significant. The null hypothesis H0=STCLi <= STCL2 cannot be rejected.
Limitations and threats. This research was applied to a limited sample of participants, which consisted of undergraduate and postgraduate students, where an optimal sample size wasn't calculated. Additionally, the sample is not heterogeneous, as a result, the outcomes cannot be generalized to different areas of engineering. Hence, the outcomes must be taken with caution, and the outcomes cannot be generalized.
6. Conclusions
This research has shed light on a specific Systems Thinking competence (STC #3). Particularly, it collected evidence about STC #3 owned by undergraduate and postgraduate students. Even when they did not know what a Systems Thinking Competence is.
Collecting this kind of information can be useful when Systems Thinking Competencies must be taught. In other words, before a teacher or trainer will teach Systems Thinking Competencies, it is recommended to apply an initial diagnostic test in order to identify the level of knowledge in each competence, after that, the results can be used in order to design a strategy to teach Systems Thinking Competencies. These actions could save training time, and reach desired objectives more quickly and more efficiently.
Outcomes indicated engineering students own some cost estimation knowledge. This can be understood because they own an engineer profile, and they are aware about aspects that have to be taken into account when a project is developed and when a cost has to be estimated. For instance, they identified aspects that imply time and money.
Outcomes obtained can be useful in order to design an educational strategy when the cost estimation topic is taught.
Additionally, this research has shown that despite engineering students not writing specific cost estimation's names, they identified them, hence just teaching specific cost estimation names will be required, and obviously, detailed theory about it.
Mann-Whitney U Test was useful in order to show that changes between phase 1 and phase 2 were
statistically significant.
References / Список литературы
[1] York S., Lavi R. et al. Applications of Systems Thinking in STEM Education. Journal of Chemical Education, vol. 96, issue 12, 2019, pp. 2742-2751.
[2] Maracha V. Systems thinking and collective problem solving practices. In Proc. of the V International Research and Practice Conference-Biennale on System Analysis in Economics, 2018, pp. 269-272.
[3] Amissah M., Gannon T., and Monat J. What is Systems Thinking? Expert Perspectives from the WPI Systems Thinking Colloquium. Systems, vol. 8, issue 1, 2020, 26 p.
[4] Valerdi R., Rouse W.B. When Systems Thinking is Not a Natural Act. In Proc. of the IEEE International Systems Conference, 2010, pp. 184-189.
[5] Boehm B.W., Abts C. et al. Software Cost Estimation with COCOMO II. Prentice Hall Press, 2000, 544 p.
[6] Richmond B. Systems thinking/system dynamics: Let's just get on with it. System Dynamics Review, vol. 10, issues 2-3, 1994, pp. 135-157.
[7] Mehrjerdi Y.Z. Quality function Deployment and its profitability engagement: A Systems Thinking perspective, International Journal of Quality & Reliability Management, vol. 28, issue 9, 2011, pp. 910928.
[8] Hebel M. Light Bulbs and Change: Systems Thinking and Organizational Learning of New Ventures. The Learning Organization, vol 14, issue 6, 2007, pp. 499-509.
[9] Sankaran S., Hou Tay B., Orr M. Managing Organizational Change by using Soft Systems Thinking in Action Research Projects. International Journal of Managing Projects in Business, vol. 2, issue 2, 2009, pp. 179-197.
[10] Miertschin S.L., Willis C.L. Using Concept Maps to navigate complex learning environments. Proceedings of the 8th Conference on Information Technology Education, 2007, pp. 175-184.
[11] Schwendimann B. Concept Mapping. In Encyclopedia of Science Education, Springer, 2015, pp. 198-202.
[12] Davies M. Concept Mapping, Mind Mapping and Argument Mapping: What are the Differences and Do They Matter? Higher Education, vol. 62, issue 3, 2011, pp. 279-301.
[13] Henderson C., Yerushalmix E. et al. Multi-Layered Concept Maps for the Analysis of Complex Interview
Data. Roundtable Discussion, Presented at the AAPT Physics Education Research Conference, 2003, 13 p.
[14] Kennedy D., Hyland A., Ryan N. Learning Outcomes and competencies. Introducing Bologna objectives and tools. B 2.3-3, 2019.
[15] Crawley E., Cameron B., Selva D. Systems architecture: Strategy and product development for complex systems. Pearson, 2015, 480 p.
[16] Lavi R., Dori Y.J. Systems Thinking of pre- and in-service science and engineering teachers. International Journal of Science Education, vol. 41, issue 2, 2018, pp. 248-279.
[17] Richmond B. Systems Thinking: Critical Thinking Skills for the 1990s and beyond. System Dynamics Review, vol. 9, issue 2, 1993, pp. 113-133.
[18] Valdés-Souto F., Naranjo-Albarrán L. Improving the Software Estimation Models Based on Functional Size through Validation of the Assumptions behind the Linear Regression and the Use of the Confidence Intervals When the Reference Database Presents a Wedge-Shape Form. Programming and Computer Software, vol. 47, issue 8, 2021, pp. 673-693.
[19] Durán M., Juárez-Ramírez R. et al. User Story Estimation Based on the Complexity Decomposition Using Bayesian Networks. Programming and Computer Software, vol. 46, issue 8, 2020, pp. 569-583 / Дуран М., Хуарес-Рамирес Р. и др. Оценка пользовательских историй на основе декомпозиции сложности с использованием байесовских сетей. Труды ИСП РАН, том 33, вып. 2, 2021 г., стр. 77-92. DOI: 10.15514/ISPRAS-2021-33(2)-4.
[20] Trendowicz A., Jeffery R. Constructive Cost Model - COCOMO. In Software Project Effort Estimation, Springer, 2014, pp. 277-293.
[21] Novak J.D., Musonda D. A Twelve-Year Longitudinal Study of Science Concept Learning. American Educational Research Journal, vol. 28, issue 1, 1991, pp. 117-153.
[22] Tripto J., Assaraf O.B.-Z., Amit M. Mapping What They Know: Concept Maps as an Effective Tool for Assessing Students' Systems Thinking. American Journal of Operations Research, vol. 3, issue 1A, 2013, pp. 245-258.
[23] White R., Gunstone R. Probing Understanding, Routledge, 1992, 208 p.
[24] Songer C.J., Mintzes J.J. Understanding Cellular Respiration: An Analysis of Conceptual Change in College Biology. Journal of Research in Science Teaching, vol. 31, issue 6, 1994, pp. 621-637.
[25] Turns J., Atman C.J., Adams R. Concept Maps for engineering education: A cognitively motivated tool supporting varied assessment functions. IEEE Transactions on Education, vol. 43, issue 2, 2000, pp. 164173.
[26] Novak J., Gowin D. Learning How to Learn. New York: Cambridge University Press, 1984, 216 p.
[27] Stewart M. Joined up thinking? Evaluating the use of concept-mapping to develop complex system learning. Assessment & Evaluation in Higher Education, vol. 37, issue 3, 2012, pp 349-368.
[28] Rye J.A., Rubba P.A. Scoring Concept Maps: An Expert Map-Based Scheme Weighted for Relationships. School Science and Mathematics, vol. 102, issue 1, 2002, pp. 33-44.
[29] 12 D. L Darmofal, D. H. Soderholm, and D. R. Brodeur. Using Concept Maps and Concept Questions to Enhance Conceptual Understanding. Frontiers in Education Conference, Boston, MA, November 6-9, 2002.
[30] McClure J.R., Sonak B., Suen H.K. Concept Map Assessment of Classroom Learning: Reliability, Validity, and Logistical Practicality. Journal of Research in Science Teaching, vol. 36, issue 4, 1999, pp. 475-492.
[31] Trochim W.M., Cabrera D.A. et al. Practical Challenges of Systems Thinking and Modeling in Public Health. American Journal of Public Health, vol. 96, issue 3, 2006, pp. 538-546.
[32] Raved L., Yarden A. Developing seventh grade students' Systems Thinking skills in the context of the human circulatory system. Frontiers in Public Health, vol. 2, 2014, article no. 260, 11 p.
[33] Odom A.L., Kelly P.V. Integrating Concept Mapping and the Learning Cycle to Teach Diffusion and Osmosis Concepts to High School Biology Students. Science Education, vol. 85, issue 6, 2001, pp. 615635.
[34] Markow P.G., Lonning R.A. Usefulness of Concept Maps in College Chemistry Laboratories: Students' Perceptions and Effects on Achievement. Journal of Research in Science Teaching, vol. 35, issue 9, 1998, pp. 1015-1029.
[35] Hu M., Shealy T. Methods for Measuring Systems Thinking: Differences Between Student Self-Assessment, Concept Maps Scores, and Cortical Activation During Tasks About Sustainability. In Proc. of the ASEE Annual Conference & Exposition. 2018, article id. 22718, 12 p.
[36] Brandstädter K., Harms U., GroBschedl J. Assessing System Thinking Through Different Concept-Mapping Practices. International Journal of Science Education, vol. 34, issue 14, 2012, pp. 2147-2170.
Information about authors / Информация об авторах
Jorge Rafael AGUILAR CISNEROS, Ph.D., Research Professor at UPAEP in the Department of Engineering. Research interests include Software Engineering and Software Processes.
Хорхе Рафаэль АГИЛАР СИСНЕРОС, кандидат наук, профессор-исследователь инженерного факультета. Область научных интересов включает программную инженерию и программные процессы.
Ricardo VALERDI, Ph.D., Distinguished Outreach Professor and Interim Department Head, Department of Systems & Industrial Engineering. His research focuses on Cost estimation, systems thinking, sports analytics.
Рикардо ВАЛЕРДИ, кандидат наук, профессор кафедры систем и промышленной инженерии. Его исследования сосредоточены на оценке затрат, системном мышлении, спортивной аналитике.
Brendan Patrick SULLIVAN, Ph.D., Assistant Professor at the University of Twente in the Department of Design, Production and Management (DPM). Research interests include intelligent control systems, decision analysis and changeable engineering systems.
Брендан Патрик САЛЛИВАН, кандидат наук, ассистент кафедры дизайна, производства и управления. Область научных интересов включает интеллектуальные системы управления, анализ решений и изменяемые инженерные системы.
