CC BY
30
COGNITIVE DIALOG GAMES AS COGNITIVE ASSISTANTS: TRACKING AND ADAPTING KNOWLEDGE AND INTERACTIONS IN STUDENT'S DIALOGS
Dr. Adem Karahoca, Bahcesehir University Engineering Faculty, Turkey E-mail: [email protected] Dr. Ilker Yengin, Institute of High Performance Computing, A*STAR, Turkey E-mail: [email protected] Dr. Dilek Karahoca, Bahcesehir University Health Science Faculty, Turkey E-mail: [email protected]
A R T I C L E I N F O
Original Research Received: January, 27.2018. Revised: March, 06.2018. Accepted: March, 14.2018. doi:10.5937/ijcrsee1801045K
UDK
159.922.72.072-057.874
Keywords:
Student modeling, Cognitive Modeling, Cognitive Assistants, Cognitive dialog game, MOOCS.
A B S T R A C T
This study introduces a system design in a form of cognitive dialog game (DiaCog) to support pedagogical factors and student learning model ideas. The purpose of the study is to describe how such a design achieves tracking and adapting students' knowledge and mastery learning levels as a cognitive assistant. Also, this study shows alternative ways for supporting intelligent personal learning, tutoring systems, and MOOCS. This paper explains method called DiaCog that uses structure for students' thinking in an online dialog by tracking student's level of learning/knowledge status. The methodology of computing is the semantic that match between students' interactions in a dialog. By this way it informs DiaCog's learner model to inform the pedagogical model. Semantic fingerprint matching method of DiaCog allows making comparisons with expert knowledge to detect students' mastery levels in learning. The paper concludes with the DiaCog tool and methodologies that used for intelligent cognitive assistant design to implement pedagogical and learner model to track and adapt students' learning. Finally, this paper discusses future improvements and planned experimental set up to advance the techniques introduced in DiaCog design.
1. INTRODUCTION
Intelligent cognitive assistants and tools can be implemented in e-learning platforms and massive open online source courses systems (MOOCS) to reduce the educational costs and improving life quality of people and to address the educational needs of people who are unable to leave their home due to their care needs. Although such users can easily access to current MOOCS platforms, they may not fully benefit from them since there are some technical challenges to overcome. For example, MOOCS platforms are usually criticized
Corresponding Author
Dr. Adem Karahoca, Bahcesehir University Engineering Faculty, Turkey
E-mail: [email protected] @©@©
This work is licensed under a Creative Commons Attribution - NonCommercial - NoDerivs 4.0. The article is published with Open Access at www.ijcrsee.com
© 2018 IJCRSEE. All rights reserved.
about failing to provide a social environment that enables having constructive cognitive feedback opportunities in a sustained engagement during the online courses (Nkuyub-watsi, 2013; Kop et al., 2011; Zapata, 2010).
Discussion forums are used where students have most of the social interaction to help each other and to discuss ideas related to the learning topics in MOOCS (Ezen-Can et al., 2015). Discussing ideas on learning topic have positive impacts on students learning and building knowledge in online platforms (Palmer et al., 2008).
On the other hand, we observe that discussion forums bring several issues in learning despite their positive benefits on learning. For example, a poor designed and implemented discussion forum activity may not support to discuss at all. The engagement of students may be very superficial (Bain, 2011; Hawkey, 2003; Thomas, 2002). Thus, discussion on online platforms may provide support for learning but it is not guaranteed that all the participants of the dialog may join to the ongoing
discussion or the discussion is deep and meaningful. There is still have more to explore in students' dialog in technology supported applications. Especially, we need to know which students actively and meaningfully participate to the dialogs and which of them learn from the dialog as the knowledge building activities emerges in ongoing dialogs.
Using new cognitive assistants can be supported with contemporary dialog technologies. By this way, it may be possible to improve students' positive learning results by tracking meaningful dialog participation and interaction. To enable these functions, it is possible to design modern applications that track and adapt knowledge level of students. It may allow us to closely understand students' dialog-based learning interactions.
To address the need of designing new applications as described here, this study discusses the design of a modern learning tool that is called as DiaCog. The remaining of the paper presents how DiaCog design enables alternative dialogic interactions and modeling learning processes in a meaningful way.
2. MATERIALS AND METHODS
2.1. Designing Interaction and Interface
Table 1 shows a part of a hypothetical dialog. This dialogue is an example of interaction between players in a typical dialogue game. Dialog may go in a way that the players may reply to previous entries so that there is no linear order in the dialog. Therefore, the ongoing dialog cannot be read as reading a page of a book in a top to down direction because the events are non-linear. The rounds are indicated by round numbers in the first bracket and the dialog moves are coded in the second bracket at the end of players' entries.
Table 1. An Example Dialog
START: Copyright laws and policy could be better aligned with the interests of both consumers and copyright holders.
Player A: I agree, although the Copyright Act attempts to balance culture and commerce through exclusive incentive models and fair use defenses, the law just doesn't seem to be keeping up with the way end users, developers, and content creators operate in the digital sphere [Round 1]. [Agree].
Player D: For instance, current copyright laws are inadequate for the digital age anyway, some legal experts says "Most of it was written more than a quarter century ago," [Round 1], [Support],
Player A: For example, we see it all the time on YouTube: people communicating through shared content without permissions. [Round 3], [Agree],
Player B: Is it the case that, the Copyright Act protects "original expression," but what is considered "original"? [Round 1], [Question],
Player C: I read that, a work can only be original if it is the result of independent creative effort. It will not be original if it has been copied from something that already exists. If it is similar to something that a heady exists but there has been no copying from the existing work either directly or indirectly, then it may be original. [Round 1], [Inform].
Player B: I am not sure about that, copyright protection should not depend on the artistic quality or merit of a work [Round 2], [Challenge].
Player C: I read that, the term "original" also involves a test of substantiality - literary, dramatic, musical and artistic works will not be original if there has not been sufficient skill and labor expended in then creation. But, sometimes significant investment of resources without significant intellectual input can still count as sufficient skill and labor. [Round 2], [Support],
DiaCog has essential functions such as a)interface interactions tied to the meaning that the users wish to communicate, b)dialog roles and specific rooms, c)structured moves categorized as interface elements, and d)dia-log rules in order to carry the user's goals (e.g. turn taking). These functions create the basic environment for successful collaborative argumentations that teachers may apply dialog-
based learning scenarios to support students in practicing their argumentation skills and creative thinking skills.
Figure 1 illustrates the typical design of DiaCog's dialog game window. In Figure 1, participants are required to select moves from the list of available speech acts (see Label # 1) and related openers (see Label#2) and type a free text of their response into the text box (see Label #3). After finishing building the expressions, participants submit their replies once they click the "Act" button ("See Label #4).
Figure 1. The typical design of DiaCog's dialog game
2.2. Design of Pedagogical and Learner Models in DiaCog
Student state tracking and adaptation is a common application in intelligent tutoring system (ITS), which can guide us for the design
decisions for cognitive assistant implementations. In the context of modern ITS design, the educational needs of students are tracked and adapted by domain model, pedagogical model and learner model. Domain model (DM) tracks the set of skills, atomic components of knowledge objects and content-based strategies. Learner model(LM) tracks cognitive, motivational, affective and other psychological states of learners during the tutoring activities. Pedagogical model (PM) decides tutoring strategies and course of action by taking input from LM and DM.
Figure 2. Presentation and relations of LM, PM. adn DM
As Figure 2 demonstrates, since LM and DM feed PM, it is a better to decide the PM
design first before deciding for the LM design (DM will not be discussed here since it may dramatically change due to the topic and learning domain which is difficult to comprehend in the study). Following section describes the pedagogical strategies for designing PM.
2.3. Pedagogical Model
To tackle the design questions of selecting a suitable learning model for MOOC, we form relationships between domain model (DM), pedagogical model (PM) and learning model (LM). Then, we need to analyze the educational needs of typical students to build PM. For PM design, we list learning theories and values to track in Table 2 and Table 3 based on an extended version of Sottilare et al (2013).
Table 2. Followed theories for Pedagogical Model
Self Regulation Theory: Goal Setting, Self- Assessment, Self Reflection
Difficulty Proximity Progress Effort
Effectiveness
Perceived success (strength /weakness) Personal Interest Improvement and Traits
Social and
Constructivist
Theory:
Recruitment, Degree of Freedom, Direction. Critical Features, Frustration control. Demonstration, Teachable - peer asents.
Attention
Interest (General) &Interest (Specific)
Goal setting - Goal commitment Ability for Degree of Freedom Task (domain) difficulty On'off task behavior Affective States and traits Session difficulty Patterns of interaction
Motivation (Self
Determination)
Theory
Autonomy. Relevance, Competence
Style of language
Type of feedback (reaction)
Positive Trust
Satisfaction
Free decision making
Preferences and interest
Value
(Gasne & Deci, 2005)
Interest (Situational) Theory
Attention Levels of learning Goals
Table 3. Followed values and variables for Pedagogical Model
Item Rating
Cognitive
Trails
load and
Error/Misconceptions
item
Success rates
difficulty
Time spent {latency)
^ Inte^ abilities /prior knowledge
As given in Figure 2, considering the roles of different models in system design, cognitive assistant adaptation and tracking related modules may take large number of variables to adaptn in theory. However, in practice, the numbers of variables are relatively small due to scale up issues.
PM can be designed by using different pedagogical strategies as presented in Table 3. In our design, we selected a purposeful PM strategy that is fitting the nature of the pedagogical needs of students' that are learning by engaging interactive dialogs in discussions. For this purpose, specifically considering the dialog-based model of Di-aCog, selected PM should track self-regulation and social constructivist educational needs (and linked values) since these are suitable for learning by dialogs strategies in where the learner arrives at solutions through active participation in dialog interactions. In Dia-Cog, tracking and adaptation of these values are achieved by LM, which feed the PM. The design process of LM and methods that LM uses explained in the following section.
2.4. Learner Model
Learner models usually infer the diagnosis like a feature to represent the current knowledge state of the students including learning difficulties and misconceptions (Co-nati, 2002). The learner model is dynamically updated as students build knowledge thus the model represents about latest understanding of the students (Bull, 2004). Learner models describes the (1) cognitive processes that governs students' interactions, (2) knowledge gap levels between the student and expert, (3) students' behavior patterns, (4) students' characteristics and profiles (Webb, et al., 2001). Using predictive learner models, it is possible to have an insight into the nature of the dia-logical interactions and learning. Using these models, each reasoning step can be traced, and the misconceptions can be detected (Johnson and Taatgen, 2005).
DiaCog uses a dialog-based learning strategy as described and modeled in PM. Thus, the learner model also should be dialog based to match with PM. In dialog based learning models, analyzing the knowledge level and learning status in ongoing dialog of students could model students learning. Basic natural language dialogue analysis techniques such as calculating student turns, number of words used in dialog turns, percentage of words per turns by students are indication of learning in an online platform (Core, 2003; Rose et al., 2003; Huang et al., 2015). More sophisticated techniques such as analyzing the different types of speech acts or dialog moves, structure of discourse and contextual meaning are also possible for tracking the learning in dialog (Sardareh et al., 2014; Prylipko et al., 2014; Vail, 2014; Rotaru, et al., 2006).
The power of learner model may change due to the sophistication of linguistic techniques that are applicable for tracking modules in MOOCS (Forbes-Riley, 2007). For instance, some techniques are easy to apply but not meaningfully informative (e.g. turn counts, dialog length) while others provide much worthwhile data for modeling (e.g. speech acts) (Bernsen, et al., 2012). Some of the techniques are well informative and largely available in the ITS literature to apply in for MOOCS but they require more effort like labeling dialog acts or designing computational algorithms for dialog analysis and labeling of corpus.
In DiaCog, we aim to tackle the issue of understanding deeper pragmatic properties of dialog by providing combination of several sophisticated methodologies that enable analyz-
ing the dialog yielding stronger learner models. Using combining techniques in DiaCog's proposed LM design it is possible to have sophisticated tracking functions in MOOCS systems. In DiaCog, LM may be improved further for additional applications for more advanced automatic adaptive MOOC functions that can provide more specific personalized and tailored instruction in addition to the pedagogical strategies that PM delivers as discussed above.
LM strategies for DiaCog can be constructed using machine-learning approaches (Webb, et al., 2001). However, requirement of explicitly labeled data is one of the biggest challenges of machine learning approaches for learner modeling and may be challenging for DiaCog's LM design. Because the correct labels may not be available from simple observation of dialogs, these labels may require a manual coding or an algorithm for label clustering (Fereday et al., 2006). After determining correct labels, speech acts may represent deeper pragmatic properties in dialog, which informs the PM about students' status. Fortunately, DiaCog interface provides a partially shortcut solution for the issue of manual labeling of speech acts. Since DiaCog interface requires users to indicate their intention using speech acts specially designed and labelled as "dialog moves" in the dialog interaction interface, DiaCog easily understand and categorize the speech acts when users build their expressions in ongoing dialogs (Yengin and Lazarevic, 2014).
Tracking students' learning (knowledge status) and comparing with expert knowledge universe handled in DiaCog as illustrated in Figure 3. To track students' learning and knowledge status, DiaCog compute the semantic match between all ideal student interactions in a dialog and compares it with an expert knowledge domain map encoded. Comparing students' current state of learning and knowledge of content with expert knowledge rule spaces is mapped in a QMA-TRIX, which enables learning-knowledge level for tracking students (Lee and Sawaki, 2009). The use of QMATRIX is relatively well-known technique in rule space learner models such as the "Additive Factor Model" (AFM) (Li et al., 2011) that is using the "Item Response Theory" (Embretson and Reise, 2013). AFM is good model for detection of students' prior knowledge, which can be used to predict later performances. DiaCog's LM design uses same principles for this purpose.
Figure 3. LM Design for DiaCog
To map student knowledge and compare it with expert knowledge in QMATRIX, DiaCog needs to know whether students have similar knowledge structures with the expert domain knowledge. In simple words, DiaCog needs to compare the student dialogs with the expert knowledge. If there is a match between student knowledge and expert knowledge; it is mapped in QMATRIX using variables that show the student knows a knowledge or content object. If there is no match, this is mapped, as the student has no knowledge. If there is a match leading a possible misconception, this is mapped as misconception in QMATRIX.
To understand if there is any matching or not, DiaCog must analyze the student input texts in ongoing dialogs. This function is carried in DiaCog by a "semantic matching" engine that measures semantic relatedness using a direct measure which is a stronger and sophisticated technique rather than words analysis. DiaCog "semantic matching" engine uses "cortical.io Retina API" services to make direct semantic comparison of the meanings stored in students' dialogs. Using semantic relations, DiaCog is capable to measure whether participants of dialogs talk (know) about on the same concepts with the expert domain knowledge universe by applying natural-language processing techniques based on the distributed representations of text segments grounded in a neuro-computational model of semantics (Corticalio, 2015; Chi, 2009). This allows DiaCog to interpret students' conceptual understanding and their meaning closeness. Figure 4 shows a conceptual dialog mapping of students.
Figure 4. A conceptual dialog mapping of students
According to the example QMATRIX in Figure 4, student knowledge of apple belongs to the knowledge space of fruits. On the other hand, the expert knowledge domain requires student to know/learn the apple as a concept of computer brand Apple (just for giving an example purpose). To detect and understand this student's knowledge level (concept of apple) and the potential misconception, DiaCog's LM should match the expert knowledge space with students' knowledge level. For this purpose, semantic matching engine calculates the distance scores for these knowledge objects (apple as fruit and Apple as computer brand) and compare the scores to see how close they are to each other. If the closeness score is less, this means student is very close to expert knowledge space and well knows the knowledge object. If there is a great distance than that means student doesn't know the knowledge object. If there is an overlap like fruit domain and computer, semantic engine calculates the overlap. The degree of distance inexpert and student semantic relatedness scores are calculated using to the median score of overall population score in the system. Finally, if the overlap is not close to expert knowledge space, then it is labeled as misconception and encoded in this space.
3. RESULTS
In this study, design of DiaCog application aimed for implementation as cognitive assistant in MOOCS and discussed DiaCog's functions for enabling tracking students' knowledge levels and behavior in dialogs. The suggested pedagogical model in DiaCog explains strategies fitting the nature of learning with dialogs in discussions. Matching the
pedagogical model, DiaCog' learner model is demonstrated how to use the semantic match technique to compare student knowledge and learning with an expert knowledge domain map to enable tracking learners' knowledge when they engage discussion in cognitive assistant supported MOOCS. These design decisions and techniques for pedagogical and learning models let DiaCog to be an alternative module as cognitive assistants for tracking expert knowledge and detect student mastery levels in learning.
4. DISCUSSIONS
DiaCog's special design and analytical functions opens a door for learner modeling enabling knowledge tracking in dialogs. In future, we plan to train a student and learning model using linguistic features derived from DiaCog by applying machine-learning algorithms to create a model of students thinking processes in dialogs that also can be applied to simulate typical thinking patterns in future dialogs then. Using such models as cognitive assistant implementations in MOCCS may have adaptive pedagogical strategies to the students' state to improve learning experience by providing additional feedback or course materials dynamically to the situations. Also predictive models based on machine learning algorithms may enable MOOCS to understand the users' knowledge level positions in different learning scenarios. In addition, cognitive assistants enabled MOOCS can determine future skill levels of students so it can have a control and guiding mechanism for the course of learning through interactions in dialog or other content.
In future, we are also planning to examine how well different student models perform and generalize with different user populations using MOOCS. We will test the effectiveness of the DiaCog on learning by conducting experiments to validate the success of implementation and pedagogic usefulness of DiaCog in real life.
5. CONCLUSIONS
In conclusion, this paper showed and explained the design of cognitive dialog game (DiaCog) as cognitive assistants. The tracking and adapting abilities of DiaCog as a cognitive assistant was illustrated. The methodology of computing the semantic that match between student interactions in a dialog is used
to inform DiaCog's learner model. Semantic fingerprint matching method used to make comparisons with expert knowledge in order to detect student mastery levels in learning.
As a result, DiaCog tool and applied methodologies can be used for implementing pedagogical and learner modeling to track and adept students' learning.
Finally, this paper discussed the improvements and possible experimental designs to advance the techniques.
ACKNOWLEDGEMENTS
This research is made possible through the help and suggestions of former members of A*STAR's IHPC's "Collaborative Thinking Technologies" research group Dr. S. Feller, and Dr. J. Herberg.
Conflict of interests
The authors declare no conflict of interest.
REFERENCES
Bain, Y. C. (2011). Learning through online discussion: a framework evidenced in learners' interactions, Research in Learning Technology, Vol 19, 29-42, DOI: 10.3402/rlt.v19s1/7779 Bernsen, N. O., Dybkjsr, H., & Dybkjsr, L. (2012). Designing interactive speech systems: From first ideas to user testing. Springer Science & Business Media. https://goo.gl/PZDM5v Bull, S. (2004). Supporting learning with open learner models. Planning, 29(14), 1. https://www.researchgate.net/profile/Su-san_Bull2/publication/228888082_Support-ing_learning_with_open_learner_models/ links/0c96052b4861a5d449000000/Supporting-learning-with-open-learner-models.pdf Chi, M. T. (2009). Active-constructive-interactive: A conceptual framework for differentiating learning activities. Topics in cognitive science, 1(1), 73-105. https://doi.org/10.1111/j.1756-8765.2008.01005.x Conati, C., Gertner, A., & Vanlehn, K. (2002). Using Bayesian networks to manage uncertainty in student modeling. User modeling and user-adapted interaction, 12(4), 371-417. https://link.spring-er.com/article/10.1023/A:1021258506583 Core, M. G., Moore, J. D., & Zinn, C. (2003, April). The role of initiative in tutorial dialogue. In Proceedings of the tenth conference on European chapter of the Association for Computational Linguistics-Volume 1 (pp. 67-74). Association for Computational Linguistics. https://aclanthol-ogy.info/pdf/E/E03/E03-1072.pdf Corticalio (2015). Sparse distributed representations. http://www.cortical.io/technology_representa-tions.html
Embretson, S. E., & Reise, S. P. (2013). Item response theory. Psychology Press. https://goo.gl/Euq1sm Ezen-Can, A., Boyer, K. E., Kellogg, S., & Booth, S. (2015, March). Unsupervised modeling for understanding MOOC discussion forums: a learning analytics approach. In Proceedings of the fifth international conference on learning analytics and knowledge (pp. 146-150). ACM. https://www.intellimedia.ncsu.edu/wp-content/ uploads/ezen-can-lak-2015.pdf Fereday, J., & Muir-Cochrane, E. (2006). Demonstrating rigor using thematic analysis: A hybrid approach of inductive and deductive coding and theme development. International journal of qualitative methods, 5(1), 80-92. http://journals.sagepub. com/doi/abs/10.1177/160940690600500107 Forbes-Riley, K., Litman, D. J., Purandare, A., Rotaru, M., & Tetreault, J. (2007, June). Comparing linguistic features for modeling learning in computer tutoring. In Aied (Vol. 158, pp. 270-277). https://www.cs.rochester.edu/~tetreaul/aied07-ios2-final.pdf
Gagné, M., & Deci, E. L. (2005). Self-determination theory and work motivation. Journal of Organizational behavior, 26(4), 331-362. https://doi. org/10.1002/job.322 Hawkey, K. (2003). Social constructivism and asynchronous text-based discussion: A case study with trainee teachers. Education and Information Technologies, 8(2), 165-177. https://link. springer.com/article/10.1023/A:1024558414766 Huang, Y. M., & Liang, T. H. (2015). A technique for tracking the reading rate to identify the e-book reading behaviors and comprehension outcomes of elementary school students. British Journal of Educational Technology, 46(4), 864-876. https:// doi.org/10.1111/bjet.12182 Johnson, A., & Taatgen, N. (2005). User modeling. The handbook of human factors in web design, 424438. https://pdfs.semanticscholar.org/83d7/a8c-7faf225b9ff8172693d1faa7e3cf45f77.pdf Kolb, D.A. (1985). Learning-style inventory: Self-scoring inventory and interpretation book-let (2nd ed.). Boston: McBer & Co. Kop, R., Fournier, H., & Mak, J. S. F. (2011). A pedagogy of abundance or a pedagogy to support human beings? Participant support on massive open online courses. The International Review of Research in Open and Distributed Learning, 12(7), 74-93. http://dx.doi.org/10.19173/irrodl. v12i7.1041
Lee, Y. W., & Sawaki, Y. (2009). Cognitive diagnosis and Q-matrices in language assessment, Journal Language Assessment Quarterly, Vol. 6, 169171, doi.org/10.1080/15434300903059598 Li, N., Cohen, W. W., Koedinger, K. R., & Matsuda, N. (2011, July). A Machine Learning Approach for Automatic Student Model Discovery. In EDM (pp. 31-40). https://files.eric.ed.gov/fulltext/ ED537453.pdf#page=44 Nkuyubwatsi, B. (2013, October). Evaluation of massive open online courses (MOOCs) from the learner's perspective. In European Conference on e-Learning (p. 340). Academic Conferences International Limited. https://search.proquest. com/openview/2616f896b4618dba39f72b248b d6d47a/1?pq-origsite=gscholar&cbl=1796419 Palmer, S., Holt, D., & Bray, S. (2008). Does the discussion help? The impact of a formally assessed online discussion on final student results.
British Journal of Educational Technology, 39(5), 847-858. https://doi.org/10.1111/j.1467-8535.2007.00780.x Prylipko, D., Rosner, D., Siegert, I., Günther, S., Friesen, R., Haase, M., ... & Wendemuth, A. (2014). Analysis of significant dialog events in realistic human-computer interaction. Journal on Multimodal User Interfaces, 8(1), 75-86. https:// link.springer.com/article/10.1007/s12193-013-0144-x
Rosé, C. P., Bhembe, D., Siler, S., Srivastava, R., & VanLehn, K. (2003). The role of why questions in effective human tutoring. In Proceedings of the 11th International Conference on AI in Education (pp. 55-62). https://www. researchgate.net/profile/Stephanie_Siler/ publication/243787583_The_Role_of_Why_ Questions_in_Effective_Human_Tutoring/ links/00b4953bd580c7117d000000/The-Role-of-Why-Questions-in-Effective-Human-Tutor-ing.pdf
Rosenberg, M., & Burkert, A. (2015). Learning Styles and their Effect on Learning and Teaching.
Forschende Fachdidaktik, 103-130. https://goo. gl/qMcgYh
Rotaru, M., & Litman, D. J. (2006, July). Exploiting discourse structure for spoken dialogue performance analysis. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (pp. 85-93). Association for Computational Linguistics. http://www. aclweb.org/anthology/W06-1611 Sardareh, S.A.; Aghabozorgi, S.; Dutt, A. (2014). Reflective dialogues and students' problem solving ability analysis using clustering. In: Proceedings of the 3rd International Conference on Computer Engineering & Mathematical Sciences (IC-CEMS 2014), 04-05 Dec 2014, Langkawi, Malaysia. http://eprints.um.edu.my/12988/ Soloman, B. A., & Felder, R. M. (2014). Index of Learning Styles Questionnaire [Internet]. North Carolina State University http://www.engr.ncsu. edu/learningstyles/ilsweb.html Sottilare, R. A., Graesser, A., Hu, X., & Holden, H. (Eds.). (2013). Design recommendations for intelligent tutoring systems: Volume 1-learner modeling (Vol. 1). US Army Research Laboratory. https://goo.gl/8HGjp4 Thomas, M. J. (2002). Learning within incoherent structures: The space of online discussion forums. Journal of Computer Assisted Learning, 18(3), 351-366. https://doi.org/10.1046Zj.0266-4909.2002.03800.x Vail, A. K., & Boyer, K. E. (2014, June). Identifying effective moves in tutoring: On the refinement of dialogue act annotation schemes. InInternational Conference on Intelligent Tutoring Systems (pp. 199-209). Springer, Cham. https://link.springer. com/chapter/10.1007/978-3-319-07221-0_24 Webb, G. I., Pazzani, M. J., & Billsus, D. (2001). Machine learning for user modeling. User modeling and user-adapted interaction, 11(1-2), 1929. https://link.springer.com/article/10.1023 /A:1011117102175 Yengin, I., & Lazarevic, B. (2014). The DiaCog: A Prototype Tool for Visualizing Online Dialog Games' Interactions. Research in Higher Education Journal, 25. https://eric.ed.gov/?id=EJ1055317 Zapata, M. (2010). Estrategias de evaluación de competencias en entornos virtuales de aprendi-zaje.
RED. Revista de Educación a Distancia. Sección de Docencia Universitaria en la Sociedad del Conocimiento. Número 1. Consultado https://www.um.es/ead/reddusc/1/eval_compet. pdf
