CC BY
13
ВЕСТНИК ТОМСКОГО ГОСУДАРСТВЕННОГО УНИВЕРСИТЕТА
2019 Управление, вычислительная техника и информатика № 49
УДК 004.891
DOI: 10.17223/19988605/49/12
A.E. Yankovskaya, I.S. Levin, I.L. Fuks MIXED DIAGNOSTIC TESTS STRUCTURE FOR LEARNING AND TESTING SYSTEMS
A new method of increasing the efficiency of students' educational activity is proposed. The method is based on the creation of learning trajectories, formed in the process of applying the mixed diagnostic tests. The learning trajectories are constructed by using specific mixed diagnostic tests, which are an integration of known unconditional and conditional tests. The mixed tests are used as in blended education and training, and in massive open online courses. The mixed testing provides the monitoring of the students' knowledge, professional and personal skills and abilities, as well as constructing learning trajectories for each student. The paper presents the innovative interdisciplinary technique for construction of the mixed diagnostic tests.
Keywords: blended education and training; massive open online courses; learning trajectory; tree of mixed diagnostic tests; interdisciplinary testing.
The search for ways to improve the effectiveness of teaching is a well-known educational problem. This task becomes even more important when the teacher disappears from the class at certain stages of the learning process. It is this problem that arose due to the emergence of blended education [1], which is an integration of the e-learning and the traditional learning [2]. Technologies of blended education very naturally fit into the concept of Mass Open Online Courses (MOOC) [3, 4], which is based on the maximum contribution of the teacher at the stage of course development and the almost complete exclusion of the teacher during the course. Today, the blended learning becomes popular both in education and in industry. The number of MOOCs implemented on the base of different educational platforms is growing. The number of people wishing to study at MOOC grows.
One of the known challengers in higher education is developing mechanisms for effective teaching of large groups of students [5]. A simple example of the blended training is a learning that uses various of resources and methods. It has to be noted that despite the wide integration of information and communication technologies (ICT) (including the technologies of artificial intelligent) the blended education are not used enough. Developing the software systems supporting the blended training is highly time and cost consuming task [6, 7].
Obviously, the students with different abilities have diverse preferences in learning and different achievements. A qualitative research [8] a specific questionnaire was used in order to study attitudes, priorities, preferences of university students. A number of the initial students' characteristics were taken into account. Among them: initial knowledge, skills, and learning experience. Taking into account the students' particular interests and capacities should make the learning process more effective. Any student of the high school, should be able to obtain the proper solution of his specific educational and industrial tasks. Moreover, he/she should understand very clearly why and how he/she will be able to utilize the obtained knowledge, professional and personal skills, and abilities to achieve personal goals.
In [1], Bliuc, et al. identify three different ways for defining the blended learning. The blended learning can be considered as a combination of: 1) web based technology models, 2) various pedagogical approaches, 3) emerging educational technologies with a face-to-face training. A comprehensive review of computer-based assessment methods including emergent blended learning is presented in [9].
The concept of the mixed diagnostic tests (MDT) was introduced in [10]. The mixed diagnostic testing is a compromise between unconditional and conditional components, which expediently to use in blended education and training. Today it is one of the popular and useful means of learning assessment. A number of innovative assessment technologies were developed on the base of the MDT [11, 12].
Our paper is a development of the classic MDT [Ibid.]. We propose to use MDT for creating various learning trajectories of the educational process. It provides a highly desirable flexibility in choosing different learning trajectories. Such property is highly appreciated by students.
A number of problems have to be solved in order to implement the MDT approach effectively: а) offering students the instrument for designing their own learning trajectories, b) providing interactions between the teacher and a student, and c) increasing accessibility of information. We believe that the proposed MDT based approach will result in growing efficiency of the educational process.
1. Scientific Background
Our society is a highly dynamic society. Its intensive transformation defines new challenges for peoples. Today students and graduates have to be ready to solve various technological and educational problems [5]. In such complicate situation, the blended learning seems to be a very appropriate since the blended learning enables developing the range of emerging skills and competences.
In [10] the MDT as a new approach of intelligent systems design has been proposed. The approach uses pattern recognition methods. An implementation of the MDT in intelligent learning and training systems was presented [11].
The main achievement of the MDT is its contribution to solving the problem of student motivation. The majority of the available testing methods doesn't motivate students sufficiently. The MDT approach provides high flexibility of the procedure of creating learning trajectories. According to our hypothesis, increasing flexibility in choosing leaning trajectories allows to motivate students more and, as the result, improve the efficiency of the learning process.
We assume that a curriculum of any academic discipline may be represented as a specific ontology's structure comprising a number of content blocks. We associate each of the content blocks with a specific didactic unit. A set of didactic units can be chosen either arbitrary, or depending on the students' abilities demonstrated in studying previous didactic units. In this way, a set of interrelated disciplines to be studied can be presented. Moreover, for each student, the study of specific didactic units can be represented as the sequence of independent content blocks. The essence of the proposed approach is the fact that the MDT is namely the structure comprising all the content blocks.
2. Basic Concepts and Definitions
The main concepts used in our paper are the following [11]:
Respondent (a person/student participating in a testing process).
Diagnostic test (a test, providing to define the level of the respondent's knowledge and skills in comparison with a predetermined level).
Mixed diagnostic test (a diagnostic test comprising both unconditional and conditional components).
Unconditional component of mixed diagnostic test (UCMDT) (a portion of the test that includes tasks undependable on previous test results. UCMDT allows the concurrent representation of constituent tasks).
Conditional component of mixed diagnostic test (CCMDT) (a portion of the test that includes tasks, sequence of which depends on previous test results. Tasks included in CCMDT are called conditional tasks).
MDT tree (a structure representing relationships between separate blocks (tasks) of the MDT. The root node is associated with the unconditional component of MDT. Each of the remaining nodes is associated with either unconditional or one of the conditional tasks. The MDT tree edges define connections between nodes).
3. Formulation of the Problem
In our paper, we demonstrate a method for constructing a mixed test, that can be used as a pretest which is run before the course. The pretest is based on a number of prerequisites. The main idea of such
a test is an assessing the initial level of the students' knowledge. The pretest is intended to clarify the level of understanding different disciplines, which are far away from each other. This fact makes the using of MDT unconditional components highly desirable. Such components, being isolated from each other during the testing, contribute their own weight into the total value of estimated knowledge. We call such a test as MDT
with extended structure of the unconditional component.
3.1. Structure of the Test Tree
The structure of the mixed diagnostic tree is introduced in [12]. The tree comprises a set of unconditional and conditional components intended for using during the learning a specific subject of the course. As a continuation of the study [12], we extend the structure of the diagnostic tree by increasing a number of possible directions. Such extension enables to form the strategy of learning different disciplines.
Next, we introduce a concept of the basic structure and a conditional fragment of the structure in order to represent the MDT structure in compact form.
The basic MDT structure is shown in Fig. 1. The structure comprises one unconditional component and a number of conditional tasks connected with this component. Conditional fragment of the MDT tree is a conditional task with a number of other conditional tasks connected with the fragment.
U
q^Gp
C4
C
C5
CD
j
CD"!
C6
L8
T
)
3
GD
C
C
q+1
K
L
q+4
L
q+6
)
)
C
q+3
L
q+5
j
C
L
q+8
)
Fig. 1. Basic structure of MDT (U is unconditional component of the MDT; Ci is i-th conditional task, where i £ {1, 2, ..., 6, q, q + 1, q + 2, q + 3}; Ljisj-th conditional task being a leaf of the tree, j £{7, 8, 9, q + 4, ..., q + 8})
We will use a minimized representation of the basic structure shown in Fig. 2a and a minimized representation of the conditional fragment shown in Fig. 2b.
T
Cp
nr
b
Fig. 2. Minimized representation: a - the basic MDT structure; b - the conditional fragment of the MDT tree
A number of extended MDT tree structures can be constructed.
I. An extended tree structure including a connection of the basic structure with t other basic structures, which are not interconnected as shown in Fig. 3a.
a
II. An extended tree structure including connections between t basic structures, which are not interconnected, but have outputs connected with the input of another basic structure, as shown in Fig. 3b.
III. An extended tree structure including connections between t basic structures, which are not interconnected, but have outputs connected with the input of conditional component of the structure, as shown in Fig. 3c.
a b c
Fig. 3. The extended MDT tree structure: a - the first type; b - the second type; c - the third type
Let us move to a number of examples of the using above extended structures of MDT trees in learning specific disciplines and assessment of student achievements.
1. The strong prerequisite for learning math is a certain level of knowledge of high school math (Fig. 4). The initial part of the MDT indicates whether student is ready for the perception and understanding the course content. In general, the MDT can be used for evaluating students' achievements in learning math.
Fig. 4. The extended MDT tree structure for mathematical courses
2. Learning the BSc course "Introduction to Algorithms" depends on students' prior knowledge of math models, discrete math and a number of initial programming skills (Fig. 5). Evaluating such requirements combined with formulating recommendations for learning additional topics enables decreasing the time required for learning a new subject.
Fig. 5. The extended MDT tree structure for the courses "Introduction to Algorithms"
f Mathematical j V Analysis J
Fig. 6. The extended MDT tree structure for the courses "Optimization methods"
3. Course "Optimization methods" is the mandatory course for students studying disciplines connected to the applied math (Fig. 6). For the successful learning of the course, a deep knowledge in calculus, algebra, geometry, probability theory, discrete math is required. The MDT allows to detect the missing topics of content in students' knowledge and, in the case the absence of such topics, to provide the permission to start leaning a new course.
4. Constructing Data Structures of the MDT Tree Nodes
4.1. In this section, the data structures of the unconditional MDT component is discussed
4.1.1. For each unconditional test component, a coefficients matrix of the correct answers has to be constructed. The matrix comprises weighed coefficients of questions, for simplicity called weights (Table 1) [12].
T a b l e 1
Matrix of coefficients for one of the unconditional test component
1 2 J k weight
1 an a12 a\k W1
2 a21 a22 a2j a2k W2
i a,1 at 2 aij aik Wi
n an1 an2 anj ank Wn
Rows of the matrix are associated with test tasks proposed to the respondent while he passes a test component, columns of the matrix correspond to a number of respondent's answers, n is the number of questions, k is the maximum number of possible answers. The weight Wi of the answer has to satisfy the condition 0 < wi < 1 for any i.
For test tasks of so-called closed type, with the only possible answer, k is equal to the number of pos-
m
sible answers. For test tasks of the closed type with multiple answer k = £ C'm, m is the number answers, Clm
i=1
is the number of combinations from m elements on i. For the test tasks comparing two lists: k = l!, where l -the length of the list. For the test tasks of recognition of the correct sequence: k = 5!, where 5 is the number of the sequence elements.
A matrix element at the intersection of the i-th row and the j-th column specifies a weight of the j-th answer to the i-th question. Note that the total number of answers for different questions may be different. As the result, the matrix may contain a number of empty elements.
Complementary to the above matrix, a so-called threshold assessment of learning material pt, 0 < pt < 1, pt (t is a number of MDT tree nodes), has to be specified. The assessment is performed simultaneously with testing. After that, the result of the assessment is compared with a threshold.
4.1.2. The results of the testing according to the unconditional components are inserted into a table as a sequences of pairs. Each pair comprises: 1) a question number; 2) a corresponding response number.
4.2. Let us consider a data structure for the conditional component of the test
4.2.1. For each of the conditional fragment of the tree (see Fig. 3) the matrix of coefficients and weight of questions have to be constructed (Table 2).
Table 2
Matrix of coefficients for one of the conditional fragment of the tree
1 2 j k weight
q aq1 aq2 aqk Wq
q + 1 aq+1,1 aq+1,2 aq+1,k Wq+1
i a,1 aa aij aik Wi
r ar1 ar2 arj ark Wr
Rows of the matrix are associated with the test tasks given to the respondent while passing a test component, the columns is the index (number) of respondent's answers, n is the interval of numbers of questions, k is the maximum number of possible answers (both correct and incorrect answers). Answers weight Wi has to satisfy the condition 0 < wt < 1 for any i.
Rules for defining k, p and filling out the matrix of the coefficient are the same as in the above unconditional matrix of coefficients.
4.2.2. The results of the testing are included during the testing into a table in a form of the sequence of pairs containing: 1) the question number; 2) the answer number. The resulting table can be used for reconstructing the test trajectories.
5. The Test Strategy 5.1. The test strategy for the unconditional component of the MDT
An unconditional node is an initial node of any test (Fig. 2). A respondent receives all the tasks on this part. His answer is recorded in a form of a pair i, j(i), where i is the number of the question, j(i) is the number of the selected answer to the i-th question. This information enables then to reconstruct the passing the test
by each student. According to test results, intermediary assessment is calculated Vu = au(i) x W J /n.
The score is compared with threshold value pu, after which there is an appeal to any other unconditional test component determined on the MDT tree, moves to the conditional component of the test, or to the recommendations to return back for learning the corresponding topic [11]. The above step is based on methods of fuzzy and threshold logic.
Similarly, every other component of the unconditional test is tested. An appearance of a node of a conditional task changes this similarity.
n
5.2. The test strategy for the conditional component of the MDT
The concept of MDT assumes that only one conditional task is selected from the list of conditional components of the test (Fig. 4). The test tree can be determined on the base on its response. The respondent's answer is registered in a form of a pair i, j(i), where i is the question number, j(i) is the number of the chosen question. An intermediary assessment is calculated Vc = at j j x wi on the base of test results. Then this value
is compared with the corresponding threshold pc,, where i is the number of a chosen conditional task. Thereafter, another component appears. The component can be determined by the MDT tree. Otherwise, it will be recommended to study a specific topic again. It should be noted that if there are several questions concerning the set of conditional components in the course material, they are supposed to be equal. Just one of them has to be chosen, for example, randomly.
5.3. The test strategy for the MDT tree first type traversing
The testing begins from a specific basic MDT structure Ui (Fig. 5). On this stage, table are filling out by the results described in 5.1 and 5.2. The received information may be widely used in order to form individual trajectories for each of students. The possible ways for future testing, first of all, depend on the calculated assessment (grade) of knowledge.
A. A certain portion of the test is passed successfully. Choosing one of the next basic structures from \Un,Ul2, • ■ •, Uu} for testing is determined by the expert, the test developer:
- the sequential enumeration of all remaining basic structures;
- the random enumeration of all remaining basic structures;
- a certain order of running specific basic test structures.
B. A certain portion of the test is not passed. Some recommendations for repeated studying specific topics of the course are provided. Particularly, repeated staying that begins from the unconditional component can be recommended.
Every basic structure of \Un,Ul2, ■ • Ult} is running independently, being isolated from other basic
structures. The final result is forming gradually. It can be reported in a form of the set of results for each of the basic structures or as integrated test based assessment (grade).
5.4. The test strategy for the MDT tree second and third type traversing
The order for the testing of the basic MDT structures from /,, U2, • • •, Ut} is determining by expert,
test developer, as it is described in section 5.3.A. The final result is forming gradually. It can be reported in a form of the set of results for each of the basic structures or as integrated assessment (grade) for the considered block of the basic MDT structures. The concluding step of the testing can be presented as the basic MDT structure (Fig. 1) and as a conditional fragment of the MDT structure (Fig. 3).
Since the evaluation of test results is produced for each traversed node of the test tree, the total score is gradually accumulated and, if necessary, testing can be interrupted at any step, with the further renewal.
The final grade can be both quantitative and qualitative if a mutual agreement between qualitative and quantitative scales is specified.
An important advantage of the MDT is the ability to use both the intermediary and the final results of testing for the cognitive analysis. A visualized representation of students' achievements demonstrates the progress in learning and can serve as an additional means for motivating students.
Conclusion
The concept of a mixed diagnostic test (MDT) combining unconditional and conditional components is introduced. A technique for the development and application of the mixed diagnostic tests is provided. We
demonstrated that utilizing the mixed diagnostic tests in blended education and training for assessing knowledge, professional and personal skills and abilities. The proposed approach for developing the MDT uses methods of fuzzy and threshold logic [11].
The use of MDT can support the process of the curriculum development. It provides valuable recommendations for constructing learning paths, which reduces the time and cost expenses for the organization and management of the educational process. Since the MDT can actually replace the teacher as a consultant, the proposed approach can be used for blended education and training as well as in MOOC based learning [13-15]. The technique is illustrated by a number of examples of interdisciplinary interaction in the study of new disciplines.
The proposed MDT approach provides a unique possibility to interrupt the testing procedure and to restart it later from the same point. Mainly since that, we strongly believe that the proposed approach is applicable for planning long-term educational processes.
REFERENCES
1. Bliuc, A.M., Goodyear, P. & Ellis, R.A. (2007) Research focus and methodological choices in studies into students' experiences of
blended learning in higher education. Internet and Higher Education. 10. pp. 231-244. DOI: 10.1016/j.iheduc.2007.08.001
2. Graham, C.R. (2006) Blended learning systems: definition, current trends, and future directions. In: Bonk, C.J. & Graham, C.R.
(eds) The handbook of blended learning: Global perspectives, local designs. San Francisco, CA: Pfeiffer Publishing. pp. 3-21.
3. Liyanagunawardena, T.R., Adams, A.A. & Williams, S.A. (2013) MOOCs: A systematic study of the published literature 2008-
2012. The Int. Review of Research in Open and Distance Learning. 14(3). pp. 202-227. DOI: 10.19173/irrodl.v14i3.1455
4. Kostyuk, Y.L., Levin, I.S., Fuks, A.L., Fuks, I.L., & Yankovskaya, A.E. (2014) Massive open online courses - the modern concept
in education and learning. Vestnik Tomskogo gosudarstvennogo universiteta. Upravlenie vychislitelnaja tehnika i informatika -Tomsk State University Journal of Control and Computer Science. 1(26). pp. 89-98.
5. Singera, F.M. & Stoicescu, D. (2011) Using blended learning as a tool to strengthen teaching competences. Procedia Computer
Science. 3. pp. 1527-1531. DOI: 10.1016/j.procs.2011.01.043
6. Brusilovsky, P., Knapp, J. & Gamper, J. (2006) Supporting teachers as content authors in intelligent educational systems. Interna-
tional Journal of Knowledge and Learning. 2(3/4). pp. 191-215. DOI: 10.1504/IJKL.2006.010992
7. Uskov, V. & Uskov, A. (2010) Computers and advanced technology in education - perspectives for 2010-2015. Proc. of the
13th IASTED Int. Conf. on Computers and Advanced Technology in Education (CATE 2010), Maui, Hawaii, USA.
8. Ausburn, L.J. (2004) Course design elements most valued by adult learners in blended online education environments: an American
perspective. Educational Media International. 41(4). pp. 327-337. DOI: 10.1080/0952398042000314820
9. Shute, V.J. & Rahimi, S. (2017) Review of computer-based assessment for learning in elementary and secondary education.
Journal of Computer Assisted Learning. 33(1). pp. 1-9. DOI: 10.1111/jcal.12172
10. Yankovskaya, A. (1996) Design of optimal mixed diagnostic test with reference to the problems of evolutionary computation. Proc. First Int. Conf. on Evolutionary Computation and Its Applications, Moscow, EVCA'96. pp. 292-297.
11. Yankovskaya, A.E. & Semenov, M.E. (2013) Application mixed diagnostic tests in blended education and training. Proc. of the IASTED International Conference Web-based Education (WBE 2013), Innsbruck, Austria. pp. 935-939.
12. Yankovskaya, A.E. (2018) Intelligent learning-testing and predictive systems with a cognitive component, based on Mixed Diagnostic Tests. Russian Foundation for Basic Research Journal. 4(100). pp. 26-32. DOI: 10.22204/2410-4639-2018-100-0426-32.
13. Bennett, R.E. (2011) Formative assessment: A critical review. Assessment in Education: Principles, Policy & Practice. 18(1). pp. 5-25. DOI: 10.1080/0969594X.2010.513678
14. Farrell, T. & Rushby, N. (2015) Assessment and learning technologies: An overview. British Journal of Educational Technology. 47(1). pp. 160-120. DOI:10.1111/bjet.12348
15. Gikandi, J.W., Morrow, D. & Davis, N.E. (2011) Online formative assessment in higher education: review of the literature. Computers & Education. 57(4) pp. 2333-2351. DOI: 10.1016/j.compedu.2011.06.004
Received: May 7, 2019
Yankovskaya A.E., Levin I.S., Fuks I.L. (2019) MIXED DIAGNOSTIC TESTS STRUCTURE FOR LEARNING AND TESTING SYSTEMS. Vestnik Tomskogo gosudarstvennogo universiteta. Upravlenie, vychislitelnaja tehnika i informatika [Tomsk State University Jounal of Control and Computer Science]. 49. pp. 101-109
DOI: 10.17223/19988605/49/12
Янковская А.Е., Левин И.С., Фукс И.Л. СТРУКТУРА СМЕШАННЫХ ДИАГНОСТИЧЕСКИХ ТЕСТОВ ДЛЯ ОБУЧАЮЩИХ И ТЕСТИРУЮЩИХ СИСТЕМ. Вестник Томского государственного университета. Управление, вычислительная техника и информатика. 2019. N° 49. С. 101-109
Предложен новый метод повышения эффективности учебной деятельности обучающихся. Метод основан на создании обучающих траекторий, получаемых в процессе применения смешанных диагностических тестов. Траектории обучения строятся с использованием специальных смешанных диагностических тестов, представляющих собой оптимальное сочетание безусловных и условных компонентов. Смешанные диагностические тесты могут быть использованы как в смешанном образовании и обучении, так и в массовых открытых онлайн-курсах (MOOC). Смешанное диагностическое тестирование обеспечивает мониторинг знаний, профессиональных и личных навыков и способностей учащихся, а также построение траекторий обучения для каждого учащегося. В статье представлена инновационная междисциплинарная методика построения смешанных диагностических тестов.
Ключевые слова: смешанное образование и обучение; массовые открытые онлайн курсы; траектория обучения; дерево смешанных диагностических тестов; междисциплинарное тестирование.
YANKOVSKAYA Anna Efimovna (Doctor of Technical Sciences, Professor, Department of Software Engineering of National Research Tomsk State University, Tomsk, Russian Federation). E-mail: [email protected]
LEVINIlya Semenovich (Professor, School of Education, Tel Aviv University, Israel). E-mail: [email protected]
FUKS Irina Lvovna (National Research Tomsk State University, Tomsk, Russian Federation). E-mail: [email protected]
