E-LEARNING EDUCATION SYSTEMS EFFICIENCY RATING MODEL Текст научной статьи по специальности «Компьютерные и информационные науки»

E-LEARNING EDUCATION SYSTEMS EFFICIENCY RATING MODEL

Vyacheslav Bratkevich1

department of Computer Systems and Technologies, Simon Kuznets Kharkiv National University of Economics, Kharkiv, Ukraine vvb1944@gmail.com

ORCID: http://orcid.org/0000-0002-7217-7767

ARTICLE INFO ABSTRACT

E-learning support system is a mandatory attribute of the modern educational process. What criteria laid down in its development depends largely on the effectiveness of further applications of the entire system. The evaluation of the training effect depends on various factors, each of which is determined by many interrelated criteria. In conditions when there are resource constraints (financial or time), the actual becomes the problem of determining the factors (criteria) that have the most significant impact on the effectiveness of e-learn-ing system. The article offers step-by-step method of constructing a model that allows to represent the original set of criteria of evaluation of the training effect of the use of e-learn-ing system in the form of appropriate level of influence grades. Introduces the concept of

configuration vector and weight vector configuration, ranking models and based on them describe the procedure of finding the most substantive relationship between the criteria. Provides formulas to calculate appropriate weighting factors for current connections. An indicator of the importance of communication is the change of vector configuration, ranking models, and quantitative characterization of the dominant relations is defined as the difference between the weight coefficients of the respective vectors of the configuration options ranking of the analyzed models. The obtained results can be applied when developing holarchical models, support systems, e-learning, which allow us to quantitatively justify decisions regarding the choice of appropriate alternatives.

© The Author(s) 2020. This is an open access article under the CC BY license http://creativecommons.org/licenses/by/4.0).

One of the main means of implementing the training and control technologies in vocational education has become e-learning support systems (ELSS), which provide informational, methodological and technological support for education. The effectiveness of the further use of ELSS largely depends on what criteria are laid down during their development.

The effectiveness of using ELSS is understood as the ratio of the result relative to the educational achievements of students in the corresponding expenses necessary to achieve this result. That is, the ratio of the effect to costs, where costs are understood as the cost of the ELSS system, on the basis of which training is carried out. As an effect, in the future, let's consider the competencies (knowledge, skills) that students receive in the period between the beginning and the end of the learning process. In the framework of this study, the main focus is on creating a model for the relationship of indicators to assess the effect of using the e-learning support system.

Assessing the educational effect is not a trivial task because it depends on various factors, each of which is determined by a set of interrelated criteria. The success of its solution is largely determined by the intuition and experience of the developer of the e-learning system. As a result, in the process of forming an integrated assessment, system developers should pay considerable attention to expert methods that allow quantitatively describing the main characteristics of the educational effect and significantly reduce the intuitive component of technological decision making.

2. Literature review and problem statement

An analysis of recent studies on the effectiveness of e-learning support systems has revealed that the authors have focused on the development of an appropriate benchmark, with a high-quality

>> c

¡3 c

J S3 g

§ £

to

S c .2

ts ^

§ £ -T3

s ^

G ^

> ë

Article history: Received date 26.12.2019 Accepted date 20.01.2020 Published date 30.01.2020

Section:

Distance, Open & Online Education

10.21303/2313-8416.2020.001157

KEYWORDS

educational effect

assessment criteria

hierarchy

holarchy

ranking model

ranking model configuration

configuration weight

1. Introduction

description of the criteria. Thus, [1] "indicators of the effectiveness and quality of e-learning functioning in the system of primary and secondary education" are proposed. Each indicator is accompanied by a list and a further description of the relevant criteria. But any quantitative estimates by the authors are not given.

As a rule [2], almost all systems for evaluating learning effectiveness are based on the model proposed by Donald Kirkpatrick [3]. The merit of this model lies in its integrity and systematic approach. The main drawback is its non-specific nature.

There are known works where SWOT analysis is used to solve strategic tasks in pedagogy [4], but the results of SWOT analysis are provided qualitatively, and causal relationships between the indicators of the evaluation of learning effectiveness are not considered.

The closest to the current topic is studies [5, 6], in which the staff of the bank's training center developed a mathematical model of calculation and identified a number of criteria that have a direct impact on the learning effect and effectiveness of e-learning. They offered their definition of efficiency as a ratio of money spent to results achieved. The following criteria were included in the expenditure category: availability of documentation detailing the organization's rules and procedure for developing e-learning; complexity of course implementation; level of complexity of knowledge; duration of course development; qualification of specialists - course developers. The outcome category also includes a set of criteria: course evaluation by the trainees; results of knowledge testing; the direct impact of knowledge on business; demand for the course; number of participants.

The disadvantages of the model [6] include the following: the list of criteria does not fully take into account the specifics of assessing the quality of the educational process of a university; the mutual influence of the evaluation criteria is not considered; there is no justification for the rating scale and its qualitative description; the model is not considered as a holistic system, which is characterized by hierarchical or holarchical relations between its elements.

In [7], to assess the effect of training, a group expert assessment based on the feedback questionnaire method is used. As a result, an integral (generalized) indicator is obtained, calculated using the corresponding multiplicative convolution. To evaluate the criteria in this case, a ranking method is used, which allows to order the sequence of criteria and assign a rank to each of them. The disadvantage of the model is the use of rather complex personal forms and the assumption that between its elements there is only a hierarchical relationship.

So, until now, the principles of creating systems for evaluating the effectiveness of using the e-learning support system have been based solely on the corresponding hierarchical models. That is, feedbacks between the relevant criteria of the known models are not considered, which significantly reduces the scope of their application. Thus, the problem of applying the methods of analysis of systems with feedbacks, which allow more fully take into account the features of the subject area of research, as a result of their further solution.

3. The aim and objectives of research

The aim of research is development of a ranking model with holarchical interconnected indicators for assessing the effect of using ELSS.

To achieve the aim, the following objectives are set:

1. To create a generalized list of criteria affecting the effect of using e-learning systems.

2. Based on expert assessments of the mutual relations between the criteria, construct an output graph with vertices - the criteria and arcs, the direction of which indicates the dominance between the current pairs of adjacent criteria.

3. To convert the output graph to the appropriate adjacency and reachability matrixes.

4. To analyze the reachability matrix and build a ranking model based on it.

5. To determine the most significant relationships between the evaluation criteria and calculate their weight coefficients.

4. Materials and methods of research

The study is based on the use of the hierarchy analysis method (HAM) [8] and its further development - the system analysis method [9].

As an example, a nine-element set of qualitative indicators for assessing the effect of using the e-learning support system is considered below.

ELSS qualitative indicators [1].

Criterion 1. Technology development course.

Course development technologies can be different: Web-technologies, telecommunication technologies, cloud technologies, mobile learning technologies, distance learning technologies, etc.

Criterion 2. Models of courses and their pedagogical design.

The course model determines the specifics of the organization and presentation of the material.

Criterion 3. Availability of the course curriculum.

The presence of the curriculum reflects authorship, structure, topics of sections or modules, its technical component and resource base.

Criterion 4. Course delivery technology.

Delivery and access technologies for courses and services require an asynchronous or synchronous training format based on the use of e-learning shells, Google training capabilities in the form of sites and blogs.

Criterion 5. Ease of access to courses and services.

Criterion 6. Convenience of course navigation.

The pedagogical design and ease of navigation should support the learning process and be user-friendly.

Criterion 7. Availability of guidelines for working with course materials.

The presence of guidelines for working with course materials helps both students and teachers.

Criterion 8. Availability of interaction technologies.

Technology for interacting with students involves the use of a blog, chat, forum, Skype, email, phone, face-to-face meetings or consultations.

Criterion 9. Adaptability and personalization of the course.

Adaptability and personalization of the course make it possible to take into account the contact nature of the audience, its readiness to perceive the material, adjust the content filling, balance the information load, and build individual learning paths.

5. Research results and their discussion

Problem statement. There are nine elemental sets of partially or fully related criteria that are considered contextually as a system.

The description of the system can take one of two different but related forms: a binary matrix or a directed graph (network) for the geometric representation of relations.

It is necessary to transform the original graph into a system with levels (rank model) [8].

In the general case, the procedure for constructing a model is as follows. The main ELSS qualitative characteristics are identified in the form of a set of relevant criteria. Further, these criteria are presented in the form of a multiply connected oriented graph and the corresponding adjacency matrix, on the basis of which the reachability matrix is constructed. The final step is a consistent analysis of the reachability matrix and the construction of a system with levels on its basis.

Let's consider this procedure in more detail in the form of a list of the corresponding steps.

Step 1. Building a geometric representation of the relationship between the criteria.

The current set of vertices H is defined as follows:

H=hv hv hv hA, hy h6, hv hg, h9},

where h—h9 - a list of criteria that were considered in the previous section.

The set H can be imaginary as a set of criteria vertices (Fig. 1), the relationships between which are determined as the result of the corresponding answers or expert judgments to the following questions.

f 4. Course delivery V technologies

Fig. 1. Criteria relationship graph

Question 1. Is there a relationship between the current pair of criteria? If the answer is yes, then the corresponding pair is connected by a line.

Question 2. Which of the criteria is more important (dominant)? The specified criterion is indicated by an arrow that points to it.

So, according to the expert, between criterion 8. "Availability of technologies for interacting with students" and criterion 9. "Adaptability and personalization of the course" is a mutual influence, and criterion 8 (according to the expert's judgment) depends on criterion 9.

Step 2. Constructing an adjacency matrix.

The adjacency matrix B={b.} is filled on the basis of the binary relationship "depends on" and is determined as follows:

b=1 if the criterion h. depends on the criterion h;

b=0 - otherwise.

So, from Fig. 1 it follows that the criterion h1 depends on the following set of criteria h2, h3, h5, h6, h8, h9. Therefore, in the first row (i=1) of the adjacency matrix, it is necessary to write the units in column 2, 3, 5, 6, 8, 9. Other rows of the matrix are filled in the same way (Table 1).

Table 1

Adjacency matrix

i 7' hi h2 h3 h4 h5 h6 h7 h8 h9

h1 0 1 1 0 1 1 0 1 1

h2 0 0 1 0 0 0 0 0 0

h3 0 0 0 0 0 0 1 1 1

h4 1 0 0 0 0 1 0 1 0

h5 0 0 0 0 0 0 0 0 0

K 0 0 0 0 1 0 0 0 0

h7 0 0 0 0 0 0 0 0 1

h8 0 0 0 0 0 0 0 0 1

h9 0 0 0 0 0 0 0 0 0

After the matrix is full, a transitivity test should be performed to identify violations of this condition. If transitivity violations are detected, then the vertices leading to this violation must be checked to eliminate it.

Step 3. Construction of the reachability matrix.

The reachability matrix is defined as a binary matrix in which the elements of unity are if the vertex of the graph is reachable from another vertex in any way, otherwise its elements are zeros. In [8], a formal procedure for its determination is provided:

(I+B)k-1<(I+B)k=(I+B)W,

according to which, on the basis of matrix B, a binary matrix (I+B) is formed, where I is the identity matrix. It is shown that there exists a smallest integer k for which each element of the matrix (I+B)k-1 is less than or equal to the corresponding element of the matrix (I+B)k, and the corresponding elements of the matrices (I+B)k and (I+B)k+' level. The matrix on the right side of the expression is called the reachability matrix.

With a small number of criteria (up to 7-10), the construction of the reachability matrix can be performed by directly tracking the links in the original graph or as a result of analysis of the corresponding rows of the adjacency matrix.

So, from consideration of the first row of the adjacency matrix, it is possible to see that with respect to the vertex h1, the vertices h2, h3, h5, h6, h8, h9. reachable. And taking into account the vertex h itself, the vertices that are reachable will be: hp h2, h3, h5, h6, h7, h8, h9. Therefore, units should be written in the corresponding cells of the first row of the reachability matrix.

This procedure is considered in more detail in [10]. The final result of the second step is shown in Table 2.

Table 2

Reachability matrix

i j K h2 h3 h4 h5 h6 h7 h8 h9

hl 1 1 1 0 1 1 1 1 1

h2 0 1 1 0 0 0 1 1 1

h3 0 0 1 0 0 0 1 1 1

h4 1 1 1 1 1 1 1 1 1

h5 0 0 0 0 1 0 0 0 0

h6 0 0 0 0 1 1 0 0 0

h7 0 0 0 0 0 0 1 0 1

h8 0 0 0 0 0 0 0 1 1

K 0 0 0 0 0 0 0 0 1

Step 4. Analysis of the reachability matrix and building on its basis a system with levels (ranking model).

Using the reachability matrix allows to divide the current set of vertices H={hv h2, h3, h4, h5, h6, h7, h8, h9} into many levels, and also divide each level into the corresponding subsets. Thus, from the set of vertices H, two types of sets can be distinguished:

- reachable vertices RV(h) - the set of vertices that can be reached from the vertex h;

- predecessor vertices PV(h) - the set of vertices from which one can reach the vertex h.. The set of those h. vertices for which RV(h)=RV(h)HPV(h) is satisfied and can't be

reachable from any of the other vertices H; according to [6], they are defined as the hierarchy level.

To build all levels, it is necessary to apply the following iterative procedure. 1. Form a table with elements that has such columns:

h,, RV(h), PV(h), RV(h)HPV(h). 2. In the table, find a lot of elements - criteria, satisfy the condition:

PV(h)=RV(h)nPV(h).

These elements form many criteria of the first (junior) level of the ranking model.

3. Cross out the current set from the table and apply the second (iterative) step.

Table 3 shows the result of the implementation of the first stage of the procedure under consideration.

Since the sets PV(h) and RV(h)HPV(h) coincide only for the fourth row (-4), according to [6], the first (lowest) level consists of one criterion No. 4 - "Delivery Technologies course ".

Further, from the Table 3, all previously selected criteria and the corresponding rows are deleted. In this example - line 4 and criterion h4.

Table 3

Determination of the vertices of the lowest hierarchy level (1st level)

h. RV(h ) PV(h.) RV(h ,)nPV(h )

hi K h2' h3' h5' h6- h7' h8' h9 h1> h4 hi

h2 h2, h3, h7, h8, h9 h1, h2' h4 h2

h3 h2' h7' h8' h9 h1, h2, h3, h4, h7 h2, h7

h4 h1, h2, h3, h4, h5, h6, h7, h8, h9 h4 h4

h5 h5 K h4, h5' h6 h5

h6 K h6 h1, h4, h6 h6

h7 h7, h9 K h2' h3' h4' h7 h7

h8 h8> h9 h1, h2, h3, h4, h8 h8

h9 h9 hV K h3' h4' h7' h8' h9 h8

The new table obtained in this way is analyzed by analogy with paragraph 2. As a result, a lot of criteria are determined that are related to an older level. Fig. 2 provides the final result of this procedure for constructing a ranking model.

Level 1

Level 2

Level 3

Level 4

Level 5

Level 6

4. Course delivery technologies

Fig. 2. The ranking model of qualitative indicators for assessing the effect of using ELSS (for the relationship between the criteria, Fig. 1)

From the figure it follows that the model has six levels and the most important criterion (first level) among the qualitative indicators for assessing the effect of ELSS use is criterion No. 9 "Adaptability and personalization of the course".

The ranking model is considered, allowing to assess the degree of influence of the current set of criteria on the final result of the assessment.

Search for the most significant relationships between the evaluation criteria

No less important for the developer of the ELSS system is the search for the most significant relationships between the criteria for assessing its quality and the quantitative assessment of the significance of the relationships that are considered.

To solve this problem, the author proposes optimization procedure based on enumeration and further analysis of the response to breaking successively of all (or selective) relationships in the original graph. It is proposed to use a change in the levels of criteria in the corresponding ranking model as an indicator of the significance of the relationship.

The configuration of the current ranking model can be imagined as the following 6-element configuration vector:

where each element contains a subset of the criteria of the corresponding level, starting from the oldest - the first.

In the initial graph of the relationship of the criteria, three types of vertices can be distinguished (Fig. 3):

type 1 - vertices of absolute dominance or independent vertices; type 2 - vertices of absolute dependence or dependent vertices; type 3 - vertices of mixed type.

In the current example, vertex 9 refers to the first type and vertex 4 to the second. The remaining vertices are vertices of a mixed type.

Among the criteria that relate to the vertices of the second and third types, it is possible to distinguish two options for the relationship between adjacent criteria.

The first option should include those which break leads to a change in the configuration vector. Otherwise, there is a redistribution of the levels of the ranking model, and in the future it is possible to call such relations "critical", to which the ELSS developer needs to pay special attention.

For vertices of the second and third types, the following tasks are relevant:

- search for critical connections with adjacent vertices;

- quantification of the importance of critical connections.

Considering the configuration vector as an indicator of the importance of the current connection, sequentially breaking the connection between the criteria, the degree of change in the configuration vector of the corresponding rank model is estimated.

The implementation of this step involves the construction of many variants of the ranking model, which makes its implementation "manually" almost impossible. Therefore, at the Department of Computer Systems and Technologies of S. Kuznets Kharkiv National University of Economics developed appropriate software that allows to fully automate the investigated procedure. In this case, the initial data can be generated both in an interactive mode, by organizing a dialogue with an expert, and by directly filling in the adjacency matrix. The processing result is generated as a configuration vector.

As an example, Table 4 shows the analysis of the influence of each of the three connections of criterion No. 3 "Availability of the curriculum of the course" (Fig. 1) on the change of the current configuration vector {9; (7, 8); (3, 5), (2, 6); 1; 4} models being considered.

In a similar way, an analysis of all the connections of the source graph is performed. As a result, not significant relationships are identified, after elimination of which the current graph (Fig. 1) is transformed to the form shown in Fig. 4.

Thus, the ranking model allows the developer of the ELSS system not only to identify the most important criteria for its evaluation, but also to determine the dominant relationships that affect the redistribution of its ranks.

{9; (7, 8); (3, 5), (2, 6); 1: 4},

Fig. 3. Classification of relations between adjacent pairs of vertices

Table 4

Changing the output configuration vector of a ranking model relative to current relationships

Breaking current connection

Configuration vector

3^9 3^8 3^7

{9; (7, 8); (3, 5), (2, 6); 1: 4} {9; 7; (3, 5); (2, 6, 8); 1; 4} {9; 8; (3, 5); (2, 6); 1; (4,7)}

Fig. 4. The result of optimizing the relationships of the current graph

Quantification of relationships between assessment criteria

In addition to a qualitative description of the relationship of the criteria for making appropriate design decisions, their quantitative assessment is also of interest. Especially this issue becomes relevant when several links influence simultaneously the redistribution of ranks. In the considered example (Fig. 4) these are the connections 1^2 and 1^-6 and 3^7 and 3^-8.

Statement.

The more the mutual arrangement of the criteria levels in the ranking model changes, the more important is the influence of the critical relationship between the current pair of criteria on the ELSS qualitative characteristic.

It is proposed to perform a quantitative assessment of the significance of the relationships of the optimized graph on the basis of the concept of the weight of the configuration vector of the rank model, which is understood as the additive convolution of the products of the rank of the vector by the number of criteria of this rank.

Definition 1.

The weight (V0) of the configuration model vector of the rank model, contains N0 levels (ranks), is the sum of the products of number 5 (5=1, 2,..., N0) of the current level (rank) by the number of criteria r0 of this level.

V =X

5 X r

(1)

So, the weight is the element vector of the configuration of the initial rank model (N0=6,

r0=1, r0=2, r0=2, r0=2, r0=1, r° =1), shown in Fig. 2 is calculated as follows:

1 2 3 4 5 6 °

V{{ 9; (7, 8); (3, 5); (2, 6); 1; 4 }=14+2-2+3-2+4-2+54+6•1=30.

Definition 2.

The weight (V^j) of the configuration vector of the current (relative to the initial) rank model, contains N.^. levels (ranks), with a relationship (i ^j) between criteria No. i and No. j, called the

sum of products of number 5 (5=1, 2,..., N^j) of the current level (rank) by the number of criteria of a given level.

V - j = 1 s x ri -J.

(2)

So, the weight of the elemental configuration vector {9; 8; (3, 5); (2, 6); 1; (4, 7)} (Table 4) of the initial ranking model with the critical connection 3^7 (N0=6, ^=1, r0=2, r0=2, r^=2) is equal to:

F3^7=H+2-1+3-2+4-2+5-1+6 2=34.

Definition 3.

The quantification of the dominance of the criterion h. over the criterion h will be called the quantity ( A^. ) normalized with respect to the weight coefficient of the initial configuration vector V0, which is equal to the absolute value of the difference in the weight coefficients of the initial V0 and current V.^. configuration vectors of the corresponding rank models:

uuu

^ J=-

•"0 J

X x ;-11

-100.

(3)

s x r

So, the quantitative dominance of criterion h7 over criterion h3 is calculated as follows:

A3^7=(a6s(30-34)/30)x100=13.3.

Fig. 5 shows the final result of quantitative assessment of the dominance of related criteria of a ranking model. Quantitative values on the arcs of the graph are normalized with respect to the weight of the initial configuration vector and increased by a factor of 100.

Level 1

Level 2

Level 3

Level 4

Level 5

Level 6

4. Course delivery \ technologies J

Fig. 5. A quantitative assessment of the dominance of related criteria of the ranking model for

assessing the ELSS educational effect

6. Conclusions

1. A step-by-step methodology for building a model is considered, which allows to present the initial set of criteria for assessing the educational effect of the e-learning support system in the form of ranks corresponding to the level of influence.

2. The concepts of the configuration vector and the weight of the configuration vector of the rank model are given and based on them, the procedure for searching for the most significant relationships between the ELSS assessment criteria is discussed.

3. The proposed step-by-step methodology and the corresponding formulas for calculating the weight coefficients for the current relationships between the criteria for ELSS evaluation.

The results can be applied in the development of a cluster of ELSS qualitative characteristics corresponding holarchical models that allow quantifying the decision-making regarding the choice of alternative options for e-learning support systems.

References

[1] Voitovich, I. K. (2014). Kriterii effektivnosti elektronnogo obucheniia i kachestva elektronnykh obrazovatelnykh programm v vuze. Vestnik TGPU, 4 (145), 152-156.Available at: https://vestnik.tspu.edu.ru/files/vestnik/PDF/articles/voytovich_ i._k._152_156_4_145_2014.pdf Last accessed: 12.02.2019

[2] Bubnov, G. G., Pluzhnik, E. V., Soldatkin, V. I. (2015). Kriterii ocenki kachestva v sisteme elektronnogo obucheniia. Cloud of Science, 2 (4). Available at: https://cyberleninka.ru/article/n/kriterii-otsenki-kachestva-v-sisteme-elektronnogo-obucheniya Last accessed: 12.02.2019

[3] European Commission (2013). Work-based learning in Europe - Practices and Policy Pointers. European Comission. Available at: https://www.skillsforemployment.org/KSP/en/Details/?dn=WCMSTEST4_057845

[4] Dave, S. K., Modi, V. K. K., Pithadiya, P. G., Bhavsar, K. M., Shah, J. (2018). SWOT Analysis of Indian Higher and Technical Education Institutes. Global Research and Development Journal for Engineering. Available at: http://www.grdjournals.com/ uploads/conference/GRDCF/010/009/GRDCF010009.pdf

[5] Iaskevich, S. V., Makovskaia, E. V. (2013). Obzor metodik ocenki effektivnosti elektronnogo obucheniia. Innovacionnye processy i korporativnoe upravlenie. Minsk, 448. Available at: http://elib.bsu.by/handle/123456789/103122 Last accessed: 12.02.2019

[6] Stulov, A. (2010). Ocenka effektivnosti elektronnogo distancionnogo obucheniia. Spravochnik po upravleniiu personalom. Web-Soft. Available at: http://eng.websoft.ru/db/wb/26D3C1E6AD7BC171442579820031F250/doc.html Last accessed: 12.02.2019

[7] Latypova, V. A. (2016). Ocenka effektivnosti processa obucheniia pri nalichii slozhnykh otkrytykh zadach s pomoschiu ek-spertnykh metodov. Inzhenernii vestnik Dona, 1. Available at: ivdon.ru/ru/magazine/archive/n1y2016/3540 Last accessed: 12.02.2019

[8] Saati, T. (1993). Priniatie reshenii. Metod analiza ierarkhii. Moscow: Izdatelstvo «Radio i sviaz», 278.

[9] Saati, T. L.; Andreichikov, A. V., Andreichikova, O. N. (Eds.) (2008). Priniatie reshenii pri zavisimostiakh i obratnykh svia-ziakh: Analiticheskie seti. Moscow: Izdatelstvo LKI, 360.

[10] Bratkevych, V. V. (2016). Otsenka kachestva system podderzhky E-learning. Systemy obrobky informatsii. Problemy i pers-pektyvy rozvytku IT-industrii, 4 (141), 219-222.