CC BY
23
ОБЧИСЛЮВАЛЬШ СИСТЕМИ
UDC 004.02:007
Ye.V. NIKITENKO*, O.V. TRUNOVA*
QUALITY TESTING AT ASSESSMENT OF PROFESSIONAL COMPETENCIES IN WEB LEARNING MANAGEMENT SYSTEM
Chernihiv National University of Technology, Chernihiv, Ukraine
Анотаця. У cmammi пропонуються шляхи тдвищення якостг тестування при ощнюванш ргвня профестних компетенцт у веб-cucmeMi управлтня навчанням, зокрема, розглянуто архтектуру системи та алгоритм ii роботи. Запропонований модуль пeрeвiрки критeрiальноi валiдностi тес-тiв грунтуеться на аналiзi показниюв асиметрп, ексцесу та тдексу дискримтацп; сукуптсть наведених характеристик вказуе на способи валiдизацii, що дозволяе тдвищити яюсть тестування. Розроблений модуль надання та ощнки тeстiв на основi нечтких множин надае можливiсть достатньо точно виставити оцтку при проходжент тестування i при цьому проводити ттегра-льний облт як кшьюсних, так i яюсних факторiв.
Ключовi слова: веб-система управлтня навчанням, модуль пeрeвiрки валiдностi, модуль надання та оцтки тeстiв.
Аннотация. В статье предлагаются пути повышения качества тестирования при оценке уровня профессиональных компетенций в веб-системе управления обучением, в частности, рассмотрены архитектура системы и алгоритм ее работы. Предложенный модуль проверки критериальной валидности тестов основывается на анализе показателей асимметрии, эксцесса и индекса дискриминации; совокупность приведенных характеристик указывает на способы валидизации, что позволяет повысить качество тестирования. Разработанный модуль предоставления и оценки тестов на основе нечетких множеств дает возможность достаточно точно выставить оценку при прохождении тестирования и при этом проводить интегральный учет как количественных, так и качественных факторов.
Ключевые слова: веб-система управления обучением, модуль проверки валидности, модуль предоставления и оценки тестов.
Abstract. The article offers ways to improve the quality of testing in the assessment ofprofessional competence in Learning Management System, including the reviewed system architecture and algorithm of its work. The proposed module of checking the criterion validity tests based on the analysis of indicators of asymmetry, kurtosis and discrimination index, set of these characteristics indicates on the validation methods that can increase the quality of testing. Designed module of providing and assessment of tests based on fuzzy sets allows you assessing accurately at the testing and thus pursue integrated account both of quantitative and qualitative factors.
Keywords: web learning management system, the module of checking the validity, module of providing and assessment of tests.
1. Problem statement
At the present stage of development of Ukraine, it is essential to improve the system of traditional university education. Creation and implementation of state standards of higher education require further improvement and the search for effective forms, tools and training methods, new methods of evaluating student achievement, creation an appropriate learning opportunities of students learning model in higher education.
One way to improve the quality of the educational process in institutions of higher education is the introduction of automated training system (ATS). ATS - a system that helps to master new material performs control of knowledge and helps teachers in preparing teaching material.
© Nikitenko Ye.V., Trunova O.V., 2016 3
ISSN 1028-9763. Математичш машини i системи, 2016, № 3
However, the results of such systems are perceived ambiguously, that hinders their widespread use [1].
One of the aspects that should be considered in these systems is evaluation of level of professional competencies that is the result of specific existing knowledge and skills, that is, what a student should have at the end of the course. The elimination of subjectivity in the examination of professional competence, promotes new advanced forms of electronic (computer) control (testing), which is part of the ATS.
The current education system is characterized with intensive development of the theory and practice of computer testing, which becomes an integral and important part of the educational information environment of any educational institution.
Computer testing is a method of control, which is a standardized procedure of using tests on a computer under a special software control that provides the required presentation of tests and test results processing to solve complex task goals.
The most significant claim to the currently existing distance learning systems such as Moodle, ILIAS, Web Tutor, CoureLab, Blackboard Learning System is a primitive control and evaluation of test results.
From the mentioned above we can make a conclusion about feasibility of allocation of individual independent components in the architecture of the developing system. This will make changes directly to individual modules without affecting the system as a whole.
The aim of article is to improve the quality of testing in the assessment of level of professional competence in Web-Learning Management System.
2. System architecture
The server of authorization processes the requests of unauthorized users to the system. Following the link to the site in the browser, the user can enter a name and password for authentication. Thanks to simple logic of work, the request to search a user in the system can be performed directly in the database. Depending on the type of user, a redirection takes place to one of the modules - test or viewing the results and quality of tests. A special key will be written to the browser's session, without which further work with the system will be impossible.
Server of web-services contains a set of Web services that are required for the work with database, services of providing and processing test results, also services of providing data about their results and quality. Modules of passing tests and module of viewing their results and quality will access data services.
The module of passing tests is a web-server that allows the passage of the tests through a graphical Web interface. The results of passing the tests can be stored in the database by calling web-service server with web services. Module of viewing the results and quality tests is a webserver. It uses a graphical web interface and allows you to view the results of passed tests, that stores in a database, as well as their quality.
When using the module of creation the graphs, it can visually learn the results. Control Module of quality tests - is servers with applications that enable to call Web- services of processing data and receiving results.
Module of competency assessment allows evaluating professional competence of students.
The system architecture is shown in fig. 1.
Fig. 1. System Architecture
3. The algorithm of the system
The procedure of detection of professional competencies in automated mode is implemented in two phases: the first phase includes the formation of test for students (from a set of test's tasks). It consists of two parts:
1. Experts offer a set of questions of custom test.
2. The test is verified on quality (validity and reliability).
2.1. Construction and analysis of matrix of results.
2.2. Determination of complexity and item determination of tests.
2.3. Determination of variation of marks (difference of maximum and minimum dialed scores), asymmetry, kurtosis and discrimination index).
Based on the questions that have been passed quality control, the base of the questions of user test is made, thus questions are divided into categories for the evaluation of a set of professional competencies.
The second phase involves obtaining the final test result in dependence on the importance of specific competence for individual student after passing user's test. Block diagram of previously given process is shown in fig. 2.
4. The module of the validity checking
The validity in theory of testing means compliance of form and content of the test to that it has to evaluate or measure on a plan of its creators.
Validity - one of the main criteria for a quality of test, the degree of coverage by the test the set of objects subject area compared with the standard. Validation - is not about gathering evidence of validity of the test, but about the process of actions, that raise its validation. As a result, the evidence base of the validity of the test will grow.
In the ATS the following analysis scheme of criteria l validity (provides the availability of external criterion - standard, which defines the validity of test) test's tasks are offered, that are based on the results of statistical data of testing with building a curve of distribution of correct answer's number on specific task of test. Not valid tasks considered those tasks, which had been received during the testing the following statements the right answers more than 84% of tested persons; correct answers less than 16% of tested persons.
Fig. 2. Block diagram of the algorithm of the system
Distributional (discriminatory) ability of the test task defines by the index (discrimination index), which indicates to what extent the results obtained in the performance of individual test tasks correlate with the results of all test and how well it's test task distinguishes tested persons with high mark and tested persons with low mark.
Distributional ability can be calculated as the correlation coefficient between the mark for whole test and a mark for a particular test task. The correlation coefficient is calculated as follows:
s
x
where - arithmetic mean score of tested persons, that successfully completed the j -th task of test, - the arithmetic mean score of tested persons, that not coped with the j -th task,
^Pj ■ qj - standard deviation of the j -th task, Sx - standard deviation throughout the test. The
value of the correlation coefficient is interpreted as follows: 0,7-1 - the relationship is very strong; 0,5-0,7 - average; 0,3-0,5 - weak.
The index can range from -1 to +1. When the index is «0», it means that all tested persons responded the same way (good or bad). If the correlation is positive (higher than 0), it is a suffi-
cient discriminatory ability of this test, it means that it has the sufficient ability to determine strong and weak of the tested persons. It is believed that difficult tasks have to be solved successfully by more prepared tested persons and easy tasks - less prepared, but good prepared also. If test results show a different picture, this test does not meet the characteristics and needs high-quality processing.
General characteristic of test for validity provides comparisons of indicators of asymmetry, kurtosis and discrimination index see table 1, which is directly implemented in the system.
Table . Analysis of tests for validity
As Ex r x, y Characteristic Methods of validation
= 0 = 0 The valid test
< 0 > 0 0,3-0,5 easy test with low resolution to complicate the task to increase the resolution of problems improve administration of procedure testing
< 0 < 0 0,7-1 easy test with high resolution to complicate the task
< 0 = 0 easy test to complicate the task
> 0 < 0 0,7-1 complex test with high-resolution remove or simplify tasks that make testing difficult
> 0 > 0 0,3-0,5 complex test with low-resolution validation of tasks
> 0 = 0 complex test validation of tasks
= 0 > 0 0,3-0,5 heterogeneous test (presence of heavy and light tasks) with low resolution validation of the test is the division into two separate tests.
= 0 < 0 0,7-1 heterogeneous test (presence of heavy and light tasks) with high resolution validation of the test is the division into separate tests.
5. Module of provision and assessment of tests
Getting the final test result has the goal to objectively assess the level of training of students, in particular to reduce the subjectivity that appears between teachers and students, and eliminate other (probably hidden) factors that interfere. Evaluation of the test task is that the student accumulates points for the test, and then total result is translated into all the usual estimate.
But, first, various questions from the proposed list obviously are measured in different units, and therefore the direct storage system is problematic. Second, as a rule, the translation method of total result into the assessment remains outside of using fuzzy logic [2], or applies the standard approach of working with random variables [4].
When building a model of formation of linguistic evaluation of student's competences, in our view, as input variables should be used as a quantitative factors (Q - number of questions,
bk - the number of correct answers, 2 - total score), and qualitative factors (a*, - mastering
level on 2 points. (unsatisfactory); x2 - level - on 3 (satisfactory); x3 - level - on 4 (good); x4 -level - on 5 (excellent)) [2].
Model of linguistic assessment of competence of the student (F) in general can be represented as follows (fig. 3).
The algorithm of formation of linguistic competence assessment of students is as follows: 1. To carry out the rationing of the accumulated amount of points accumulated on the interval [0;100] by levels.
Determine auxiliary variables xn / = 1,4 normalizing the accumulated amount of points
on the interval [0;100] regardless of the number of questions in the test:
b - the number of correct answers to the difficulty level 1; b2 - the number of correct answers to the difficulty level 2; b3 - the number of correct answers to the difficulty level 3; kx - coefficient for the correctness of answer to questions of 1 level of complexity; - coefficient for the correctness of answer to questions of 2 level of complexity;
¿3 - coefficient for the correctness of answer to questions of 3 level of complexity, at
0 < < k2 < k3 ;
int - division function that returns an integer; mod - division function that returns the remainder.
The variable xi, 7 = 1,4 (a*, - unsatisfactory level; x2 - satisfactory level; x, - good lev-
el; x4 - excellent level) standardized depending on meanings of the auxiliary variable x. on interval [0;100]:
20x , if
24 + 62,5(x' -1,2), f
49 + 20(x -1,6), f
t
57 + 12,8(x -2), f
X -1,2,
1,2 x 1,6, f
1,6 < x ^2, 2< X <2,7,
66 + 11,4(x -2,7), f 2,7 < x ^ 3,4, 74 + 4,5(x - 3,4), if 3,4 < x ^ 5,4,
100-3(7,4- x ), if 100, f
5,4 x 7,4, t
x >7,4.
Formulas compiled in accordance with the 100-point scale that used to convert points into traditional assessment and numeric value xi, / = 1,4 obtained in the experiment (see table 2).
Table 2. The numerical values necessary for the formation of the argument x of membership
functions o ? quality o ? output varia ble
Transition coefficient 1,2 16 2,0 2,7 3,4 5,4 7,4
The numerical value of membership function 20 62,5 20 12,8 11,4 4,5
Total points for all types of the educational activities 24 49 57 66 74 83 100
Evaluation by national scale X - unsatisfactorily X - satisfactorily x3 - good X4 - excellent
Assessment ECTS F FX E D C B A
For example, the value 7,4 corresponds to the maximum value x4 at the passage of the adaptive test which begins from 1 level of complexity; values 3,4; 2 and 1,2 - the average value the values 1,6; 2,7 and 5,4 obtained by adding the value of the left value and the dif-
ference between neighboring values divided in half (1,6=1,2+(2-1,2)/2; 2,7=2+(3,4-2)/2, and so on. d.).
On each of the received segments there were compliance coefficients of numerical values to marks. For example, between [3,4; 5,4] are points from 74 to 83, that's why a numerical value on this interval corresponds to 4.5 points ((83-74)/(5,4-3,4)).
2. Write the classification scale of linguistic variables x., / = 1,4 Linguistic variables: x1
- level of 2 points (unsatisfactory); x2 - level - of 3 (satisfactory); x3 - level - of 4 (good); x4 -level - of 5 (excellent), interpreting as a term-set with three-dimensional scale T2={NL, CS, CC}, where value NL - «does not correspond to the level of (2, 3, 4 or 5)», CS - «corresponds to slightly» and CC - «completely corresponds».
3. Ask a membership function of quality of variables x., / = 1,4 .
Each of linguistic variables «level on 2 (3, 4, 5)» has one triangular curve of membership (1) and two T-curves of membership (2) (¡_ix , //v , //v , / = 1,4), that in general may be given by expressions:
0,
juJA(xi,a,b,c) =
x-a b-a c-x
ITU'
o.
x<a.
a<x<b,
b < x < c.
c < x.
(1)
where / = 1,4 ; ye (72} ; a, b,c - some numerical parameters that characterize base of the triangle a, c and its top (b), moreover, the condition a<b<c must be met.
0, x-a b-a 1,
d-x d-c 0,
x < a, a <x<b, b < x < c, c <x<d, d < x.
(2)
where / = 1,4; y g {72}; a,b,c,d - some numerical parameters that characterize the lower base of the trapezoid a, d and the upper base of the trapezoid b, c , that acquire arbitrary real values and organized by ratio: a<b<c<d.
With considering (1) and (2) the membership function of fuzzy-term set of linguistic variable «level 2 (3, 4, 5)» will be as follows:
1) (xx, 0, 0, 15, 30); ^(x.,20, 35, 50); ju^ {x^ 40,55,100, 100);
2) jutnxL2 (x2 , 0, 0, 20, 40); ^(x2, 30, 50, 70); ju^ (x2, 60, 70, 100, 100);
3)//r^(x3,0,0,50,70); (x3, 55, 70, 85); ^c(x3, 70, 90, 100, 100);
4) juTNxL(x4, 0, 0, 70, 80); ^(x4, 70, 80, 90); (x4, 80, 90, 100, 100).
Implementation of vague conclusions is based on logic algorithm of Mamdani [Mamda-ni], because the problem is solved of gaining knowledge from data (in the form of linguistic rules). The algorithm works like a «black box». The input received numerical value, is the output [3].
The parameters a, c, b, d can be adjusted according to experimental data.
When using classifiers (triple and quaternary scales) on the carriers of fuzzy set the importance of linguistic variables is defined on the real axis interval [0; 1].
4. Determine the classification scale and membership functions of quality of investigated parameter (output variable) «Evaluation of student competencies» as a term-set of values T1={unsatisfactory (U), satisfactory (S), good (G) excellent (E)}. Linguistic variable «Evaluation of student competencies» has two triangular curves of membership and two T-shaped curves of
5. Identify the knowledge base that needed to set fuzzy production rules of assessment of student competencies is formed by specialists of subject area. It is as follows (table 3).
Table 3. Fuzzy production rules
Rules View term Variable F
xi x2 X3 X4
PR1 NL + Fe - excellent
CS + + +
CC +
PR2 NL + Fg - good
CS + +
CC +
PR3 NL + Fg - good
CS +
CC + +
PR4 NL Fg - good
CS +
CC
PR5 NL + + FS - satisfactory
CS + +
CC +
PR6 NL + + + FS - satisfactory
CS + +
CC
PR7 NL + + + FU - unsatisfactory
CS +
CC +
PR8 NL + FU - unsatisfactory
CS + +
CC +
In our view, at the development of competency level assessment module the most appropriate is integration already existing in environment MATLAB package of Fuzzy Logic Toolbox in created software addition. Fig. 4-5 shows windows of the editor of membership functions and the editor of fuzzy inference rules of membership of environment MATLAB Fuzzy Logic Toolbox, where alternatively defined membership functions for each of the terms of the input and output variables.
HQ Membership Function Editor: test
[Hi £3
File Edit View FIS Variables
XX
inputl
XX
output!
irput2
XX
input3
XX
Membership function plots plot points: 131
H y X 0
1 1 1 1 1 V Iii
10 20 30 40 50 60 70 output va ria Ые "о utp ut1 "
SO 90 100
Current Variable Name outputl
Type output
Range
Display Range p 100]
[0 100]
H
Current Membership Function (click on MF to select) Name Type
Params [0 0 25 50]
trapmf
Help
Close
Selected variable "outputl"
Fig. 4. The editor of membership functions
n Rule Editor test
S
£3
File Edrt View Options
1. If .:input1 Is не; and (inputs Is CH; and (input3 Is CH) and (Input4 Is CH] then (outputl Is O; (1}
2. If (inputl is HC) and [inputs is CH) and (input3 Is CH) and (Input4 Is nC)then (outputl is O) (1)
3. If (inputl is HC) and [inputs is CH) and (inputs Is ПС) and (Input4 Is CH)then [outputl isX) [1)
4. If (inputl is HC) and [input2 is CH) and (inputs is ПС) and (Inputs Is HCJthen [outputl is X) (1)
5. If [inputl is HC) and [input2 is ПС) and (inputs Is ПС) and (Input4 Is HC)then (outputl isX) (1)
в. If [inputl is HC) and [input2 is CH) and (input3 Is CH) and (Input4 Is HC)then [outputl isX) (1)
7. If [inputl is HC) and [inputs is ПС) and (inputs Is CH) and (Input4 Is HCJthen [outputl isV) [1)
E. If (inputl is CH) and [inputs is ПС) and (inputs Is CH) and (Inputs Is HCJthen [outputl is У) (1)
Э. If (inputl is HC) and [inputs is HC) and (inputs is HC) and (input- is HCJthen [outputl isy) (1)
10. If (Inputl is CH) and (inputs is CH) and [inputs is HC) and (input4 Is HC)then (outputl isY) (1)
and
and
and
Then
Inputl is
input2 Is
Input3 Is
input4 is
outputl is
ПС
CH
HC
none
-
HC
ПС
CH
none
-
HC
ПС
CH
none
-
HC
CH
ПС
none
-
H
У
X
SB
none
□ not
r Connection — or 4 and
□ not
Weight:
not
not
not
Delete rule
Add rule
Change rule
FIS Name: test
Help
Close
Fig. 5. Rule Editor of fuzzy inference system
6. Accumulation of conclusion by all rules carried out using max-disjunction operation. The main methods of defuzzification (transformation of fuzzy set of findings in a clear number) are a lot of methods of reduction to the definition: the method of selecting a maximum of membership function; method of center of gravity; median method; the method of choosing center of highs etc.
Experimental studies prove that the most accurate - is a method of center of gravity (see fig. 6) for a discrete set of values of membership functions
■/max
->
J max
YMfr)
r=1
where fmax - number of elements in f sampled for calculation of the «center of gravity» area F
[5].
Fig. 6. Rules viewer program
7. Evaluation of student mastering level of competencies derives from the maximum amount of points that equal to 100 [4]. If each of level of test's tasks F1,F2,...,Fn is known a linguistic evaluation ju{x)-(ju1(x),ju2(x),...,jun(x)) and the weight's coefficients were defined p — ), then the operator of information's aggregation is the weighted sum and is
characterized by its linguistic assessment, that defined by membership function on 01-classifier
n
Pr
7=1
It should be noted that the results of the testing (evaluation of mastering level of student's competence) can be used for future monitoring of professional's level of competence and checking validity as a reference.
6. Conclusions
The results of approbation of created Web system, whose main function is to automate the process of determining the level of formation of professional competence of graduates in each of the selected areas indicate the reliability and efficiency of adaptive testing environment. Simplicity, adaptability and versatility of the software complex allow to use it for either examination of professional competence or to assess knowledge of the discipline.
It was experimentally established that using the algorithm of formation the linguistic evaluation of student's competences based on vague sets allows you to accurately enough set the assessment at the testing and thus pursue integrated account both quantitative (number of questions, the number of correct answers, total score) and qualitative factors (x - the level of assimilation on 2 (unsatisfactory), x2 - level on - 3 (satisfactory), x3 - level on - 4 (good), x4 - level on - 5 (excellent), considering the uncertainty of the last. By installing criterion significance of reliability's level of membership functions of input (output) variable's quality, you can change the final results depending on the level of preparedness of students.
The developed application has the following advantages: learning regardless of time and spatial location; checking tests for the validity allows teachers to choose more quality educational material and does not go beyond student's knowledge level; the ability to view material not just as text, but also through media increases the effectiveness of training program; the ability to view test results and demonstration of statistics as graphs are more clearly.
REFERENCES
1. Нащональна доктрина розвитку осв^и Укра!ни у ХХ1 столик проект // Освга. - 2001. - № 60 -62. - 24-31 жовтня.
2. Домрачев В.Г. Нечеткие модели рейтинговых систем оценки знаний [Электронный ресурс] / В.Г. Домрачев, О.М. Полещук, И.В. Ретинская. - Режим доступу: http://www.ict.edu.ru/ vconf/files/tm01 627.doc.
3. Дьяконов В. Алгоритмы нечёткого вывода: алгоритм Мамдани и алгоритм Сугэно / В. Дьяконов, В. Круглов // Математические пакеты расширения MATLAB. Специальный справочник. - Санкт-Петербург: Питер, 2001. - С. 307 - 309.
4. Ротштейн А.П. Влияние методов дефаззификации на скорость настройки нечеткой модели / А.П. Ротштейн, С.Д. Штовба // Кибернетика и системный анализ. - 2002. - № 5. - С. 169 - 174.
5. Трунова О.В. Алгоритм ощнки устшност засвоення навчально! дисциплши студентом на основi теорп неч^ких множин / О.В. Трунова, А.Г. Гребшник, 1.С. Сютер // Математичне та iмiтацiйне моделювання систем. МОДС 2014: Дев'ята мiжнар. наук.-практ. конф. Тези доповщей (Чершпв-Жукш, 23-27 червня 2014 р.). - Черншв: Чершпвський державний технолопчний ушверситет, 2014. - С. 296 - 299.
Стаття над1йшла доредакцп 07.06.2016
