CC BY
10
Gavrilina E.A., Zakharov M.A., Karpenko A.P., Smirnova E.V., Sokolov A.P.
Bauman Moscow State University, Moscow, RF
MODEL OF INTEGRAL ASSESSMENT QUALITY OF TRAINING GRADUATES OF HIGHER ENGENEERING EDUCATION
ABSTRACT
Abstract. This study reviews stationary aspect of the problem of graduates meta-competencies quantitative evaluation. The mathematical model for such evaluation is constructed. Structure of software system META-3 and its main characteristics is described.
KEYWORDS
Engineering education, quality of education, model evaluation. Гаврилина Е.А., Захаров М.А., Карпенко А.П., Смирнова Е.В., Соколов А.П.
Московский Государственный Университет им. Н.Э. Баумана, г. Москва, Россия
МОДЕЛЬ ИНТЕГРАЛЬНОЙ ОЦЕНКИ КАЧЕСТВА ПОДГОТОВКИ ВЫПУСКНИКА
ИНЖЕНЕРНОГО ВУЗА
АННОТАЦИЯ
В статье рассматривается часть проблемы количественной оценки мета-компетенции выпускника инженерного ВУЗа. Описывается математическая модель для такой оценки, структура программной системы META-3 и ее основные характеристики.
КЛЮЧЕВЫЕ СЛОВА
Инженерное образование, качество образования, модель, оценка. Introduction
We distinguish meta-objective, meta-creative and meta-cognitive (meta) competencies of a student [1, 2]. Quantitative evaluations of the components of these competencies are called indicators.
By meta-objective competencies we mean meta-objective notions assimilated by students and so called universal learning activities (regulative, cognitive, and communicative). In other words, we suppose that the components of meta-objectivity include assimilated meta-notions as well as regulative, cognitive, communicative competencies.
Meta-cognitive competencies are defined after J. Flavell as personal knowledge concerning students' own cognitive processes and the results of their cognitive activity. We highlight indicators of a student's meta-cognitivity that evaluate his or her abstract thinking, verbal abilities, quantitative skills, perceptive abilities, spatial thinking, and technical thinking.
Meta-creativity is defined as an integral quality of a student which provides for him or her possibility of exiting beyond the frames of stimulus situation, as well as an ability to recognize how the exit is carried out and choose the most suitable strategies for that. We offer indicators of a student's meta-creativity based on the evaluation of his or her mental flexibility, mental productivity, mental fluency, mental originality, and status of the problem development.
For all the reviewed types of meta-competencies the following levels of their assimilation are distinguished: declarative knowledge; conceptual knowledge; procedural knowledge; situational knowledge; behavioural knowledge [2].
The study reviews the evaluations of students' meta-competencies on the basis of the analysis of their behavior in social networking services, such as Twitter, Facebook, VK, Odnoklassniki, Linkedin. This means that meta-competencies are evaluated on the basis of the analysis of direct (personal data, statements, comments) and indirect data (subscription to network groups, events, places, other participants) extracted from the aforementioned social networking services [3, 4]. We introduce software system (SS) META-3, that performs extraction of the required information from the social networking services and evaluation of a student's meta-competencies on the basis of this information. Using this values SS makes an evaluation of the given student's learning style and his or her cognitive abilities. Also basing on the latter evaluations SS forms an evaluation of student's cognitive abilities, as well as evaluation of his or her type of behavior. The system is aimed at use in educational institutions and HR departments of companies.
Mathematical model
We introduce the following symbols: M = (mi, i g [1: 3]) is a fixed set of meta-competencies in review, thus m1, m2, m3 are accordingly meta-objectivity, meta-cognitivity, and meta-creativity; Si = (si j, j g [1: Si ]), i g [1: 3] is a set of sub-meta-competencies of meta-competency mt, where Si is a number of these sub-meta-competencies; Ft = Ft (f j, j g [1: Si ]), i g [1: 3] is a set of integral evaluations of the level of meta-competency mt, where fi j is an evaluation of the level of sub-meta-competency si ■; Ai , = (at , k, k = [1: Ai , ]), i = [1:3], j g [1: si ■ ] is a vector of characteristic
features (VCF), that define a level of sub-meta-competency si ■, where
is a length of this vector. Names
of the sub-meta-competencies and components of VCF, as well as their numbers are defined by an expert during software system setup procedure.
We use the so-called expanded matrix mathematical model (MMM) for quantitative evaluation of meta-competencies (table 1). The following abbreviations are used in the table: MC - meta-competency; MO - meta-objectivity; MCg - meta-cognitivity; MCr - meta-creativity; SubMC - sub-meta-competency.
Table 1. Expanded matrix mathematical model for quantitative evaluation of students' meta-competencies
MC Level SubMC Level VCF
MO, m1 F •V /1,1 Ai
V 'K = Sr / K, K n S1 K n S1
MCg, m2 f2 s2,1 f2,1 A2,1
S2,K , K = S2 f2,K , K = S2 AK , K = S2
MCr, m3 F3 S3,1 /3,1 A3,1
S3,K > K = S3 f3,K , K = S3 Ak , K = S3
The following values should also be defined: T - evaluation of learning style of the given student;
I - evaluation of his or her cognitive abilities; V- student's way of thinking, which is formed on the basis of his or her learning style and ways of thinking; B - type of student's behavior.
We believe that evaluations of competencies and meta-competencies are defined on integer scales
a(F ) = (4mn(F ),..., ^maxF)) , Af ) = (¿mm (f, ), ..., ¿max (f , ))
(1),
where Amm (•), Amax (•) are lower and upper evaluations, accordingly Scales are defined by an expert during
software system setup procedure. Evaluations of learning style, cognitive abilities, way of thinking мышления and students' type of behavior are determined accordingly on linguistic scales
Л(Т ) = (ti ,.t2,..., tlTl ) , л (I ) = (il ,.i2,..., ^ ) , Л (V ) = (v , v2,..., v,v| ) , л ( B ) = (b , b2,..., b,B| ) (2),
which are also defined by an expert during software system setup procedure.
Structure of SS META-3 and its main characteristics
Software system has a client/server architecture and contains client and server ends. Components of client end: administration module; students' user interface module; teachers' user interface module. Components of server end: setup module; data collection module; data analysis module; skills evaluation module; module for storage of large extra data volumes, which includes server for databases of study
materials in text files, file repository of graphic materials, streaming data server for storage of audio and video records.
Software system is designed with the use of cloud technologies and services. This solution provides an opportunity to get dynamic, available on request, self-sufficient, and scalable services of cloud calculations (up to the level of business processes). Software system functions under the control of network operating system like Linux powered by free software and uses open components. Client end of software system provides function access through web browsers of the recent versions. Software system uses DBMS like MySQL.
Client end of software system
Administration module. This module realizes the following main functions: addition, removal and authentication of users; control and demarcation of access rights (including students' access to the functions of software system listed below); user groups' management; storage and editing of user data; assignment of default values for free parameters related to client end.
Students' user interface module. This module provides the following main functions: extraction of student's VCF from different user environments (social networks, MOOC-environment, LMS, electronic learning resources (ELR)); calculation of normalized evaluations of the student's sub-meta-competencies based on the VCF; determination of all three student's meta-competencies by means of an applicable trained strategy; likewise, determination of the student's learning style and cognitive abilities; likewise, determination of student's behavior type in specified user environment.
Teachers' user interface module performs the following main functions: extraction student's VCF, from different user environments; calculation of normalized evaluations of the student's sub-meta-competencies based on the VCF; determination of all three student's meta-competencies by means of an applicable trained strategy; likewise, determination of the student's learning style and cognitive abilities; likewise, determination of student's behavior type in electronic learning system; formation of subject-oriented student groups; formation of student groups with the purpose of development of synergetic effect within these groups.
Server end
Setup module provides assignment by the expert of values for free parameters of the used methods and algorithms (table 2).
As predefined proximity measure p(Fi k , Fjk^) for all the meta-competencies we use Euclidean vector norm
ij'É {fi.j K ~ fi,],k1 J (3).
We use the following learning styles by default (| T| = 4) : tx - activist, t2 - reflector, t3 - theorist, t4 - pragmatist. Likewise, the following cognitive abilities are predefined in the software system (|/ = 5): ix - disciplinary, i2 - synthesizing, i3 - creating, i4 - respectful, i5 - ethical. There are following default types of student's behavior in electronic learning system: bx - activist, b2 - reflector, b3 - theorist, b4 - pragmatist.
Table 2. Predefined software system settings (for VKontakte social
networking service)
p( Fv F^) =
F F
i ,kl' i ,k2
Meta-competency
Sub-meta-competency
VCF
Meta-competency
Sub-meta-competency
Meta-
objectivity
(MO)
Regulativity (R)
Communicativity
(C)
Insightfulness (I)
Availability of portfolio and other Meta-achievements; their range and number; objectivity horizontal and/or vertical professional (MO)
mobility._
Number of contacts / subscribers / followers / groups in which a student participates; likes received by him or her; number of comments to his or her posts; number of his or her comments to the posts of_
Regulativity (R)
Communicativity (C)
_other network users._
Width of interests; number of Insightfulness (I)
professional groups, in which a student participates; breadth of vocabulary; use
of sign-symbolic means, general solution patterns; performance of logic operations of comparison, analysis,
generalization, classification; analogy identification.
Meta- Abstract and Number of specialized groups in which a Meta- Abstract and
cognitivity mathematical student participates; areas of his or her cognitivity mathematical
(MC) thinking (A&M) professional activities; use of abstract (MC) thinking (A&M)
Verbal abilities notions.
(VA) Syntactic complexity of texts. Verbal abilities
Perceptive (VA)
abilities (PA) Proportion of added music, texts, video, Perceptive
etc. abilities (PA)
Spatial thinking (Supposedly it could be evaluated only in Spatial thinking
Meta- (ST) electronic learning environment). (ST)
creativity Technical Professional orientation Technical
(MCr) thinking (TT) (technical/humanities); use of special thinking (TT)
Mental flexibility, notions.
(MFx) Volume of generated content; its Meta- Mental flexibility,
diversity; number of diverse groups in creativity (MFx)
which a student participates. (MCr)
Formation of subject-oriented student groups will be carried out by default in compliance with the following cognitive abilities: visual (simultaneous), lateral, critical, divergent, and combined. Formation of educational groups of students based on their immersion in various synergetic situations proceeds from the following default types of these situations: cumulative, emergent, cognitive-bioinformative resonance. An expert has an opportunity to assign degree of complexity on non-dimensional scale A(W) for each of synergetic situations defined in the software system.
Data collection module realizes the following main functions: formation of learning and test samples of students; extraction of students' VCF from different user environments.
1) During the process of formation of learning and test samples administrator of the software system (expert) has an opportunity to set sizes of learning and test samples, specify sources from which students' VCF should be extracted. After determination of VCF for each of the subjects of the samples and calculation of evaluations of all the students' sub-meta-competencies on this basis we obtain learning and test samples
UL, {f.M}) , UT, {fjk} (4).
Software system permits the situation when evaluations of some or all VCF components for a given student were received from different sources.
2) Software system provides extraction of students' VCF from different user environments: social networking services, MOOC-environment, LMS, ELR.
Data analysis module performs calculation on the basis of the evaluations of student's sub-meta-competencies specified by VCF, normalization of calculated evaluations.
1) For calculation of evaluations of sub-meta-competencies on the basis of student's VCF we use classical additive scalar convolution of the form
fj=!>.■,;, k (5),
k
where wik is a weighting factor formalizing relative importance of characteristic feature a, j k in a row
of other components. Weighting factors are assigned by an expert according to his or her preferences, so that more "loaded" feature corresponds to a bigger value of weighting factor. We use unit values of all the weighting factors as predefined.
2) All the values Fi , fi j, a, j k are normalized as per following pattern (pattern is shown for value
F).
a) Let us assume that somehow software system's database accumulated unnormalized evaluations Fik, k e [1: |U|] of value Fi, where |U| is a size of a learning sample; i e [1:3].
6) Let us find minimal and maximal evaluations Fimm, Fimax accordingly.
b) Normalized evaluations of values Fi k, k e [1: N] we determine by formula
~ F - Fmin
F k = —-—, k e [1: N] (6).
k fmax fmin 3 L J ^ J
It should be noted that in these designations construction of scale A(F ) is reduced to division of
interval Fi""; FJ to a required number of sub-intervals, so that Amin (F i) = F i"", Amax (F^ = F imax.
Skills evaluation module performs the following functions: learning of all the used strategies for the purpose of determination of meta-competencies, including verification of learning accuracy by means of test sample and, if proved necessary, continuation of learning of the specified strategies; similar learning for the purpose of determination of learning style and cognitive abilities; learning for the purpose of determination of way of thinking; learning for the purpose of determination of student's behavior type in the specified user environments; visualization of learning results.
1) Learning of the used strategies for the purpose of determination of meta-competencies is
performed on the basis of the sample (UL, {f j k }. The following types of machine learning are employed:
supervised learning; unsupervised learning; semi-supervised learning; reinforcement learning (genetic algorithms); active learning; multitask learning [5 - 9].
-Supervised learning (classification). We use the following algorithms as classifiers: logistic regression; artificial neural networks; support vector machine; k-nearest neighbors algorithm.
-Unsupervised learning (clustering). For clustering we use the k-means method (Hartigan-Wong algorithm) and either a number of clusters set by user or an automatically determined number of clusters.
-Semi-supervised learning. We use the method of self-training with the following main concept: unlabeled data classified with a high level of confidence is added to the initial learning sample and after that classifier uses the augmented sample.
-Reinforcement learning (genetic algorithms) is performed as multi-objective optimization which is applied for classifier learning based on support vector machine. During optimization model parameters and kernel type are set simultaneously by means of genetic algorithm (NSGA-II and others).
-Active learning. Uncertainty Sampling technique is applied: questionable cases, when observation may belong to several classes, are presented to expert for labeling. After the labeling these observations are placed to the learning sample and classifier is retrained.
-Multitask learning is performed as multi-task method kNN.
2) Learning for the purpose of evaluation of learning style and cognitive abilities. Learning is carried out on the basis of the sample (UL,{fij k . Evaluation of learning style and cognitive abilities is
performed on the basis of the following types of machine learning: supervised learning; unsupervised learning.
3) Learning for the purpose of determination of way of thinking. Learning is carried out on the basis
of the sample (UL, {tk}, {ik, i.e. on the basis of previously obtained evaluations of learning style and
cognitive abilities. The following types of machine learning are applied: supervised learning; unsupervised learning.
4) Learning for the purpose of determination of student's behavior type in the specified user environments. Two learning types are used: on the basis of the sample (UL, {f j k } ; on the basis of the
sample (UL, {k}, {ik . In both cases the following types of machine learning are applied: supervised
learning; unsupervised learning.
5) Visualization of learning results. There is an augmentable range of visualization methods with the possibility of addition including method of parallel coordinates and scatter matrix method as predefined.
Module for storage of large data volumes. The module includes server for databases of study materials in text files, file repository of graphic materials, streaming data server for storage of audio and video records
Conclusion
For the further development of the study the authors plan to carry out extensive testing of designed model, methodical and software during the process of solution of real tasks concerning evaluation of students' meta-competencies on the basis of analysis of their behavior in social networking services.
This study reviews stationary aspect of the problem of students' meta-competencies quantitative evaluation. However, these evaluations change during the process of education and patterns of these changes with time contain important information on student's meta-potencies. That is why the authors also plan to review dynamic aspect of the problem of students' meta-competencies quantitative evaluation.
The study is performed with support from the Ministry of Education and Science of the Russian Federation (project2014-14-579-0144-043).
References
1. Purysheva N S, Romashkina N V and Krysanova O A 2012 On meta-objectivity, methodology and other universals Annals of N. I. Lobachevsky State University of Nizhny Novgorod 1 11-17
2. Gavrilina E A, Zakharov M A and Karpenko A P 2015 Quantitative evaluation of students' meta-competencies on the basis of the methods of machine learning Science and education 4 http://technomag.bmstu.ru/doc/764221.html
3. Company "Sociohub.ru" - URL: http://www.sociohub.ru/
4. Yakushev A V, Deykstra L Y and Mityagin S A 2012 Distributed crawler for social networks on the basis of the model Map/Reduce Data-measuring and controlling systems 11 47-53
5. Scikit-learn. Machine Learning in Python. URL: http://scikit-learn.org/stable/
6. Zhu Xhu 2008 Semi-Supervised Learning Literature Survey Computer Sciences TR 1530 (Madison: University of Wisconsin) p 60
7. Sutton R and Barto A 2002 Reinforcement Learning (Cambridge, Massachusetts, London England: A Bradford Book The MIT Press)
8. Margie M 2007 Clickers in the classroom: an active learning approach EDUCAUSE Quarterly 2 71 - 74.
9. Varnek A et al. 2009 Inductive transfer of knowledge: application of multi-task learning and feature net approaches to model tissue-air partition coefficients Journal of Chemical Information and Modeling 49 1 133-144
Поступила: 28.09.2016
Об авторах:
Гаврилина Елена Александровна, кандидат философских наук, доцент кафедры «Социология и культурология» МГТУ им. Н.Э. Баумана;
Захаров Михаил Александрович, аспирант кафедры ИУ-6 факультета Информатика и Управление Московского Государственного Университета им. Н.Э. Баумана;
Карпенко Анатолий Павлович, профессор, д-р физ.-мат. наук, зав. кафедры РК-6 (САПР) факультета РК Московского Государственного Университета им. Н.Э. Баумана;
Смирнова Елена Валентиновна, к. т. н. доцент кафедры ИУ-6 факультета Информатика и Управление Московского Государственного Университета им. Н.Э. Баумана;
Соколов Александр Павлович, к. ф.-м. н., доцент кафедры РК-6 факультета Информатика и Управление Московского Государственного Университета им. Н.Э. Баумана.
