GENERAL PRINCIPLES OF ASSESSMENT OF STAFF SKILLS BUILDING SPECIALTIES Текст научной статьи по специальности «Математика»

DOI: 10.12731/2227-930X-2016-4-63-76

GENERAL PRINCIPLES OF ASSESSMENT OF STAFF SKILLS BUILDING SPECIALTIES

StroganovD.V., Sakun B.V., YartsevM.I.

Nowadays the culture of controlling and estimating quality becoming more and more important. Estimating - is a process of gathering and processing information, which provides to members of education process opportunity to make decisions, which could help to improve quality of work. Database of estimating is bunch of governmental standards, control and estimating of efficiency systems are tests. Last 10-15 years the usage of tests is growing all over the world, that is why many companies and governmental structures appeared, which are creating new tests, promoting them, organizing mass testing and all-time mining the information, to understand the quality of education. Western countries are way far from our quality of testing of new scientific personnel, by the level of testing methods development, by the technical support of development processes, the use, handling and storage for tests. Last year's our scientists see the positive trends in this field. Strategy and tactics of controlling systems improvement. Education testing zooming and governmental standards developing are generating new problems, prerequisites and requirements, which are aimed on controlling systems perfection and trainees 'preparation quality estimating, the process itself estimating and its efficiency by immediate response on the environment.

Keywords: controlling systems; IRT theory; staff skills; test's task; markov's chain; stationary probabilities; complexity level; wrong classifications; adaptive process.

1. Introduction

Unlikely from classic tests' theory, IRT theory is pursuing of fundamental theoretical approach, and yet - finding the right solutions for practical problems [1-3, 5]. In a practical point of view it's always goes with many problems, which testers are missing sometimes.

In IRT-model we are making a formal model of the conditional probability of right execution of test's questions with different complexity by 7-th examinee with ©{ level of knowledge's, considering, that ©. - is 7-th option of 7 -th examinee, and P.. - is an independed variable. In this way -the conditional probability will be the function of latent variable p.

^,{x=1|0,}=/(0i - P) 7=1..N. (1)

In the same way we are inputing the conditional probability of execution of j-th task, with p. complexity with different examinees. © is independed variable and P.. is a paramethr, which determines complexity ofj-th test's task.

P,{Xj=1|Pj}=/(© - p.) 7=1..N, (2)

Where x.={0,1}, 1 - if 7 -th examinee answered correct on j-th test's task, 0 - if 7 -th examinee answered wrong on j-th test's task.

The conditional probability of right execution of test with P level of complexity and © level of knowledge's formula could be presented, using double-parameter A. Birnbaum model, like this [1]:

complexity i

Fig. 1. Characteristic curve of test's task

With geometrical interpretation, we could determine P. as a position characteristic of j-th task's curve relativity to © - axis. The a. paramethr is related to slope of the curve in its inflection point.

2. Problem statement

2.1. Making the markov's chain of test's control adaptive process Estimating the knowledge level by gathering the answers on test's task presents as a management task, which means, that during the next step we're getting the next task with some level of complexity. Making the "task choosing" procedure is going through the procedure of complexity choosing, by this formula:

p(»+D = Fn)(P(1),..., p(n)) + $(n)(P(1),..., Р(и)), (4)

Р(и) - complexity of task on n-th step of the procedure Fn) - some functional transformation of answers £,(n)(P(1),..., P(n)) - random value, modelating the answer on n-th task. We are going to use the finite Markov's chains apparatus, to modulate the procedure behavior [4, 6-10]. So, discrete random sequence

- is Marbv's C^n onfy if p{4+1= aw 1 y= p{^+i= aM 1 In short - with fixed "present" H, "future" B and "past" F would be independed. This property of "future" and "past" independence calls Markov's property. If "future" B={$k+1=ak+1,..., 4k+n=ak+n} n>0, "present" Н={$ = ak}, and «past» F={$ = a1 ,..., $k-1= ak-1}, then Р{В|Н^} = Р{В|Н} или Р{ВР1Н} = Р{В|Н} Р{Р|Н}.

$=(C,P,F) - determines homogeneous Markov's chain with C={c.} as set of states, a row-vector with initial probabilities F={p.} and matrix of transition probabilities P=|p..||.

Pj=p{$1=j | ... = p{4n= j | 4n-1=i}. (5)

From now on let's consider, that the amount of complexity levels is limited. Now we need to make a discrete numeric measure for it, for example b=1, 2,... . Every complexity level matches some state

of Markov's chain. Transactions between states happens at discrete time moments, which match the i-th test answer result. Could be only 2 results - "yes" - the answer is right, or "not" - the answer is wrong. Every variant of answers matches the transaction on different state, which is different complexity level. Besides, before making the Markov's chain we need to set initial distribution of complexity. It is going to match the first task's complexity. In particular case distribution will be degenerated, if you choose the exact complexity level in the beginning. Besides, it's assumed, that answers on tasks are independed values.

For example, let's look on the state transaction graph (pic. 2) [11]. As a first state, let us choose state №1 on the pic. State №1 matches the lowest level of complexity. After giving the right answer on that task, examinee gets task with level 4 of complexity. If the answer is wrong - then 2 level. If in this procedure examinee gets task with 6 level of complexity, than he gets task with 7 level of complexity, if he's right. If wrong - №3 etc. The stop criterion is a hard math task, and it doesn't consider in this post.

Fig. 2. Random Graph

For example - someone came up with this kind of a procedure. The question is - is it effective enough? To be sure, about effective part - let us use the adaptive control procedure (pic. 3) [12, 13]. In this model if the task has been solved correct - examinee gets level up complexity, if not - level down. Arcs on this picture are not the right answers, its possibility of answering correctly, which is determining from logic-line of IRT-model.

Fig. 3. Transactions of Markov's chain diagram

The result depends not only on complexity p, but on examinee's knowledge level 0 as well. After passing adaptive test, we get some trajectory, which fully matches sequence of complexity levels.

From every state of Markov's chain C, which matches the complexity level p., only two transitions are available, based on logical curve (3).

Probability of transition on higher level of complexity, if the answer is right equal:

Pi = exp

1 + exp a (0 - p;- )

(6)

Probability of transition on lower level of complexity, if the answer is wrong equal:

<7, =1/

1 + exp -(0-p,)

(7)

So, as a pattern model of adaptive test's control procedure we get MC §=(C, F, P), where:

C={C.} - set of states of Markov's chain, which match the complexity level of tasks.

F=|[p.|| - row vector of tasks' initial distribution.

^=1^11 - matrix of transition probabilities, which determines by the transition diagram and probabilities determine by proportion of (5) and (6) (tab. 1).

Table 1.

Structure of transition probabilities matrix

i \ J 0 1 2 3

0 ?0 P0 0 0

1 ?i 0 Pi 0

2 0 0 P2

3 0 0 0

I-1 0 0 0 0

I 0 0 0 0

I-1 I

0 0

0 0

0 0

0 0

0 PI-1-

q 1n Pn

Because of transition diagram structure, we can see, that this is a reduced chain. That's why there is only one stationary probabilities distribution, which is not related to the initial state.

2.2. Stationary probabilities the Markov's chain Let's mark stationary state probabilities C. as n With these notations, row vector of stationary probabilities k = (k0,k^, k2 ,—, tc/_1 , Kj) determines as a solution of this equation system:

7T = 7T P, =1. (8)

Now we describe this system in scalar type for several strokes of this matrix.

jTo^o-qo+Jtrqi TCi=7KrPo+Ji:2'q2 7t2=7Ii-pi+7I3-q3

Solving this relatively on zero-state we get this:

For a first one

For a second one

1 1 I

7I2 =-(jl1 -JIQ -p0) =- JI0 •

1 -q0

■ no ■ Po =

q2 q2]

" 1 " " 1 q2q, 1 0 q^2

(10)

(11)

Q2I Q1 Q1 For a third one

1 1

— (TT2-tvPi) =

W3 43

"o fP0P1 P0M2

™3 =— (n2-n1 -Pi)= H ™0 'TL-Pi i =

<?2

<72

q i«?2

nM(l-q2)=n0-PM

(12)

Then we say, that stationary probability for 7-th state equal

71; = Tin

' P

,0 .n ^^ And this proportion is correct:

k=1

<7*

"mPm + *;+i<7n+i = PM^otl — + f[ — :

k=1 Qk k=1 Qk

Plr-1

/"-1

DÛ

fc=1

Pi-1 + C/n+1

Pfc-1 Pfc-1

i-1

TT T~r Plr-1

"oil-

k=1 Ik

Pi-1 + ln+1

Pfc-1 Pk-1

Therefore, the solution of stationary probability equation system is:

M

«=1 Qk+1

Tin =

1 j-1

1 + Zn

p*

V j=o /(=1 Qx+1 Using transition probability equation (6) and (7), we expressed through logical function(£/:= l-^(ß.©). i = V, (¥(•) - random logical curve), we get:

__(i. _ _ exp(©-ßj n_1

1 \I/ n \ "'ill , «u-niö R "l ' <1

' 'iil-Tfe-ßJ 'Ul + e*p(©-ßJ

1 + exp(0 - ßA )

= it? fi1exP(& ~ ß* ) == exP I (® " ß* )

Moreover, in the end we get n, = n0 expj £ (© - pfc ) j, i = 1,1.

Where values are probability distribution of set of Markov's chain states. This stationary distribution has been gotten by introduction some discretization in complexity space [14, 15]. We can see graphics of stationary distribution of different level of knowledges on the picture 4.

a) b) Fig. 4. Stationary distribution of probabilities

In the end we can see, that maximum of MC stationary probabilities falls on test's task complexity value, which match relevant knowledge level, which gives us a lot of information about knowledge estimation, because the answer probability is A.

2.3. Estimating of test's control adaptive algorithm To estimate the algorithm we need to make a procedure, which is going to classify on K groups, where Wx, W2,..., Wk matches some category. First we need to create a model of a right answer, which depends on complexity level of the task (the result is matrix m*£, where m - amount of test's repeats). Probability of a right answer determines by Bernoulli distribution with probability p:

||z,J = P(Qq,Vj | p), i = \...m,j = (15)

р - simulation model's option vector, n - amount of tasks in test. To prove, that answers of examinee are independed - we multiply independed events:

m fllz,- -IL если X = U

A = K =п , II II v ,,, (16)

,=i 1 - \\Zjj L если X = -,U

U - is a reliable event.

As a second step - we will use the maximum likelihood algorithm.

Consistently, we will go thruogh the 7-th line of a matrix, and finding

the maximum of column, which index will be determined as value

of Ш = arg max (Aqll]). We are forming L column-vector, which

" " j еП.к] 4

elements are value of estimating the likelihood algorithm by every

<| m

iteration. \\hk II = — У1,.—,, , is normalized vector L . In the end, we make

m^ I''p* q

a matrix = HTq. which elements are value of estimating the likelihood algorithm with k*n size.

Basing on this procedure, we made a comparative analysis between the procedure we have just created and static making tasks procedure. On table 2 we can see the result of simulation experiment for seven-level complexity test, with 5 tasks on each level.

Table 2.

Static plan of test with 5 tasks each complexity level

Class 1 2 3 4 5 6 7

1 ,822 ,155 ,002 0,000 0,000 0,000 0,000

2 ,176 ,700 ,125 0,000 0,000 0,000 0,000

3 ,002 ,145 ,775 ,185 ,002 0,000 0,000

4 0,000 0,000 ,097 ,636 ,100 0,000 0,000

5 0,000 0,000 ,001 ,179 ,757 ,152 ,002

6 0,000 0,000 0,000 0,000 ,139 ,686 ,169

7 0,000 0,000 0,000 0,000 ,002 ,162 ,829

The results for adaptive algorithm with same amount of tasks and with a same model are represented on table 3.

Table 3.

Adaptive plan of test with NT=35

Class 1 2 3 4 5 6 7

1 ,906 ,085 0,000 0,000 0,000 0,000 0,000

2 ,094 ,821 ,090 0,000 0,000 0,000 0,000

3 ,000 ,094 ,818 ,098 0,000 0,000 0,000

4 0,000 0,000 ,092 ,803 ,090 0,000 0,000

5 0,000 0,000 0,000 ,099 ,813 ,102 0,000

6 0,000 0,000 0,000 0,000 ,097 ,804 ,090

7 0,000 0,000 0,000 0,000 ,002 ,0904 ,910

After analyzing these two tables we can make a decision, that classify by adaptive way gives better results for each level of examinees' knowledge level. To present the matrix of wrong classification, we'd better use the graphic form:

a) Static test b) Adaptive test

Fig. 5. Graphic presentation of wrong classifications matrix

Wrong classifications matrix is a pairwise probability of belonging to different knowledge levels function, which always will have a saddle-node type. Graphics of wrong classification probabilities' are uneven, because of static error of simulation experiment.

Fig. 6. A graphical representation of a matrix of incorrect classifications

Dotes of curve on the picture are estimated test's efficiency. As low the dot is as low the probability of error classify, as more efficiency test is. This model has been using to estimate tests with different complexity level. Easy tasks (L), medium tasks (M) and difficult tasks (D), and for estimating the adaptive test (A). As we see on graphics - easy test do not do good with high level of examinee's knowledge, and difficult one - do not do good with low level of knowledge. In addition, as for adaptive tests - probability of wrong classify is much less for any level of knowledge of examinee.

3. Conclusion

We made an analysis of mathematical expectation estimating by belonging to the class [16, 17]. One of the problem here is finding the right answers probability function, because on the low level they are almost not distinguishable. In 9-level knowledge model - adaptive algorithm is even doing better at wrong classify probability. Nevertheless, when model is 3-4-level - win is minor.

Because of making tests control model - we proved the efficiency

of test's control procedure.

References

1. Birnbaum A. Some Latent Trait Models and Their Use in Statistical Theories of Mental Test Scores. Reading, Mass.: Addison Wesly, 1968.

2. Chernen'kii V.M. Forming simulation process based on algorithmic description of information systems // Science and education, MSTU. NE Bauman. Electron. Zh. 2011. № 11. URL:

3. http://technomag.edu.ru/doc/291975.html

4. Nikolaev A.B., Solntsev A.A., Sanaya A.G., Yakunin P.S. Organization and modeling techniques dealer networks // In the world of scientific discoveries number 12 (36). Krasnoyarsk: SIC, 2012, pp. 163-174.

5. Lychkina N.N. Simulation modeling of economic processes // Textbook Series Higher Education. INFRA-M, Moscow, 2012, 254 p.

6. Malikov R.F. Basics of development of computer models of complex systems // Study Guide, ISBN 978-5-87978-829-7. Publishing house of the Belarusian State Pedagogical University, Ufa. 2012, 256 p.

7. Ivakhnenko A.M., Akhokhov A.Ch. Modeling of quality control of technological processes and industrial output. Monograph// Moscow: "Tekhpoligraftsentr". 2008. 146 p.

8. Karasyov A.A., Stroganov D.V., Solntsev A.A., Yakunin P.S., Batov R. V., Influence of entry conditions and duration of modelling on characteristics of conditional and non-stationary processes//Science and education. MGTU of N.E. Bauman. Electronic journal. 2012. No. 4.

9. Karasyov A.A. Dynamic configuration of system of imitating modelling / Karasyov A.A., Prikhodko L.V., Falcon A.A., Yartsev M.I. // Imitating modelling of control systems. M.: MADI, 2012, pp. 7-12.

10.Nikolaev A.B. Monitoring and modelling of operating modes of transport and technological machines / Nikolaev A.B., Pashayev M.Ya.,

Smimov S.Yu., Atayeva S. K.//In the world of discoveries. 2015. No. 6 (66), pp. 9-18.

11. Nikolaev A.B., Stroganov V.Yu. Automated information processing systems and controls on the motor transport; the textbook for secondary vocational education. Under the editorship of A.B. Nikolaev. Moscow: Publishing centre "Academy", 2003. 224 p.

12.Nikolaev A.B. Analitiko-simulation modeling of technological processes of production, service and repair of road-building technique: Monograph / Nikolaev A.B., Dyomin V.A., Eskova I.A., Okunkova A.A., Prikhodko M.V. // Saint-Louis, MO, USA: Publishing House Science and Innovation Center, 2014. 242 p. ISBN 9780615671277.

13.Nikolaev A.B. Automation of planning and control in construction and on transport. Volume 1. Automation of planning when building distributive objects: Monograph / A.B. Nikolaev, A.Ch. Akhokhov, D.V. Zaytsev, S.N. Satyshev, V. Yu. Stroganov. Saint-Louis, Missouri, USA: Publishing House Science and Innovation Center, 2015. 330 p. ISBN 978-0-615-67148-2.

14.Nikolaev A.B. Automation of planning and control in construction and on transport. Volume 2. Control automation information software: Monograph / A.B. Nikolaev, B.V. Sakun, A.A. Solntsev, P.A. Timofeev, L.A. Hvoinsky. Saint-Louis, Missouri, USA: Publishing House Science and Innovation Centre, 2015. 436 p. ISBN 978-0615-67148-2.

15.Nikolaev A.B., Demin V.A., Eskova I.A., Prikhodko M.V. Using multilevel simulation models of regenerating processes when evaluating rhythmicity of production processes//International journal of advanced studies: Publishing House Science and Innovation Centre, Ltd. (St. Louis). ISSN: 2328-1391. No.: 3 (3), 2013, pp. 10-16.

16.Nikolaev A.B. Ismailov A., Lvova A., Ostroukh A.V. Organization and Management of Transport Provided for the Guests and Participants of the Olympic Games//Middle-East Journal of Scientific Research. 2013. Vol. 17 (8), pp. 1098-1104.

17.Nikolaev A.B., Prikhodko V.M., Stroganov V.Y., Tregubov P.G. Tools of production and logistics support life cycle of high-tech products // Life Science Journal 2014; 11 (10s), pp. 238-242. 18.Shannon R. Imitating modeling of systems - art and science. M.: World, 1978. 655 p.

DATA ABOUT THE AUTHORS Stroganov Dmitriy Victorovich, Doctor of Technical Sciences, Vies Rector

State Technical University - MADI

64, Leningradsky prospekt, Moscow, 125319, Russian Federation stroganov@mail. ru

Sakun Boris Vladislavovich, Candidate of Technical Sciences

General Society with limited liability "TSM"

26 a, Khamovnichesky Val, Moscow, 119048, Russian Federation

info@tsm-msk. ru

Yartsev Maksim Igorevich, Postgraduate Student

State Technical University - MADI

64, Leningradsky prospekt, Moscow, 125319, Russian Federation gizmo1654@gmail. com