CC BY
27
A. Bochkov, N. Zhigirev, A. Kuzminova RT&A, No 3 (69)
INVERSION METHOD OF CONSISTENCY MEASURE ... Volume 17, September 2022
Inversion Method of Consistency Measure Estimation
Expert Opinions
A. Bochkov1*, N. Zhigirev2, A. Kuzminova3
^Deputy Head of Integrated R&D Unit, JSC NIIAS, 107078 Moscow, Russia e-mail: a.bochkov@gmail.com 2KALABI IT, 107045 Moscow, Russia, e-mail: nnzhigirev@mail .ru 3 Associate Professor, Department of Computer Systems and Technologies, Institute of Intelligent Cybernetic Systems National Research Nuclear University, MEPhI, 115409 Moscow, Russia e-mail: avkuzminova@mephi. ru
Abstract
The problem of collective choice is the problem of combining several individual experts' opinions about the order of preference of objects (alternatives) being compared into a single "group" preference. The complexity of collective choice consists in the necessity of processing the ratings of the compared alternatives set by different experts in their own private scales. This article presents the author's original algorithm for processing expert preferences in the problem of collective choice, based on the notion of the total "error" of the experts and measuring their contribution to the collective measure of their consistency. The presentation of the material includes the necessary theoretical part consisting of basic definitions and rules, the statement of the problem and the method itself based on the majority rule, but in the group order of objects.
Keywords: collective choice, permutation, group, inconsistency, inversion, graph, rating, Schulze method, skating method, Pareto-optimal solutions.
1. Introduction
In practice, the efficiency of decision-making requires the development and application of specialized algorithmic and methodological support. If a group of experts participates in the decision support process, the so-called collective (group) choice problem arises. The existing algorithms for solving collective choice problems [1-3] can be divided into three classes.
A representative of the first class is the Schulze method [4] (based on the proof of the Arrow theorem) with the selection of Pareto-optimal solutions (Schwartz exception) from the first ranking to the last, with the selection recalculating the criteria for the next step. The disadvantage of the method is a rather complicated algorithm of constant recalculation, which significantly complicates the practical use of the method.
A typical representative of the second class is the skating-system well-proven in ballroom dance competitions [5]. It is simple in computational calculations and is based on the so called
understandable majority principle. Unfortunately, in many ways, this simplicity can lead to unstable decisions, and therefore, the impossibility to distribute the final places among the competitors in one round, or recognize a draw between competitors [6, 7].
The third class consists of regression models, type nonlinear factor analysis and other methods of information compression [8, 9], in which the desired solution is constructed in the form of the problem of minimization of accumulated errors. The difference between the methods of the third class is that they are not focused on the choice of the leader in the ratings, but are determined by the optimum, which is influenced by the entire volume of data.
The mentioned methods of solving the problems of collective choice in general are inherent to the problem of coordinating the experts' evaluations when comparing the evaluated objects.
In 1951 C. Arrow formulated [10] the theorem "On the impossibility of collective choice within the framework of the ordinality method", mathematically generalizing the Condorcet paradox [11]. The theorem states that within the framework of this approach there is no method for combining individual preferences for three or more alternatives, which would satisfy some quite fair conditions (the axioms of choice) and would always give a logically consistent result.
When ambiguous expert opinions are superimposed on the uncertainty of the objects themselves, some hierarchy is assumed in solving the choice problem. This is the case, for example, in the method of hierarchy analysis [12], when each of M of experts has his/her own, different from the others, opinion concerning the weights of the objects under consideration N objects through the
coefficients of the preference matrix (S™ = (i = 1, ...,N; j = 1,...,N ;i & j;m= 1,...,M)).
Usually, weights are averaged and work with a generalized matrix S^ this usually leads, as a rule, to a violation of the basic axioms of the "right" choice (universality, completeness, monotone, lack of a dictator, independence) proposed by W. Pareto [13, 14], R. Koch [15], C. Plott [16] and others. The rejection of one or another averaging procedure complicates the choice problem and leads, for example, to the need to solve the problem of "merging multidimensional scales" [17].
Earlier [18], the authors argued that to obtain consistent decisions, experts need to reach consensus, at least within the accuracy of determining private ratings in the full order of objects, and then seek agreement in the weighting coefficients between the neighboring nearest objects, setting a single scale. In this article we consider a method belonging to the third class of algorithms in decision theory, aimed at finding the optimum of the consistency measure, the restoration of the full collective order in preferences based on private ratings of experts.
2. Basic definitions and rules
Let us introduce several basic definitions.
Definition 1. Arbitrary mutually one-valued mapping g:X ^ g(X) of multiple first N natural numbers X = (1,2,3,...,N) is called a permutation Nof row (permutation):
X= { x% = 1 x2 = 2 ... xN = N }
g: I I III
g(X)= { g% = g(x%) g& = g(x&) ... gn = g(xn ) }
The set G = {g} forms a group of dimensionalities N!.
Definition 2. An inverse permutation to g is defined as (g ~%{j) = k) ^ (g(k) = j) Vj, k.
An example of all permutations for N = 4 is given in Table 1. In principle, for any N each permutation can be assigned an index in the lexicographic order (LG-order) of values g(X) (columns 1, 2 in Table 1).
Definition 3. The first permutation by index which is equal to the unit permutation in the group %g = (1234) = E we will call "true" or natural order. The last permutation with ordinates in reverse
order: 24g = (4321) = E- the complete inversion, which is at the last determinable level N(N — 1)/2.
Definition 4. For a permutation of g = (g% g2 g*„gN) a pair of indices (gi, gj) is called an inversion [19] if (i < j)&{gj > gt).
Definition 5: A table of permutation inversions a is a sequence of numbers {b% b2 ... bN} where bj - is the number of elements greater than j and to the left of j. In other words, bj - is the number of inversions with the second term equal to j.
Table 1. Full table of permutations for N = 4
Index g Numeric code g in the LG order Root synonym g Synonym g, based on in versions, [other synonyms of minimum word length] Error level, word length Reverse reinstallation Inversion Tables, their sum, 2
1 2 3 4 5 6 7 8 9 10 11
1 <1234> E E, [-] 0 <1234> 0 0 0 0 0
2 <1243> c c, [-] 1 <1243> 0 0 1 0 1
3 <1324> b b, [-] 1 <1324> 0 1 0 0 1
4 <1342> cb cb, [-] 2 <1423> 0 2 0 0 2
5 <1423> bc bc, [-]. 2 <1342> 0 1 1 0 2
6 <1432> bcb cbc, [-] 3 <1432> 0 2 1 0 3
7 < 2134> a a, [-] 1 < 2134> 1 0 0 0 1
8 < 2143> ac ac, [ca] 2 < 2143> 1 0 1 0 2
9 < 2314> ba ba, [-]. 2 < 3124> 2 0 0 0 2
10 < 2341> cba cba, [-] 3 < 4123> 3 0 0 0 3
11 < 2413> bac bac, [bca]. 3 < 3142> 2 0 1 0 3
12 < 2431> bcba cbac, [cbca] 4 < 4132> 3 0 1 0 4
13 < 3124> ab ab, [-] 2 < 2314> 1 1 0 0 2
14 < 3142> acb acb, [cab]. 3 < 2413> 1 2 0 0 3
15 < 3214> aba bab, [-] 3 < 3214> 2 1 0 0 3
16 < 3241> acba cbab, [caba]. 4 < 2413> 3 1 0 0 4
17 < 3412> bacb bacb, [bcab] 4 < 3412> 2 2 0 0 4
18 < 3421> bacba cbacb, [cbcab, bcbab, bcaba] 5 < 4312> 3 2 0 0 5
19 < 4123> abc abc, [-] 3 < 2341> 1 1 1 0 3
20 < 4132> abcb acbc, [cabc]. 4 < 2431> 1 2 1 0 4
21 < 4213> abac babc, [abca]. 4 < 3241> 2 1 1 0 4
22 < 4231> abcba cbabc, [acbca, cabca, cabac] 5 < 4231> 3 1 1 0 5
23 < 4312> abacb bacbc, [babcb, bcabc, abcab] 5 < 3421> 2 2 1 0 5
24 < 4321> abacba cbacbc, [cbcabc, bcbabc, bcabac, bcabca, bacbac, bac bca, babcba, abcaba, abcbab, acbcab, acbacb, cabcab, cabacb] 6 < 4321> 3 2 1 0 6
Definition 6. For any g there exists a set of AT(g) (English, Adjacent Trans position) - "adjacent, neighboring" permutations, the number of which is exactly (N — 1). All the edges E x AT(E) consist of the forming elements of the group G. The elements of the multiplicity of formants E x AT(E) can be regarded as symbols s of some alphabet A (Table 2).
Table 2. Nodes of the 1st error level consist of one symbol of the alphabet A
In order, adjacent to the truth E Error level Alphabet A The node of the 1st error level is a word of one symbol A
0 E = ExE
1 1 a a = Exa = (2,1,3,4,...,N -1,N)
2 1 b b = Exb = {1,3,2,4, ...,N -1,N)
3 1 c c = Exc = {1,2,4,3,...,N -1,N)
1
N - 1 1 z z = Exz = {1,2,4,3,...,N,N -1)
z - conditional symbol (N — 1) of the formant. For N = 4, Ex AT(E) = {a, b, c}, consequently: z = c.
Definition 7. A weighted graph of a group V(G, G x G) consists of nodes G and the weight of an edge (g% x g2) is equal to s, when (g2 E AT(g1))&(g2 = g1s).
The structure of the graph V(G, G x G) is determined dynamically by the error levels. At the upper (zero) level there is only a single permutation E. At the second and further levels there are only nodes formed by joining only one symbol of the alphabet A.
The parity property of permutations is noteworthy: aa = bb = cc = ■■■ = zz = E, because of which the graph of the group can be treated as an undirected graph.
Definition 8. Each g can be interpreted as a path, or some sequence of directed segments of the graph 7and vice versa.
The way from v E G в v' EG passes through the edges connecting neighboring permutations and is equal to v' = vs% • ...• sT where s% ...sT - symbols of the alphabet A, T - is the length of the word.
Definition 9. The set of all finite words S over a finite alphabet A is countable. Hence, each nonzero word can be assigned an index q.
2 T$
s = , Sq = \
q=l
where Tq - is the number of symbols in the word Sq.
For each word Sq = (5/ • ... • S/$} there is one inverse word Sq*: Sq* = js/ = s/q;... Definition 10. The words Sq u Sk - are synonymous if s/ • ... • s/q • s% • ... • s%&* = E.
= sl }•
Definition 11. Among identical synonyms we can distinguish a finite set of minimal words in length Tmin from the node g = v' in "truth" E (v~% = E): g = v~%v' = s%... sT - (synonym(g)).
Definition 12. Among the words from (synonym(g)) there is a "root synonym" with a minimal form of LG-order based on the order of elements in A.
A root synonym is a word derived from a numerical code by the "bubble" sort algorithm [20] when moving toward the "truth" E. A different way of obtaining root synonyms is presented in Table 3. It proceeds from the method of sequentially destroying inversions followed by transforming a sequence of synonyms from the current synonym to the root synonym using group-forming equations (see Definition 13).
Each permutation g has exactly one root synonym of length T(g), coinciding with dynamically determined error rate and total number of inversions in the table of inversions - £ (Table 3). In our case for <4321> it is a word of 6 symbols "abacba".
Table 3: Algorithms of building synonyms by different methods
Method of sequential inversion reduction ("first" optimal in word length synonym)
Error level Code 1 2 3 4 E Error level Code 1 2 3 4 E
6 4321 3 2 1 0 c 6 6 4321 3 2 1 0 a 6
5 4312 2 2 1 0 b 5 5 3421 3 2 0 0 b 5
4 4132 1 2 1 0 a 4 4 3241 3 1 0 0 a 4
3 1432 0 2 1 0 c 3 3 2341 3 0 0 0 c 3
2 1423 0 1 1 0 b 2 2 2314 2 0 0 0 b 2
1 1243 0 0 1 0 c 1 1 2134 1 0 0 0 a 1
0 1234 0 0 0 0 0 cbacbc 0 1234 0 0 0 0 0 abacba
Bubble sort algorithm (optimal word-length "root" synonym)
When the numerical code coincides, a network structure of the graph V is formed when the neighboring nodes are at a distance of one on the inversion level.
( 1234 )
S <19> \ ( 4123 )
X-abcV
\ " \ c
l 41 32 1 V abcb J
Figure 1: Structure of the graph for N = 4
0
1
2
3
4
b
5
6
Fig. 1 shows the structure of such a graph V for N = 4. The thick and thin edges represent the symbols s, in layers participating in generation of new neighboring nodes. The content of filling nodes is a representation of the permutation index g (column 1 in Table 1), its numerical code (column 2) and the content of the root word symbol for permutation g (column 3). The thick lines of the graph V in Fig. 1 correspond to its representation as a dictionary of root synonyms of permutation in the form of a spanning tree graph V. Their direction coincides with whether there is an inversion (up) or not (down) at the specified place. The six end nodes of the tree are represented by elements with yellow filling, such as {<19>; < 4123>; abc}. The underlined inversion, for <19> only (a) brings the next node closer to the "truth" E, reducing the number of errors by exactly one.
The dictionary of root synonyms is built according to the following principle: only those relations between permutations that are older in LG-order remain in cycles. For example, the cycle <1><7><8><2><1> (a) we break by the connection < 2 >< 8 > since the connection < 1 > <7 > (Fig. 2a) is smaller than connection < 1 >< 2 > (c).
A similar operation must be repeated for the lower sections of cycles <6>, <14>, <15>, <12>, <16>,
<20>, <21>, <18>, <22>, <23> and twice on <24> to break 12 more cycles and form the tree. For N = 4 analyzing the equalities describing the right and left branches of the cycles, we come to the necessity and sufficiency of six equalities: = E, bb = E, cc = E, ac = ca (Fig. 2a), aba = bab, bcb = cbc, (Fig. 2b), which are necessary to construct synonyms of words in permutations (Table 1, columns 3,
4).
a) b)
Figure 2: Illustration of breaking network cycles
For N> 5 such enumeration of relations is difficult, so it is reasonable to introduce the notion of canonical formative equations (CFE).
Definition 13. The canonical formative equations are a necessary and sufficient list of equations that fully specify the rules for constructing root and other synonyms through the alphabet A.
Thus, in Table 3 it is possible to perform conversions with the help of CFE: "cbacbc" = "cbabcb" = "cabacb" = "acbcab" = "abcbab" = "abcaba" = "abacba".
Table 4 shows regularities that take into account the parity laws of permutations and lozenge closures at the 2 error level.
The sign "*" in Table 4 marks the necessary inverse permutations of the 2nd level of inversions. For example, if the left-hand side of the equations "ac" sets the equivalent right-hand side of "ca" then you must replace "ca" with "ac", etc. By adding third-level inversion relations versions: aba = bab; bcb = cbc; cdc = dcd; ded = ede; ...; xyx = yxy; yzy = zyz, we define a complete set of CFE.
Table 4: Automatic construction of CFE at 1-2 error levels
The second argument
a b c d e x y z
a E ac* ad* ae* ax* ay* az*
b E bd* be* bx* by* bz*
c Ë IM c ca E ce* cx* cy* cz*
d da db E dx* dy* dz*
?H CO -M e ea eb ec E ex* ey* ez*
M _5H îJH
X H x xa xb xc xd xe E xz*
y ya yb yc yd ye E
z za zb zc zd ze zx E
3. Problem statement
Let us consider N comparison objects 0i,..., Ok ,... , 0N which indices are the first N members of the natural series EP0I = (1, ...,k, ...,N) - correspond to the order of presentation of the objects for the expertise. In the examination of objects participate M experts Ei, ...,Em ,... ,EM. Each of the experts Em has his own idea of the order of objects gm = (gm%,..., gmn, ..■, gmN) which indexes increase with decreasing of some quality of objects from the expert's point of view. The value gmi corresponds to the index of object 0kl, taking part in examination with maximal quality according to expert's opinion Em, a gmN - the worst-quality object with the index 0k( :
(9i,i ..• 9i,n\ ... ■ ... I
9m,i — 9m,n)
Thereby gm - it is a permutation of object ratings (POR), the argument of which is the order of epor = (1, ..-,n, ..■,N).
Places pm = (pmi,..., pmik,..., pmN) by values inverse to POR gm (pm = g—i) are permutations of object indices (POI) with argument EP0I :
P 1Pm,nn)
m=l,M n=iN
Pi,i = 9il
\Pm,i = 9m%n
Pi ,n = 9i,N
_ —i
Pm,n = 9m,n )
It is necessary to find the compression of all private POR rankings gm (m = 1, ...,M) in the form of a POR g*m = (g%*,..., g*N) which would reduce the total inconsistency of expert evaluations gmn ^ g# (based on the equality of all participants in the examination), measured in the inversions of the transitions from gmn k g#, that is
K* = min K{g) = miniy Km{(9i ,...,9n
where Km{(g1 ,...,gN )) - is the sum of inversions in the evaluations of the m expert, K*- is the limiting measure of inconsistency of experts' opinions chapter.
Finding an optimum in permutations of object rankings is equivalent to finding an object index over p*: p* = (p*,..., p*N), since K(g**l) = K(p*m )where p*=(g*)~% (the lengths of the reciprocal paths (E ^ g) are the same as the forward paths (p = g~% ^ E) at any g ) (Table 5).
Table 5: Solution search table P(g) with table of inversions B(g, P)
POR Arg E 1 n N POI Arg E i k N Criterion inconsistencies
POR Func g gi an a# POI Func p .QT1 v-%1 Qn1
POI Arg E 1 k N POI Arg E 1 k N
1 Piia) Pxa! Pi'" Pl,'n Bi{vi(a)) Bi.1 Bit n Ki(g) = 0 BXk k=1
m Pm(a) Pm,'! Pmau Pm,'N Bm{Pm(a)) Bi.1 Bm,k N Km(d) = 0 &m,k k=1
M PM(g) P*,gi P*,'k Pm,'N Bm{Pm(3)- BM,1 B*,k N KM(d) = 0 BM,k k=1
M K(g) = 0 Km(g) m=1
4. Method description
This problem belongs to the class of integer programming problems (on the structure as a graph -V(G,G x G) POR graph, arranged by error levels). Methods for solving such problems are well developed [21, 22], but none of them guarantees that, starting with some permutation, we will
certainly get into a global minimum, which may not be the only one. At the very least, what can be guaranteed is a complete search of all POR. This option is possible for N < 10. For each g counts K(Pm, g) and the sum of K(g), and the current state of the set of global minima is "memorized".
A subset of g E G for which K(g) = K*, we call the set of global minima - GK. Since M is odd, it, like the set of local minima, consists of isolated solutions (permutations).
Let us consider a pair of (I, I + 1) columns in P(g). I = 1,... ,N — 1 corresponds to the symbol s? of the alphabet A (Table 6).
Table 6: Neighboring Pair Table P(g)
Expert POR g ai ai+i
i i + i
1 Pi(g) Pi,m iE) pi,gi+1iE)
m Pmia) Pm.gi (E) Pm,g++1 (E)
M pMia) PM,gi (E) PM,gi+1 (E)
Rule 1. If Pm g? (E) < Pm g?+1(E), then the sum of inversions K(gm) is increased by 1, and if Pm 9i(E) > Pm g?+i(E), the sum of inversions decreases by 1.
Rule 2. The decrease and increase of the sum depend on the number of rows in which the second condition M& (Rulel) dominates the first condition M1. The ratio is M1 + M& = M. Then the sum K(g) from the influence of s? will decrease by exactly M& — M1 units (if M& > M/2) or increase by M1 — M& units (if M& <M/2).
Rule 3. "Cutoff condition." The POR g belongs to the set of local minima Gp if for all j = 1,... , N — 1 the sum of errors only increases with rotation of neighboring columns by the symbol Sj. That is g E Gp has neighboring nodes of the graph V, exceeding by sum the found local optimum g by at least one.
The search Gp makes sense with large N, but with small N it is also effective, since the decrease (increase) of some selected pair does not depend on the place where the pair is located, but only on the contents of the resulting inversions.
Depending on the number of compared objects (N) two variants of the range are possible.
Variant 1. "Direct calculation". At small N (N < 6) it is possible to create a "directory" in LG-order. Then to build Gpand GK it is necessary to exclude from it the POR where the "cut-off condition" is not satisfied. Calculate for all g E Gp value K(g) and choose the optimal one.
Let us explain the above on the example for N = 4 (Table 7).
Table 7: Initial data P(E) and optimality criterion calculation K(E)
POI E 1 2 3 4 Inversion Tables KmiE)
Oi o2 0s 0* 1^1 2^2 3^3 4^4
Ei Pi 1 4 2 3 0 1 1 0 2
E( P( 2 3 1 4 2 0 0 0 2
Es Ps 3 2 1 4 2 1 0 0 3
E* P* 4 2 3 1 3 1 1 0 4
Es P+ 1 4 3 2 0 2 1 0 3
E, p, 2 4 1 3 2 0 1 0 3
Ey Py 2 1 4 3 1 0 1 0 2
Optimality criterion K(E): 20
Let us create a matrix of full pairwise comparisons of columns for the POI P(E) for (i = 1,N; j=1,N; j) (Table 8).
Table 8: Results of counting inversions on a pair of columns
X 1 2 3 4
1 X 3 4 1
2 4 X 5 4
3 3 2 X 3
4 6 3 4 X
A fragment of the calculation for (i = 1; j = 2,3,4) is given in Table 9.
Table 9: Checking the "cutoff condition" (sum of column inversions (i = 1)
1 —» 2 1 —» 3 1 — 4
1 4 1 2 1 3
2 3 2 1 1 2 4
3 1 2 3 1 1 3 4
4 1 2 4 1 3 4 1 1
1 4 1 3 1 2
2 4 2 1 1 2 3
2 1 1 2 4 2 3
3 4 1
Table 8 shows that the "cutoff condition" is not satisfied by 6 pairs of columns: 1—3, 2—1, 2—3, 2—4, 4—1, 4—3. As you can see, for the N = 4 directory g (Table 1) will contain 24 POIs. At the level of inversions (a) from 24 indexes values 12 elements with indexes (3-4, 7-12, 19-20, 23-24) will be discarded (due to "wrong pairs"), at the level (b) - 7 POI with indexes (1, 6, 15-17, 21-22), at the level (c) - 4 POI with indexes (2, 5, 13, 18). As a result, Gp and GK consist of one POI with the index 14g = g* = (3 1 4 2) for which we will further calculate the value of the optimal criterion (Table 10).
Table 10. POR optimum g* and calculation of the optimality criterion K(g*)
POR g* 3 1 4 2 Inversion Tables Km(g*)
0) 0# 0* o2 1—1 2—2 3—3 4—4
E# P# 2 1 3 4 1 0 0 0 1
E( P( 1 2 4 3 0 0 1 0 1
E) P) 1 3 4 2 0 2 0 0 2
E* P* 3 4 1 2 2 2 0 0 4
E+ PS 3 1 2 4 1 1 0 0 2
E6 P, 1 2 3 4 0 0 0 0 0
Ey Py 4 2 3 1 3 1 1 0 5
Optimality criterion K(g*): 15
As can be seen from Table 10, expert E6 "guessed" the optimal solution K6(g*) = 0. Experts E1 and E2 made only one error each, E3 and E5 made two errors each, and E4 and E7 made too many errors. The next step is to use the POR g* to reconstruct the optimal POI p* = (g*)~%. Consequently, the required places p*(E) = (2,4,1,3) .
Variant 2. "Iterations." In general, you can use a cutoff rule directly starting with some starting POI, e.g. from g = EP0I. The complete absence of cutoff guarantees that the local minimum is found in the 3g = bac (Table 11).
Table 11.
1 2 3 4 5 6 7 8 9 10
9 °g =e Km(0g) 1g = b 1g) 2g = ba 2g) 3g = bac 3g)
(1234) (1324) (3124) (3142)
E# P# 1423 2 1243 1 2143 2 2134 1
E( P( 2314 2 2134 1 1234 0 1243 1
E) P) 3214 3 3124 2 1324 1 1342 2
E* P* 4231 5 4321 6 3421 5 3412 4
E+ P+ 1432 3 1342 2 3142 3 3124 2
E, P, 2413 3 2143 2 1243 1 1234 0
E- P- 2143 2 2413 3 4213 4 4231 5
a=3; b=5; c=3 20 a=4; b=2; c=4 17 a=3; b=3; c=4 16 a=3; b=1; c=3 15
The presence at the end of iteration (a=4; b=2; c=4) of ambiguity of choice makes us return to the beginning of this stage and consider another alternative z+g = be c K(2+g)=l6 and conclude that the search is terminated because 3+g = bca c K(3+g)=l5 is a copy of {3142) by CFE.
5. Concluding remarks
It is beyond the scope of this article to compare the proposed method with other methods of information compression (e.g., factor analysis, the averaging method, or the Schulze method), which will be discussed later.
The further development of this method implies its application in ranking determinations that allow equality of evaluations of compared objects when determining the weights of compared objects (similar to pairwise comparisons in the method of hierarchy analysis [12] and solving problems of heterogeneous scales merging [17, 18]).
References
[1] Kemeny J., Snell J. Cybernetic modeling. Some Applications. Moscow: Soviet Radio, 1972. -192 p.
[2] Larichev O.I. Theory and Methods of Decision-Making, and Chronicle of Events in the Enchanted Lands. 2nd edition, revised and enlarged. Moscow: Logos Publisher, 2002. - 382 p. - ISBN 5-940l0-l80-l.
[3] Kaplinsky A.I., Russman I.B., Umyvakin V.M. Modeling and algorithmization of weakly formalized problems of choosing the best choice of systems. Voronezh: Publishing house of the All-Russian State University of Civil Engineering, 1991. - 168 p.
[4] Markus Schulze, The Schulze Method of Voting. Computer Science and Game Theory. -Cornell University, 2018. URL: https://doi.org/l0.48550/arXiv.l804.02973.
[5] Ralf Pickelmann,l999. Das Skating system. URL: http://www.tbw.de/rpcs/skating.
[6] Ferran Rovira,l997. ¿Porque ganamos, porqu e perdemos? TopDance, l5, l6. URL: http://inicia.es/de/ballrun/skating.htm.
[7] Xavier Mora, The Skating System. 2nd edition. July 200l. URL: https://mat.uab.cat/~xmora /escrutini/skating2en.pdf
[8] Lisitsin D.V. Methods of Regression Models Construction. Novosibirsk: NSTU, 20ll. 77 p.
[9] Kim O. J., Mueller C.W., Klecka W.R., et al. Factor, discriminant, and cluster analysis. Ed. by I. S. Enyukov. - Moscow: Finances and Statistics, l989. - 2l5 p.
[10] Kenneth J. Arrow, l95l, 2nd ed. Social Choice and Individual Values, Yale University Press.
ISBN 0-300-01364-7
[11] Condorcet J. Esquisse d'un tableau historique des progres de l'esprit humain. - Librocom, 2011. - 280 p. - (From the Heritage of World Philosophical Thought. Social Philosophy). ISBN 978-5397-01568-4.
[12] Saaty T. Decision Making. Method of hierarchy analysis. M. Radio and Communications. 1993. - 278 p. URL: http://rosculturexpertiza. ru/files/valuation/saati. pdf.
[13] Nogin V. D. The Set and the Pareto Principle. - St. Petersburg: Publishing and Printing Association of Higher Education Institutions, 2022, 2nd ed. Revised and supplemented - 111 p.
[14] Pareto V. Textbook of Political Economy. RIOR, 2018. 592 p.
[15] Koch R. The 80/20 Principle. Eksmo, 2012. 443 p.
[16] Ross M. Miller, Charles R. Plott, and Vernon L. Smith. Intertemporal Competitive Equilibrium: An Empirical Study of Speculation. Economics. Quarterly Journal of Economics. -Volume 91. Issue 4, November 1977. pp. 599-624. URL: https://doi.org/10.2307/1885884
[17] Bochkov, A.V., Lesnykh, V.V., Zhigirev, N.N., Lavrukhin, Yu.N. Some methodical aspects of critical infrastructure protection // Safety Science, V. 79, Nov. 2015. Pp. 229-242. URL: https://doi.org/10.1016/ j.ssci.2015.06.008.
[18] Zhigirev, N.; Bochkov, A.; Kuzmina, N.; Ridley, A. Introducing a Novel Method for Smart Expansive Systems' Operation Risk Synthesis. Mathematics 2022, 10, 427. URL: https://doi. org/10.3390/math10030427.
[19] Knuth D. E. The Art of Programming. Volume 3. Sorting and Searching. The Art of -Computer Programming. Volume 3. Sorting and Searching / ed. by V. T. Tertyshny (Ch. 5) and I.V. Krasikov (Ch. 6). - Moscow: Williams, 2007. T. 3. 832 p. - ISBN 5-8459-0082-1.
[20] Levitin A. V. Chapter 3. Brute Force Method: Bubble Sorting // Algorithms. Introduction to development and analysis. Moscow: Williams, 2006. pp. 144-146. 576 p. ISBN 978-5-8459-0987-9
[21] Coffman A., Henri-Laborder A. Methods and Models of Operations Research. Integer -programming. Textbook. - Moscow: Mir, 1977. 432 p.
[22] Schraever A. The theory of linear and integer programming. Monograph in two volumes. Translated from English: World, 1991. (360 p.) 344 p.
