CC BY
13
Mathematical apparatus of quality assessment of complex systems operation: methods and algorithms
K. Z. Biliatdinov
Abstract — the article proposes new methods of assessment of quality and effectiveness of functioning of complex systems. The methods are developed for the purpose of evaluation and recording of the specificity of functioning of different systems, moreover, for conducting assessment in the conditions of increasing volume of various sources of information together with stochastic character of dynamics of structured and unstructured data about complex systems.
The article also presents a model and formulas of control algorithms in complex systems, where managerial decisions are made on the basis of quality assessment of functioning of complex systems and (or) their subsystems (elements) and taking into consideration influence of external environment.
The modified DEA method used for assessment of systems effectiveness presents a combination of a classical DEA method, calculation of correlation of dependence of indices' values and application of veto coefficient.
The article presents the directions of improvement of methods for calculating the probabilistic characteristics of complex systems of various physical nature based on the application of the methodology for assessing the probability of failure of a given number of elements of a complex system, depending on the probability of failure one element in its composition, and methods for assessing the probability of achieving the goal of functioning of a complex system, depending on time characteristics and the number of failures that occur during operation. In each technique, on the basis of a systematic approach, a sequence for assessing the corresponding probabilistic characteristics has been developed for rational implementation in computer programs.
Methods of calculation of complex quality indices include basic formulas and formulated conditions of their application.
The proposed variant of presenting methods and algorithms allows to maximum rationally use them in software for assessment of effectiveness and quality of complex systems.
Keywords: — algorithm, probability, model, complex systems, quality, method, assessment, methodology, modified DEA method, time, element.
I. Introduction and problem statement
The relevance of the development and improvement of the mathematical apparatus for assessing the quality and efficiency of the functioning of complex systems is primarily based on the need for the formation and the most rational use of internal integration reserves to increase the efficiency of the functioning of complex systems [1, 2, 3, 4, 5]. Hence,
Kamil Z. Biliatdinov - PhD, Docent ITMO UNIVERSITY (email: [email protected]).
the problem arises of developing and implementing innovative algorithms for managing complex systems (hereinafter referred to as algorithms), as well as methods for assessing the quality and efficiency of these systems (hereinafter referred to as methods).
In turn, the need to improve management and make managerial decisions in order to improve the efficiency of complex systems (hereinafter referred to as systems) predetermines the following basic general requirements for the developed algorithms and methods [2, 3, 6, 7]:
- ensuring that the effects of the external environment, conditions and specifics of the functioning of systems are taken into account;
- practical focus on reducing the time for assessing the quality and processing information in the interests of significantly reducing the time for making managerial decisions;
- a decrease in the subjectivity of the perception of information by the decision-maker (hereinafter - DM), as a weakly formalized factor in the management system;
- ensuring the possibility of rational implementation in computer programs.
In modern conditions, the effectiveness of the study of complex systems, regardless of their physical nature (hereinafter referred to as systems) largely depends on a timely and well-founded assessment of their probabilistic characteristics. Currently, the imperfection of the methodological apparatus for assessing probabilistic characteristics leads to the emergence of an urgent semi-structured problem, which consists in the need to resolve the contradiction between the requirement for a significant reduction in the time and resources spent on assessment and the need to conduct expensive experiments to collect and process large amounts of information about the state of systems [8, 9, 10, 11].
One of the rational ways to significantly weaken the negative impact of this problem in practice is to further improve the methods for calculating the probabilistic characteristics of systems by using the binomial distribution law in samples with return and the Volterra integral equation of the second kind of the second order.
Analysis of modern scientific research in this subject area [6, 7, 8] has revealed trends in the complex application of proven techniques based on the integration of a wide range of research results in various fields [3, 12, 13, 14]. In practice, this is expressed in requests for a more complete
use of expert and statistical information without spending additional resources and time, as well as in the possibility of a reasonable choice of methods depending on the conditions of functioning of the evaluated systems [2, 3, 6, 7].
Therefore, today one of the promising areas for improving management and decision-making [3, 8, 13, 15] can reasonably be considered the development and implementation of innovative algorithms and methods applicable to assess the quality and efficiency of various systems [1, 2, 6, 14, 16, 17].
II. Control algorithms in complex systems
On the basis of the systematization of the results of scientific works [1, 2, 4, 9, 16, 18] and with the aim of the most rational development and implementation of algorithms and methods a model of interaction of system elements in the management process when making management decisions based on the results of assessing the quality and efficiency of the system, was developed (hereinafter - the model) (Fig. 1) and the following restrictions and assumptions were formulated.
Restrictions:
1. In the system model (Fig. 1), the control subject (control subsystem of a complex system) cannot change the state of the external environment, but can affect control objects (subsystems (or elements) of the system) - E according to the known environmental influences (G) and object state (P).
2. The results of the impact of the external environment (G) and the state of the control object (P) are reflected in the results of assessing the quality and efficiency of the system functioning on the basis of the actual values of quality indicators.
3. The state of the object (P) affects the state of the needs of the subject of management:
A=(a¡, a2, ..., a, ..., ak),
where a¡ is the state of the i-th need of the subject of management.
Assumption: based on the results of the analysis of the subject area and to take into account the specifics of the functioning of the system, we introduce a restriction that the control subsystem builds its behavior based on minimizing the needs for resources (X) during the functioning of the evaluated system (1):
a (E,G) ^ min(i = 1,k)
rUX (1)
Let Ex* be a solution to problem (1). Then the method for solving problem (1) will be called a control algorithm -formula (2):
Ex* = ^(A,G) (2)
where 9 is a control algorithm depending on the impact of the external environment G and the needs of the subject At.
The needs of the subject of management At are a function of time, reflecting the change of priorities during the life cycle of the system and depend on the change in the state of the system and the effects of the external environment (Fig. 1).
Further, it is expedient to write the control algorithm 9 in
a recurrent form:
En+i =(En, At, G).
Then, at each step (N + 1) of this algorithm, the process of improving (increasing the efficiency) of management by making timely and justified management decisions based on the results of quality assessment can be expressed by the formula (3):
At+i(G,EN+1) < A/G,EN) (3)
Further, the synthesis of the control algorithm ф can be divided into two parts (Fig. 1):
A ^ Z ^ E
At the first stage, the management objectives Z * are formed, at the second stage, the control action E * is synthesized according to the management objectives:
Z* = ф1 (G, A) E* =
The division of control algorithms into two parts ф1 and ф2 reflects the division of control tasks into tasks solved by an automated control system (ACS) as part of a control subsystem and decisions made by DM based on the results of quality assessment [21], on the basis of which control actions are synthesized on control subjects (E*).
The algorithm for applying the proposed methods and techniques is shown in Figure 2.
In the presented algorithms, the quality assessment problem (Fig. 1 and 2), solved by the ACS of the control subsystem, can be formalized and executed by software and hardware, in particular by computer programs [21, 22, 23].
III. The essence and content of the modified DEA
METHOD FOR ASSESSING THE EFFECTIVENESS OF SYSTEMS
Unlike the classical DEA method [7, 8, 16, 19, 20], the modified DEA method [2] uses Pearson's formula to calculate the correlation in order to identify the cause-and-effect relationships of the quantitative values of the result [2, p. 615], and also systematization of the obtained values in tabular forms.
To take into account the specifics of the functioning of the system, the veto coefficient ((ф (Q,)) is used, that is, a function that, when any of the most important unit indicators (Q,) goes beyond the permissible limits, becomes equal to zero, and in all other cases remains equal to one, the formula (4):
[1, если Qi n < Qi < ф(й') [0, если Qw< Qi < Qimn
The Modified DEA Method applies:
- pairwise comparisons of quantitative values of all indicators of resource consumption (X1, X2, ... X) and quantitative values of the achieved result (Y);
- comparison of the achieved result (Y) with its established base (required) values (Yb) and (or) time periods of the system functioning.
Figure 1 - Model of interaction of elements of a complex system in the management process when making management decisions based on quality assessment and the efficiency of the system
Figure 2. Scheme of the algorithm for the application of methods for assessing the quality and efficiency of the functioning of complex systems
The efficiency coefficient j of the system (RJb) is calculated by the formula (5):
Y,.
Rjb - Y~ b
(5)
The complex indicator of the economy of resource consumption (Xj, X2, ... X)
by the ./-system to achieve the result (Cj) is calculated by the formula (6):
Cj =z
i=1
(6)
where ct is the normalized coefficient of importance of the consumed resource i;
Sxi is the coefficient of efficiency of resource consumption Xi in the process of system operation to achieve the result Yj.
The Sxi value is calculated using the formula:
S = X-
Criteria for the effectiveness and efficiency of systems have been developed.
The calculation of the coefficient of efficiency of the systems is provided with the compilation of the rating of the efficiency of the evaluated systems in tabular form.
The rating is determined by the highest value of the system efficiency coefficient (Ej) - formula (7):
Cj
Ej = eRRlb + ec(l -j)
Z q
n-1
quantitative indicators of which, ideally, should be the maximum possible (for example, the operating time in unfavorable conditions), 2.l is the number of selected quality indicators of the systems of the second group.
2. If it is decided not to divide the quality indicators into the above groups, then the third and fourth methods of quality assessment are used based on places in the rating of systems, where MPiz is a place in the rating of system z according to the i-indicator of quality assessment (Pi).
The following designations are used in the calculation formulas:
- Pib is the basic i-indicator for assessing the quality of systems, respectively, PjJb - group 1 and P2.ib - i-indicator of group 2;
- gi - normalized coefficients of importance of quality indicators (can be calculated based on the opinions of experts (Fig. 2)).
The first method (main). calculation of a complex indicator for assessing the quality of the system z (Qgz) using the values of gi - the basic formula (8):
Qgz - Z^
/
i-1.1
^pP-) + Z(qpP^
P1.ib i - 2.1 P2.ib (8)
The second method. Calculation of a complex indicator for assessing the quality of the system z (Qgz) without using the values of the normalized coefficients of the importance of quality indicators (gi) - the basic formula (9):
1.i
Qz - Z(1
i-1.1
p 2.l p -) + Z(-Tr^)
1.ib
i - 2.1
P-
2.ib
(7)
where eR is the normalized weighting factor of the importance of the performance coefficient, Rjb, and eC is the normalized weighting factor of the importance of the complex indicator of efficiency, Cj [2, p. 614].
IV. Methods for calculating complex indicators for
ASSESSING THE QUALITY OF SYSTEMS
For the most rational application of the five developed methods, the following rules are provided, taking into account the conditions for the functioning of the systems being evaluated [1].
1. The first two methods are applied when the quality indicators used in the calculations are divided into two groups.
First group.
Indicators where the smallest value of the quality assessment indicator (Pj.i) is considered the best, that is, the indicators of their quantitative values should ideally be as low as possible (for example, resource consumption), j.l is the number of selected quality indicators of the systems of the first group.
Second group.
Indicators where the highest value of the quality assessment indicator (P2.i) is considered the best, that is, the
(9)
The third method. Calculation of a complex indicator for assessing the quality of the system z (QgMz) based on the rating and gi values - the basic formula (10):
Q = Zg (1 + (n - MPiz )
i=1 Z Mpz
z=l (10)
In the third method, the rating of systems is carried out according to the highest value of QgMz:
QgMz_ > QgMz-y > ••• > QgMzm D 1,2,•••, *
accordingly, this rule also applies in the fourth method for the values of QMz [1, p. 22].
The fourth method. Calculation of a complex indicator for assessing the quality of the system z (QgMz) based on the rating and without using the values of gi - the basic formula (11):
Q = Z (1 + (n - MPZ )
^Mz / , V n
=1
)
Z Mpz
z=l (11) The fifth method. Comprehensive assessment of the quality of systems through the combined application of the four abovementioned methods or their combinations - the basic formula (12):
Qk = (Qgz, Qz, QgMz, Qmz) (12)
Расчет вероятности выхода из строя элементов системы (неисправности изделий)
Сохранение Справка
п - общее кол-во элементов (изделий)
произвести расчет
]
Ввод данных
Р - вероятность выхода из строя х - количество выходящих из одного элемента (изделия) строя (неисправных) изделий
0.04 41
очистить все расчеты
очистить кроме последнего
Вероятность Р(х) выхода из строя элементов
Кумулятивная вероятность F(x) выхода из строя элементов
0.75 S 0.5 0.25
Зависимость Р(х) от F(X)
Работа с таблицей Изменение точности (кол-во цифр в дробной части) ;
выбор таблицы соответствующего графика:
13 is го 2л ¿а зз за лл
Таблица 1 -
X Р(*> FM
0 1 0.0764 0.0764
0.2005 0L2769
2 0.2590 0.5560
3 0.2195 0.7555
4 0.1372 Ol-.":1«;
5 0.0674 0.960 t
6 00272 0.9872
7 0.0092 0.9964
8 0.0027 09991
9 0.0007 0.9993
10 0.0002 I.OOM
n 0.0000 1.0000
12 0.0000 1.0000
13 0.0000 l.OOOO
14 0.0000 1.0000
15 0.0000 1.0000
IG о.оооо 1.0CC0
17 0.0000 1.0000
13 0.0000 1.0000
19 0.0000 1.0000
20 II 0.0000 ..Ü.6Ü&Ö- 1.0000
Figure 3 - Calculation results and graphs for n = 63, x = 41, Pn (1) = 0,04
This method has found its application in a comprehensive quality assessment methodology [1] and can be used for a comparative analysis of the assessment results obtained by different methods [21, 23].
Table 4
Results of assessment of the probability of achieving the objective of a system's functioning P(tH, t)
VI. Methodology for assessing the probability of
FAILURE OF A GIVEN NUMBER OF ELEMENTS OF A COMPLEX
SYSTEM, DEPENDING ON THE PROBABILITY OF FAILURE OF ONE ELEMENT IN ITS COMPOSITION
Purpose (possibilities) of the methodology and computer program:
1. To calculate the actual values of the probability (Pn(x)) of the simultaneous failure (malfunction) of the number of elements (x) in the system, with a simultaneous failure of which the system is guaranteed not to perform its functions, and the cumulative probability (Fn(x)). Calculations are carried out depending on the given probability of failure of
one element (Pn(1)) and the total number of estimated elements (n) in the system, on which the performance of one or several functions of the system and (or) the functioning of the system as a whole for a given period of time depends.
2. To analyze the dependences of the values (Pn(x) and Fn(x)) on each other, as well as on the probability of failure of one element (Pn(1)) and the number of estimated elements
(n).
3. To determine the number of elements (x and n) and the requirements for the values of the basic probabilistic indicators (Pn(1), Pn(x) and Fn(x)) in the sphere of performing individual functions (functions) of the system or in the sphere of the effective functioning of the entire system
4. To calculate and compare the values of x, n, Pn(1), Pn(x) and Fn(x) with their basic indicators (requirements) or with values for different periods of system operation or for comparison with the values of these indicators of other similar systems.
5. To determine the «weak link» in the functioning of the system: when performing which function of the system, the probability of failure of the function will be the greatest, that is, the case will come when the value of Pn (x) will be the greatest.
6. To save time and resources for testing the number of evaluated elements (x and n) in real conditions with a known value of Pn(1), by predicting the probability in the sphere of stable functioning of systems (by constructing tables of values of Pn(x) and Fn(x)) [11, 12, 18, 24].
A short sequence of actions when performing the technique:
1. Determination of the initial data for assessing the quality of systems.
1.1. Formulation of the purpose (purpose) of the application of the methodology and computer program, by choosing and combining the above-described points 1 - 6 of the methodology purpose.
Number of failures, S Variants of values of time characteristics
(measurement units)
1 variants j variant
tH1 toi P(t„i, toi) tHj t0j P(tj, toj)
1
2
i
1.2. Determination of the function (functions), the performance of which is influenced by the evaluated elements, or the establishment of the minimum required number of systems (elements of one system), in which the system will be in the required state.
1.3. Depending on the purpose of the application of the technique (p. 1 - 6 assignment), set the initial values x, n, or Pn (j).
2. performing calculations and plotting graphs - examples in the figure 3.
2.1.
Pn ( x) = CXPx (1 - P)n-x =
n!
x! (n - x)!
Px (1 - P)n
(1)
where Cnx is the number of combinations of n elements in
x.
Fn (x) = Z Pn (k )
2.2. k=0 (2) The cumulative probability depends on x, n, P. 2.3. If in practice the volume n is small, then to calculate Pn (x) it is rational to use the formula (3):
Pn(x) =
Pn( 1 + x)2(1-P) P(n-x)
(3)
3. Registration of the results of the assessment. The main disadvantage of the method: for calculations, it is necessary to know the actual value of the probability of failure of one element of the system (P(J)).
dF (t) is the probability of the first system failure at the moment t <tn.
Restriction: the initial number of partitions of numerical intervals for numerical simulation is N1 = 100.
Assumption: the accuracy of calculations of numerical calculations is e.
The sequence of actions when performing the Methodology (algorithm of the computer program [2] - the second stage of the Methodology):
Stage I. Definition and input of initial data.
1. tn - the minimum required time to achieve the goal of the system functioning in favorable conditions (there are no malfunctions, accidents and other destructive influences). That is, this is the operating time of the system when there are no failures (malfunctions of elements) that affect the achievement of the goal.
2. t is the required (set) time to achieve the goal of the system functioning, while the condition must be met: t > tn, the value of t is not entered into the program for calculations (the second stage of the Methodology).
3. t0 - standby time, calculated as the difference between the values of the required time (t) and the minimum required time (tn): to = t - tn.
4. S is the limiting number of failures, in the form of a set of damages, malfunctions, lack of resources and the consequences of untimely and (or) incorrect and (or) untimely management decisions that negatively affect the achievement of the goal of the system's functioning.
5. Parameters of the distribution functions F and Fv.
VII. A TECHNIQUE FOR ASSESSING THE PROBABILITY OF ACHIEVING THE GOAL OF FUNCTIONING OF A COMPLEX
SYSTEM, DEPENDING ON THE TIME CHARACTERISTICS AND THE NUMBER OF FAILURES
The technique allows one to take into account random influences on the parameters of the systems functioning processes associated with the influence of the external environment and many other poorly predictable factors. The methodology and the computer program are based on calculating the probability of timely achievement of the goal of the system functioning using the Volterra integral equation of the second kind of the second order with respect to the specified probability [5, 25]: tH trj
P(tH,t) = 1 - F(tH) + {{P(tH — t ,to - &)dFv(©)dF(t)
00 (4)
In the basic formula (4), P (t„ t) is the probability of timely achievement of the goal of the system functioning under the condition of the required (normal) functioning of the system in the initial period of time. In this case, 1 - F (tn, t) is the probability of failure-free functioning of the system during time tn.
P (tn - t, t0 - &) is the probability that the system, being in the initial period of time in failure, will restore operability and complete tasks before the rest of the reserve time t0 - & is consumed.
dFv (&) is the probability of system recovery in time
& <to.
Stage II. Performing calculations using a computer program (algorithm):
1. Input of tn, t0, S, F and Fv values.
2. Dividing the ranges [0, tn] and [0, t0] evenly into 100 segments, since it is assumed that Nj = 100.
3. At the nodes tn, the calculation of the values of the parameters of the distribution functions F and Fv, for
n = 0, N F (tn ) = F (n) = 1
on the segment [0, tn]
Fv (tn ) = Fv (n) = 1 - ¿
-At„
e n
on the segment [0, t0].
Similarly, the values of the derivatives of these functions are calculated at the nodes tn at the corresponding time intervals.
4. Calculation of the values of functions at the nodes tn:
Po(tH) = 1 - F(tH) h Pvo(to) = Fv(to)
M .
5. At the nodes tn on the segment [0, tn], the calculation of the value of Pt (n),
i=1S n=On
P (n) = £ (P-M - k)F'(k))aht,
k=0
Conditions: if k=0, then m=0,5 and if k=n, then m=0,5.
- weights of numerical integration by the trapezoid method, h-step between nodes tn [25].
6. At the nodes tn on the segment [0, t0], the calculation of the value of Fvi (n),
i=1s n=On
Fvi(n) = £(Fv-1 (n - k)Fv 1 (k))a>h,
k=0
Conditions: if k=0, then m=0,5 and if k=n, then m=0,5.
- weights of numerical integration by the trapezoid method, h-step between nodes tn.
7. Upon completion of steps 2-6, the result is calculated:
P(K, to) = iPk (tH) Fvk (to)
k=0
Next, the accuracy of the calculations performed with the parameter e is checked.
If the obtained values do not satisfy the specified accuracy, then N2 = N1 + N1 and steps 2-7 are repeated with the value N2 until the accuracy e is achieved.
8. Output of the value P (tn, to).
Stage III. Formation of assessment results in tabular form (table 4).
Disadvantages of the methodology: in practice, it might be difficult to precisely predict entered values of variables (except for the required time of achievement of the objective) under destructive influences. Moreover, this methodology does not contain dependence of the timely achievement of the objective from the spent resources.
VII. Conclusion
Thus, the essence of the proposed innovations lies in the complex application of the developed methods and methods based on the use of real values of quality indicators in assessing the quality and efficiency of systems (Fig. 2).
The importance of implementing a systematic approach in the studied subject area and the structural and functional complexity of the systems being evaluated [8, 16, 20, 24, 25, 26] increase the importance of these innovations when improving systems management processes.
It is important to note that, theoretically, one of the promising areas of rational application of the developed methods can be the registration of complex events (within visibility on the event horizon), the calculation of the probabilistic characteristics of their development scenarios, interrelationships and assessment of possible consequences in order to increase the efficiency of the study of the dynamics of the states of complex systems [5, 9, 12, 24, 27].
The presented algorithms, ways and methods provide for the possibility of their rational implementation through the use of computer programs [12, 21, 23], which in practice can
significantly reduce the cost of resources and time for assessing the quality and efficiency of systems during operation.
REFERENCES
[1] Biliatdinov K.Z. Complex methodology of technical systems quality assessment in the process of maintenance // Scientific and technical Volga region bulletin. No. 11, 2020. - P. 20 - 23. (in Russian)
[2] Biliatdinov K.Z., Meniailo V.V. Modified DEA method and methodology of assessment of technical systems' effectiveness // Information technologies, 11, 2020. P. 611-617.
[3] Han P., Wang L., Song P. Doubly robust and locally efficient estimation with missing outcomes. Statistica Sinica. 2016. Vol. 26. No. 2. Pp. 691-719.
[4] Kalimoldayev M., Abdildayeva A., Mamyrbayev O. Information system based on the mathematical model of the EPS. Open engineering. 2016. Vol. 6. No. 1. Pp. 464-469.
[5] Zou G., Faber M.H., Gonzalez A. A holistic approach to risk-based decision on inspection and design of fatigue-sensitive structures // Engineering structures. 2020. Vol. 221. Article 110949.
[6] Luetje A., Wohlgemuth V. Tracking Sustainability Targets with Quantitative Indicator Systems for Performance Measurement of Industrial Symbiosis in Industrial Parks. Administrative sciences. 2020. Vol. 10. No. 1.
[7] Ratner S., Ratner P. Developing a strategy of environmental management for electric generating companies using DEA-methodology // Advances in Systems Science and Applications. 2017. Vol. 17(4). - P. 78-92.
[8] Banker R., Kotarac K., Neralic L. Sensitivity and stability in stochastic data envelopment analysis. The Journal of the Operational Research Society. 2015. Vol. 66. No. 1. Pp. 134-147.
[9] Calabrese R., Osmetti S.A. A new approach to measure systemic risk: A bivariate copula model for dependent censored data // European Journal of Operational Research. 2019. Vol. 279(3). P. 1053-1064.
[10] Calabrese R., Osmetti S.A. A new approach to measure systemic risk: A bivariate copula model for dependent censored data // European Journal of Operational Research. 2019. Vol. 279(3). P. 1053-1064.
[11] Price M., Walker S., Wiley W. The Machine Beneath: Implications of Artificial Intelligence in Strategic Decision making. PRISM. 2018. Vol. 7. No. 4. P. 92-105.
[12] Zhang Z., David J. Structural order measure of manufacturing systems based on an information-theoretic approach // Expert Systems with Applications. 2020. Vol. 158. Article 113636.
[13] Karagiannis G. On structural and average technical efficiency. Journal of Productivity Analysis. 2015. Vol. 43. No. 3. Pp. 259-267.
[14] Putz M., Wiene, T., Pierer A. A multi-sensor approach for failure identification during production enabled by parallel datamonitoring. CIRP annals-manufacturing technology. 2018. Vol. 67. No. 1. Pp. 491-494.
[15] Yang R., Zheng W. Output-Based Event-Triggered Predictive Control for Networked Control Systems // IEEE transactions on industrial electronics. 2020. Vol. 67. № 12. - P.10631-10640.
[16] Dula J.H. Computations in DEA // Pesquisa Operacional. 2002. Vol. 22, № 2. - P. 165-182.
[17] Trevino M. Cyber Physical Systems: The Coming Singularity. PRISM. 2019. Vol. 8. No. 3. Pp. 2-13.
[18] Gerami J. An interactive procedure to improve estimate of value efficiency in DEA // Expert Systems with Applications. 2019. № 137. P. 29-45.
[19] Banker R.D., Charnes A., Cooper W.W. Some models for estimating technical and scale efficiency in data envelopment analysis // Management Science. 1984. Vol. 30, № 9. - P. 1078 - 1092.
[20] Chen J.-X. Overall performance evaluation: new bounded DEA models against unreachability of efficiency. The Journal of the Operational Research Society. 2014. Vol. 65. No. 7. Pp. 1120-1132.
[21] Biliatdinov K.Z. Assessment of systems stability (Certificate of the state registration of software № 2020615328, 21.05.2020).
[22] Biliatdinov K.Z. Realization of the method of rational processing of information resources and formation of information reserves of a system (Certificate of the state registration of software № 2020610335, 13.01.2020).
[23] Biliatdinov K.Z. Analysis and assessment of systems effectiveness (Certificate of the state registration of software № 2020610389, 14.01.2020).
[24] Duer S. Assessment of the operation process of wind power plant's
equipment with the use of an artificial neural network // Energies. 2020. № 13. Art. 2437.
[25] Yizhen P., Yu W., Jingsong X., Yanyang Z. Adaptive stochastic-filter-based failure prediction model for complex repairable systems under uncertainty conditions // Reliability Engineering and System Safety. 2020. Vol. 204. Article 107190.
[26] Shafik M.B., Chen H., Rashed G. Planning and reliability assessment to integrate distributed automation system into distribution networks utilizing binary hybrid PSO and GSA algorithms considering uncertainties // International Transactions on Electrical Energy Systems. 2020. Article e12594.
[27] Baker J., Henderson S. The Cyber Data Science Process. The Cyber Defense Review. 2017. Vol. 2. No. 2. P. 47-68.
