CC BY
12
-□ □-
За результатом теоретичного анапъзу вгдомих та широко застосованих в практичнт дiяльностi стандартiв з управлт-ня ятстю та проектного менеджменту, моделей оцтки оргат-зацтно'г зрiлостi процепв виокремлено 13 ключових ризик-доми нуючих факторiв, як потенцтно впливають на зрглкть процепв управлтня ятстю проектъв. Розроблено методику впорядкування ризик-домтуючих факторiв, як в найбшьшому ступеню впливають на оргатзацтну зрЫкть процеЫв управлтня ятстю в проектах. В основу розробленог методики покладено експертний метод Дельфi. На вгдмту вiд традицтного методу, впорядкування фак-торiв запропоновано здтснювати в один тур завдяки залученню до експертного оцтювання двох щльових груп: «Виконавщв процесу» та «Замовнитв результатов процесу та представнитв зацжавле-них сторт». Вгдмтною особливктю методики е отримання впо-рядкованог номенклатури ключових ризик-домтуючих факторiв з урахуванням компетентность експертъв у певнт галузi. Системне науково-обгрунтоване впорядкування ризик-домтуючих факторiв у вгдповгдностг iз запропонованою методикою сприятиме об'ектив-ному оцтюванню потенциалу процепв управлтня якхстю в проектах, тдвищенню ймовiрностi отримання очжуваних результа-тхв процепв. Впровадження запропонованог методики дозволить визначати прюритети серед напрямiв росту й оргатзацтних змт процепв для досягнення щльових рiвнiв зрiлостi. Запропонована методика може бути реалiзована в процеа сертифжацп, самоощ-нювання та аудитах СУЯ.
Проведено практичну апробацю запропонованог методики на прикладi процесу «Запуск проекту». За результатами експертного оцтювання «Виконавщв процесу» встановлено наступне. В бшь-шому ступеню на зрШсть дослгджуваного процесу впливае фактор «Ступть документованостъ процесу», а в меншому ступет - фактор «Ступть застосовностъ результатъв оцтки дiевостi процепв для гх вдосконалення». За результатами експертного оцтювання «Замовнитв результату процесу та представнитв зацжавлених сторт» встановлено наступне. В бшьшому ступеню на зрШсть дослгджуваного процесу впливае фактор «Ступть можливостъ ттеграцп процесу з тшими внутрШтми та зовшштми процеса-ми», а в меншому ступеш - фактор «Поведтка виконавщв процесу». Встановлено високий ступть погодженостъ ощнок експертъв всередин кожног групи, та мiж оцтками двох груп експертъв
Ключовi слова: ризик-домтуючий фактор, управлтня ятстю
проектъв, оргатзацтна зрЫкть процеав, експертне оцтювання -□ □-
UDC 65.015. 6
|DOI: 10.15587/1729-4061.2018.14079б|
DEVELOPMENT OF A METHOD FOR RANKING FACTORS THAT INFLUENCE THE MATURITY OF PROJECT QUALITY MANAGEMENT PROCESSES
I. La z ko
PhD, Professor Department of standardization and management by quality LTD «Khimtechnologya» Severodonetsk Institute of PJSC «HEI«IAPM» Smetanina str., 26а, Severodonetsk, Ukraine, 93400 E-mail: irina.lazko@khimtekh.lg.ua
1. Introduction
Quality management principles of the national standard DSTU ISO 9001 [1] (harmonized with [2]) should be implemented with systematic evaluation and improvement of the quality management system (QMS) processes. Herewith, the strategic goal of improving the processes is to achieve their desired level of organizational maturity [3-6]. In accordance with the national standard DSTU ISO/IEC 33001 [7] (in harmony with [8]), organizational process maturity is «the degree to which an organizational unit consistently performs processes in a specific area that promotes business needs». While developing this concept, the term «organizational maturity of the project quality management process» is understood by the author as the characteristics of the quality management process reflecting the degree of its suitability, adequacy, and effectiveness. Such an approach to the defini-
tion of this concept is due to the appropriateness of harmonization with the requirements of the standard [1].
It is known that the Process Maturity Levels concept was formed at the Software Engineering Institute (SEI) at the Carnegie Mellon University in the 1990s. This concept was first developed to support analysis of the maturity of the programming process capabilities (Capability Maturity Model, CMM). Its final version is the Capability Maturity Model Integrated, CMMI [5, 6]. This model has been generalized for a wide range of processes of organizations of various industries; therefore, it can be used to analyse the organizational maturity of QMS projects.
It is obvious that for proper consideration of the organizational maturity of QMS projects, it is necessary to have appropriate objective information. It concerns, among other things, information on the priority of the influence of risk-dominant factors on the organizational maturity of the
©
system processes. The availability of such information will help make reasonable management decisions on the form, means and methods of ensuring the organizational maturity of QMS processes. Eliminating less influential risk-dominant factors and allocating resources to enhance the impact of prioritization factors will potentially contribute to increasing the organizational maturity of QMS projects' processes. Thus, the issue of ranking risk-dominant factors on the priority of their impact on the organizational maturity of QMS projects is quite important. The solution of this issue will be facilitated by the development of a formalized methodology for ranking risk-dominant factors.
Among the traditional approaches to organizing the investigated alternatives in terms of the degree of manifestation of quantitative or qualitative features, a well-known and widely implemented approach is ranking [9-11]. The ranking of alternatives of a quantitative feature is based on comparing numbers among themselves and does not cause significant difficulties in practical implementation. In contrast, the ranking of alternatives, in particular, risk-dominant factors, which are mainly compared on a qualitative basis, is impossible without the participation of highly skilled professionals. In this case, an increase in the objectivity of the result of a ranking can be achieved by involving, in the evaluation of alternatives, two target groups of specialists who have different interests regarding the results of the process performance. That is, it requires observing the principle of taking into account the interests of the implementers of a QMS project process and the parties interested in the results of the process [12].
It should be noted that the issue of ranking alternatives within the framework of solving various managerial tasks has been considered in a significant number of studies [13-15]. However, at present, there is a lack of a formalized, quantitative methodology for assessing the priority of the influence of risk-dominant factors on the organizational maturity of QMS projects' processes. This situation is explained by the need for evaluation, as a rule, under poorly structured inputs and information constraints, often in the absence of sufficient experience and competence. Under these conditions, the practical value of collective expert knowledge [16] and the effectiveness of applying the method of collective expert evaluation [9-11] become more essential for solving the problem of ranking risk-dominant factors.
It is known that a significant advantage of the method of collective expert evaluation in comparison with the individual assessment consists in the greater objectivity of the result [9, 17, 18]. At the same time, the process of its implementation is connected with the need to solve a number of problems. These problems are conventionally divided into two groups [19]. The first group should include problems associated with the structure of the process of collecting expert data. Traditionally, it includes the choice of an approach to organizing and evaluating alternatives, defining the rules for the coordination of expert assessments, and selecting the resulting ranking. The second group should include problems associated with the resources of an expert system. The traditional problems of this group are the justification of the array of alternatives that need to be ranked, the level of expertise of experts, etc. Taking into account the above, the task of developing a methodology for ranking risk-dominant factors based on an improved method of collective expert assessment with regard to the competence of experts is important and needs to be addressed.
2. Literature review and problem statement
The practical experience of ranking the alternatives, which is reflected in published papers [13-15], convincingly demonstrates that this process is multistage. The sequence and contents of the steps are related, preferably, to the ranking and properties of the object under study. As a result of analysing previous research findings [9-11] and the practical experience of expert evaluation in the field of QMS projects, the following six typical stages of ranking alternatives are singled out:
1. Identify the problem area of the research object and ascertain the key alternatives that need to be arranged.
2. Choose the approach to the organization of expert evaluation and the means of obtaining expert assessments.
3. Choose the optimal quantitative and personal composition of an expert group.
4. Receive and process expert opinions.
5. Determine the degree of agreement among expert opinions.
6. Determine generalizing ranking of alternatives and analyse the obtained results.
Each of the isolated stages has its own peculiarities, related to solved tasks, applied instruments, etc.
Thus, at the first stage of ranking, the choice of the range of risk-dominant factors can be made, for example, by the method of «brainstorming» [9-11]. In this case, the source of the choice argumentation can be the practical experience of the experts, the results of the theoretical analysis of standardized requirements for the QMS [1, 20, 21] and project management [22], and approaches to assessing the maturity of processes. One of the approaches to assessing the maturity of processes has been standardized to some extent and given in [23, 24]. Another modern approach is provided in [25].
After identifying the overall risk-dominant factors, the question arises as to their optimal number, since an excessive number of ranking alternatives tends to affect the objectivity of ranking [9-11]. As a result of analysing published scientific findings, it has been established that for obtaining an objective resultant ranking it is necessary to select, as a rule, no more than 20 alternatives [18, 26].
At the second stage of ranking alternatives, it is necessary to choose the approach to the organization of expert assessment and the means of obtaining expert assessments. According to many scholars, among the methods of collective expert evaluation, the Delphi method is one of the most widespread and promising [27-30]. The usefulness of the method is due to its properties, among which the essential features are the possibility of implementing the method in any field, the anonymity of experts, a regulated feedback during the interview, and obtaining a group assessment [10, 11]. The algorithm for implementing the Delphi method is well-known and has been studied in detail [30]. The experience of successful practical implementation of this method for solving the problem of ranking alternatives in various fields of research is reflected in many scientific papers [9-11]. However, despite the significant advantages of the Delphi method, the analysis of research findings has shown that the implementation of the method does not always take into account the competence of experts, inorganic and technical. Thus, some studies [9, 17] emphasize the need for conducting at least five rounds of examination. Thus, optimization of the number of expert assessment tours can be justified as one of the ways of improving the Delphi method. Such an optimization can be
accomplished, for example, by conducting an examination on the principles of taking into account the interests of the im-plementers of a project's QMS process and those interested in the results thereof [12]. That implies conducting an expert examination in one round by two expert groups that have different interests regarding the results of ranking alternatives. This approach helps avoid the multistage procedure.
At the third stage of ranking alternatives, it is necessary to solve one of the important problems of the theory and practice of expert research. This is a problem of making a reasonable choice of the optimal quantitative and personal composition of the expert group. The analysis of the scientific findings on this issue has shown that the optimal number of experts involved in the expert group depends on many factors. Among such factors are the purpose and the availability of evaluation possibilities, the importance of the problem under investigation, the competence of the experts, the predicted level of the error of expertise, etc. [27]. The analysis of scientific publications has shown that different approaches to determining the number of experts are based on the consideration of some of these factors.
Thus, in [31], the proposed methodology for determining the number of experts is based on the Kendall's coefficient of concordance as well as on the normal and gamma-ratio random variables. The analysis of the proposed approaches has shown that the use of the coefficient of concordance to determine the minimum number of experts for a reliable examination is limited. This is due to the fact that the number of experts needed to bring about a unanimity of opinion is less necessary than to bring about divergences. In accordance with another approach [31], the minimum number of experts is proposed to be calculated as the root of the number of objects under study. The advantage of this approach is the ease of calculating the minimum number of experts. However, at the same time, the calculated minimum number of experts may not be objective because it does not take into account their competence.
Paper [32] proposes an approach to determining the number of experts based on calculating the ratio 2 / o2. The study concludes that an acceptable number of experts in an expert group is 11 specialists provided that |i2/ o2 = 0.1. This approach to determining the number of experts is justified, since it is based on mathematical statistics methods. However, it would be advisable to determine not the total acceptable number of experts in an expert group but the minimum number of experts with which the results of the examination can be objective [31].
According to another approach [33], the minimum number of experts is calculated as the root of the ratio of the maximum acceptable standard error of the predicted expert estimation to the standard deviation of expert estimates. It should be noted that in order to calculate the minimum number of experts under this approach, it is necessary to have information on the law on the distribution of expert assessments or to accept the hypothesis regarding the nature of the distribution. This may cause some difficulties in implementing the proposed approach.
The most appropriate approach is to determine the minimum number of experts, depending on the level of error of expertise, which is given in paper [34]. In accordance with this approach, a group expertise should involve at least 4 to 7 specialists, provided that the level of error of expertise is 10 %. This approach can be used to determine the optimal number of experts to rank risk-dominant factors.
Thus, the results of analysing the above-mentioned approaches and the experience of carrying out an examination in the field of QMS projects have shown that the minimum acceptable number of experts in an expert group is from 7 to 15 specialists. Involving an excessively small number of experts (fewer than 7 specialists) in an examination may result in a non-objective assessment. However, the involvement of an overly large number of experts (more than 15 specialists) in the examination may cause difficulties of organizing an expert survey.
Another problem of the practical implementation of this phase is the assessment of the competence of experts involved in the regulation of risk-dominant factors. The analysis of published scientific findings on this issue has shown that in spite of the uniqueness of each expert examination, the normative basis of universal approaches and criteria for evaluating the competence of experts have been formed in the theory of expert evaluation.
Thus, among the widespread approaches to assessing the competence of experts, self-esteem and mutual evaluation should be noted [9-11, 35]. Despite certain advantages, the assessment of competence in these approaches is predominantly subjective and depends on the influence of psychological factors on an expert, the personal qualities of the subject of evaluation, etc. The analysis of the scientific studies has shown that the element of subjectivity in the process of competence assessment can be minimized by taking into account the personal qualities of an expert [36]. It should be noted that according to some experts, self-assessment of their own capabilities is often misleading [37]. The limitation of the approach based on the mutual assessment of the expertise of experts by members of an expert group or a separate working group is mainly due to the lack of knowledge of experts about one another's professional capabilities. The analysis of the scientific findings has shown that it is possible to partially eliminate the disadvantages of the considered approaches and obtain the most objective results by a generalized assessment of competence, which includes self-esteem and mutual evaluation [38].
It is known that the problem of evaluating the competence of experts is primarily related to the problem of measuring knowledge, which is directly connected with the object of expertise [35]. At present, the most common approaches that allow such measurements are attestation and testing [9]. Practical experience convincingly demonstrates that it is inappropriate to consider attestation as an effective approach to competence assessment. Its results are usually formal and related to the general scientific level of an organization. On the other hand, a test approach aimed at solving test questions by experts, reflecting the specifics of the subject matter of expertise, allows solving the problem of competence assessment more objectively [11, 39]. However, the application of this approach may be limited by its labour intensity. Testing requires a considerable amount of time and reasonable similarity of test tasks with those that will have to be addressed in practice [35].
Another approach to competence assessment is based on objective documented expert characteristics [10, 11]. Despite the sufficient objectivity of the approach, the process of «curtailing» the documentary characteristics of an expert as the indicator of competence can be implemented very subjectively [33, 40]. The analysis of the scientific findings has shown that documentary characteristics of an expert often have signs of his or her formal professional status: position, degree, overall work experience, etc. [41-44]. Obviously,
such an approach will contribute to the formation of a professional expert group with extensive practical experience in a certain field [45]. However, the application of this approach alone does not always fully reflect the degree of competence of an expert in a particular area. Taking into account the above, it is expedient to assess the competence of an expert by an integrated indicator of competence.
Thus, the analysis of previous studies has shown that the simplest approaches involve finding an integrated competence based on two components. Such components can be the degree of argumentation and expert acquaintance with the problem [44, 46]; the degree of argumentation and awareness of the problem [47]; the degrees of qualification and argumentation [48]. When implementing such an approach, the coefficient of argumentation is determined by self-assessment of the degree of using such arguments as theoretical and practical experience in the field of expert assessment, intuition, etc. The application of such an approach, based on the subjective assessment of the expert, cannot fully guarantee the objectivity of the result. Therefore, it seems appropriate to combine this approach with approaches based on the documentary statistical characteristics of an expert. An example of such a combination is given in [49]. In this work, when determining the characteristics of the quality of an expert, five estimates with different weight coefficients are taken into account. Among these assessments are three heuristic (self-esteem, mutual evaluation, evaluation by the working group) and two statistical (deviations from the average and reproducibility of the result). The problem in this direction is the definition of the nomenclature of essential criteria of experts' competence in assessing the priority of the influence of risk-dominant factors. Let us consider the generalized approaches to the definition of such criteria.
Thus, in [32] it is proposed to assess the competence of an expert by taking into account equally five aggregate factors. These factors include: the coefficient that reflects the level of professional training and awareness of the i-th expert (Ki1); the coefficient that reflects the basic level of argumentation of the ¿-th expert when making a decision (Ki2); the coefficient that reflects the personal qualities of the ¿-th expert and is calculated by self-evaluation (Ki3); the coefficient reflecting the personal qualities of the ¿-th expert and calculated by expert colleagues (Ki4); the coefficient that reflects the level of coordination of the actions of the ¿-th expert with the members of the working group (K^).
According to another approach [50], the definition of the competence coefficient of the ¿-th expert is carried out by taking into account equally four aggregates. These factors reflect the level of professional training and awareness (Ki1); the level of basic argumentation when making a decision (Ki2); personal qualities (Ki3); the level of coordination with the members of the working group while doing the test (Ki4).
An example of a successful combination of different approaches to assessing the competence of an expert is also presented in [35]. The essential components of an expert competence in accordance with this approach are the expert's knowledge, reasoning, qualification, assessment by the working group, assessment of qualitative competence, assessment of professional competence, and lack of conformism.
It should be noted that, in accordance with the considered approaches, the competence of experts can be assessed on the basis of rating scales, questionnaires [9-11], and the like. However, at present, the method of pair comparison (matching) is most used [51-55].
Thus, a certain set of criteria is used to assess the competence of an expert, so there is a need for some integral estimation. At the same time, it should be taken into account that the choice of the system of criteria for the competence of an expert is complicated by the need to meet partially contradictory requirements: completeness, minimality, absence of loss-making, operational, and measurability [11].
In [56], it is demonstrated that the generalizing expert judgment depends, in particular, on the method of processing individual expert assessments. Therefore, to implement the fourth stage of the ranking of alternatives, let us consider the advantages and disadvantages of existing methods.
Thus, the method of d¿rect evaluat¿on [10, 11, 57] is attractive, since it helps obtain a numerical evaluation of an alternative. However, the probability and accuracy of this assessment is often low due to the weakness of the structured alternative that is being evaluated. In addition, this method cannot be used in case of incomplete knowledge of the expert about the properties of the alternative.
Methods that use fuzzy, linguistic and ¿nterval expert eva-luat¿ons [58, 59] can be applied in conditions of fuzzy input data to evaluate qualitative alternatives. However, this method is limited because of the considerable subjectivity in the process of choosing membership functions and the formation of fuzzy rules; moreover, it requires to use special software.
The disadvantage of the method of pa¿r comparison [60, 61] is its bulkiness. It is known that increasing the number of alternatives increases the number of comparison pairs. The procedure of pairwise comparison of alternatives, which is the basis of the hierarchy analysis method (HAM), generates a number of shortcomings that directly affect the effectiveness of the method [62-64]. It should be noted that the method of pairwise comparisons is used in cases where the differences between alternatives are so small that direct estimation or ranking does not ensure their reasonable arrangement. A distinctive feature of the method of pair comparison is a considerable labour intensity, which is offset by a greater objectivity of the result obtained by comparison with the ranking method.
In the process of implementing the ranking method [10, 57], each expert is invited to place alternatives in accordance with their influence on the object under study. In this case, the scale is formed automatically: from 1 to N, where N is the number of identified alternatives that need to be ranked. After the ranking procedure, the rank number of an alternative in the selection is called a rank. Averaging for all experts, we get an average rank. Averaging by the number of ranks, we receive an assessment of the priority of the impact of a particular alternative on the investigated process. The method is easy to understand and does not cause difficulties in the implementation process. At the same time, the accuracy of the ranking depends to a large extent on the number of alternatives to rank. The main disadvantages of the ranking method should also include the loss of information about the alternatives that are evaluated as a result of ranking them only on the basis of their relative location without taking into account their essential features.
It should be noted that the fundamental difference between the first two methods is that they are based on an independent evaluation of each alternative outside of its connection with other alternatives. The other two methods, on the contrary, are aimed at identifying the place of a particular alternative among other alternatives, regardless of the absolute values of the characteristics of the alternative itself.
Thus, it seems expedient to use the ranking method to rank risk-dominant factors, provided that the optimal number of evaluated alternatives is optimized.
The lack of consensus among experts within the group may be a serious aspect that limits the study of any problem. Therefore, at the fifth stage of the ranking of alternatives, it is necessary to assess the degree of coherence of expert assessments within the group. For the quantitative assessment of the degree of consistency, the Kendall and Babington Smith coefficient W [65-67] is traditionally used. The value of the coefficient of concordance is in the range from 0 to 1 (0 < W< 1), where W= 0 denotes the complete opposite of the rankings, and W= 1 means the complete match of the rankings. In practice, coincidence is considered acceptable if the fulfilled condition is W > 0.7. It should be noted that Kendall's concordance factor does not indicate the degree of agreement of expert opinions for each alternative. It only shows the correlation of expert conclusions.
Obviously, in the case of involving different experts in the evaluation of the same alternatives, as a rule, a set of alternative ranking is obtained. Therefore, at the sixth stage of ranking, it is necessary to obtain a generalizing ranking of alternatives and to carry out an analysis of the results obtained. As a result of the analysis of scientific sources, it becomes clear that the method of arithmetic mean, median method, and the Kemeny's median method are widely used for solving the problem of obtaining general ranking. The most common are the methods of arithmetic mean points and the median method [9, 68]. The first method is not sufficiently correct, since the scores are measured in the order of magnitude. It is more reasonable to use medians as average points. However, ignoring the average arithmetic points is inappropriate because of their habit and prevalence. Therefore, in accordance with [68], it is expedient to use both methods simultaneously. Such an approach is in harmony with the general scientific concept of sustainability, which recommends applying different methods for the processing of identical data in order to distinguish common conclusions [69].
In addition to the above, a final ranking is the one the amount of distances from which to each individual ranking is minimal [70]. This approach is based on the Kemeny's median method [70-74]. It should be noted that this method satisfies four of Arrow's five conditions. At the same time, among the shortcomings of the method restricting its practical application are the complexity and labour intensity of calculating Kemeny's median.
3. The aim and objectives of the study
The study is aimed at proposing and implementing a practical testing of the methodology of scientifically substantiated ranking of risk-dominant factors that to a large extent influence the organizational maturity of project quality management processes.
To achieve this aim, the following objectives are solved:
- to identify and form an incoming set of key risk-dominant factors that potentially affect the organizational maturity of project quality management processes;
- to distinguish the typical stages of ranking risk-dominant factors and to analyse the peculiarities of each stage;
- to conduct the practical testing of the proposed methodology using the example of a specific quality management project.
4. Development and practical testing of risk-dominant factors' ranking techniques
4. 1. Development of a method of ranking risk-dominant factors
In the course of the study, a methodology has been developed for ranking risk-dominant factors that influence the organizational maturity of project quality management processes. In accordance with the developed methodology, the organization of risk-dominant factors is proposed in six stages.
The first stage. Highlighting the key risk-dominant factors that have the greatest impact on the maturity of a particular project quality management process. To distinguish the range of risk-dominant factors of influence on the organizational maturity of QMS processes, it seems appropriate to define the «space» of the functioning of these processes. As a result of analysing the standards [20-22], it is concluded that the «space» of project management processes encompasses three areas. Thus, the first direction is the management group of processes (Initiating, Planning, Implementing, Controlling, and Closing). The second direction is the subject group of processes (Integration, Stakeholders, Scope, Schedule, Cost, Quality, Risk, Resource, Procurement, and Communications). The third direction is the stage of the life cycle of the design (for example, pre-design proposals, input data collection, the «Project» stage, the «Work Documentation» stage, author's supervision, etc.). Within this «space», the nomenclature of quality management processes is traditionally formed on the basis of a standard [1] or standards [20, 21]. The application of this or that approach depends on the strategic goal of implementing a QMS, resource constraints, level of knowledge and awareness of the personnel in terms of quality, etc. Thus, according to the approach outlined in [1], grouping of processes is carried out in such areas as leadership, planning, support and operation, performance evaluation, and improvement of the QMS. This approach is universal and widespread in world practice, but it has not been adapted to the design industry. Instead, in studies [75, 76] a more effective, from the author's point of view, approach is proposed to distinguish the range of QMS processes in projects, which is based on the integration of requirements of standards [1, 21, 22] and the key requirements of project management. Risk-dominant influence factors in accordance with this approach are located on the plane «Responsibility Management - Planning - Functioning - Maintenance - Effectiveness Evaluation - Improvement».
To further distinguish key risk-dominant factors, it seems appropriate to consider the aspects (criteria) that are used to assess the organizational maturity of processes. In particular, the study focuses on the approach based on the principles of standards [23, 24] and the process maturity model as well as the Process and Enterprise Maturity Model [25].
Thus, with the approach in accordance with standard [23], which is harmonized with [24], and the process possibilities are evaluated on a six-digit scale. The scale of the properties of the processes has a gradation N, P+, P-, L+, L-, F from the lower zero level «Unfinished Process» to the upper fifth level «Innovative Process». At the same time, nine process properties (PA - process attribute) demonstrate its affiliation to the appropriate level of capabilities, namely:
- PA1.1 »Process Effectiveness» - characteristic for Level 1, «Process Implementation»;
- PA2.1 »Performance Management» and PA2.2 «Pro-cess Result Management» - characteristic for Level 2, «Ma-naged Process»;
- PA3.1 »Process Definition» and PA3.2 «Process Deployment» - characteristic for Level 3, «Established Process»;
- PA4.1 »Quantitative Analysis» and PA4.2 «Quantitative Control» - characteristic for Level 4, «Probable Process»;
- PA5.1 »Process Innovation» and PA5.2 «Implementation of the Process Innovation» - characteristic for Level 5, «Innovative Process».
It should be noted that each subsequent level of process capabilities includes features of the process of a previous level to a greater extent. The analysis of the investigated approach has made it possible to conclude that a number of risk-dominant factors are located in the plane «Process Implementers - Process Owners - Means and Methods of Process Formalization - Means and Methods of Planning and Evaluating the Process - Factors».
The author of another approach [25] suggests five aspects that measure the maturity of a process. In this case, for each aspect, the following areas of analysis are defined:
1) design (the presence of goals of constructing a process, the degree of integrating the process with other processes of the organization and external processes, and the degree of documenting the process);
2) implementers (knowledge, skills, behaviour);
3) owner of the process (person, activity, authority);
4) infrastructure (information systems, personnel management);
5) factors (definition, use).
It should be noted that in the PEMM model, each criterion for evaluating the process is characterized by one of four levels of maturity, as opposed to the first approach:
- level Pr-1 - a reliable and predictable process;
- level Pr-2 - the process provides better results on the interfunctional level;
- level Pr-3 - the process provides optimal results at the interfunctional level, and it is integrated with other processes of the organization;
- level Pr-4 - the process is improving, going beyond the organization and extending from suppliers to consumers.
Thus, the theoretical analysis of standards [1, 20-22] and approaches to maturity assessment [23, 25] has helped determine the plane in which key risk-dominant factors are located. As a result of the analysis, it is concluded that these factors are located in the plane «Process Design - Process Implementer - Process Owner - Infrastructure for Implementing a Process - Factors for Determining and Using a Process». Next, a group of 20 qualified specialists was selected. The team included representatives of senior management and quality service, implementers and owners of QMS projects, internal auditors of three design institutions. The group was tasked with the «brainstorming» method to identify the range of key risk-dominant factors affecting the organizational maturity of the QMS projects in a given plane. The group recognized 13 key risk-dominant factors (mi, i = 1,13): m1 -the existence and essence of the objectives of constructing a process; m2 - the degree of the possibility of integrating the process with other internal and external processes; m3 - the degree of documenting the process; m4 - knowledge of process implementers; m¡ - skills of process implementers; m6 - behaviour of the process implementers; m7 - personality of the process owner; m8 - the activity of the owner of the process; m9 - authority of the process owner; m10 - the degree of use of information technology in the operation of the process; mu - the current system of hiring, training, remuneration;
m12 - certainty and transparency of indicators of process efficiency; m13 - the degree of applicability of the results of evaluation of the effectiveness of the process for its improvement.
The second stage. Choosing the approach to the organization of expert assessment and the means of obtaining expert opinions. In the framework of the proposed methodology, expert evaluation is organized according to the Delphi method. However, the distinguishing feature of the proposed methodology is the involvement of two expert groups in the examination. In the first group of experts, it seems appropriate to involve process implementers, and in the second group -the consumers of the process results and stakeholders. This approach to the formation of expert groups is based on observance of one of the principles of the standard [1] - «Consumer Orientation» and is the key to obtaining a balanced ranking of risk-dominant factors, taking into account the interests of process participants and stakeholders. Comparing the results of ranking the factors derived from these expert groups will allow the owner of the process to make a sound management decision on the directions for its improvement. In implementing this stage, it should be taken into account that the composition of the expert groups to evaluate different processes may vary.
The third stage. Making a reasonable choice of the optimal quantitative and personal composition of the expert group to rank risk-dominant factors. In the framework of the proposed methodology, the minimum number of experts nemin is proposed to be calculated by the formula [34]:
= 0.5 x
3
— + 5
(1)
where g is the possible error of the examination results (0 < g < 1). If we assume that the value of the reliability of the obtained result is equal to 85 % (that is, the magnitude of the error is 15 %), then according to formula (1) for expert evaluation, it is necessary to involve nemin = 12 experts. The data obtained do not contradict the data obtained by the approach given in [31].
Within the framework of the developed methodology for assessing the competence of the expert Ki, it is suggested to use the formula:
ki = XP K
(2)
j=1
where K is the coefficient reflecting the objective component of the competence of the ¿-th expert; Ki2 is the coefficient reflecting the level of professional knowledge of the ¿-th expert; Ki3 is the coefficient reflecting the level of professional skills of the ¿-th expert; Ki4 is the coefficient reflecting the level of the basic argumentation of the expert assessment of the ¿-th expert; Ki5 is the coefficient reflecting the level of managerial competence of the ¿-th expert; Ki6 is the coefficient reflecting the level of communicative competence of the ¿-th expert; Ki7 is the coefficient reflecting the personal qualities of the ¿-th expert, and P j are the weight coefficients of the individual coefficient Kj.
The determination of the coefficient reflecting the objective component of competence (Ki1) can be accomplished by self-assessment of the criteria given in [41]. These criteria include: education and a scientific level in the field of science and technology; overall work experience; experience in the relevant field of science and technology; work experience of an expert in the field of science and technology; work in the position. The criteria are evaluated on a 9-point scale.
n
The coefficient reflecting the level of professional knowledge (Ki2) can be estimated by testing experts. To do this, it is first necessary to distinguish the areas of special professional knowledge that should be assessed in order to decide on the professional competence of an expert in the field. Professional knowledge in the field of study includes the following:
1. Professional knowledge in terms of scope, identification, resources and communication of the investigated process with other processes.
2. Professional knowledge about the stages of the life cycle of the QMS process (planning, operation, monitoring and evaluation of the process).
3. Professional knowledge that reflects the main aspects of analysing and evaluating the risks and opportunities that arise during the operation process.
4. Professional knowledge that reflects the main aspects of identifying the critical control points needed to manage the risks and opportunities of the process.
5. Professional knowledge that reflects the main aspects of implementing corrective actions and opportunities for management at critical control points in the process of occurrence of inconsistencies.
To conduct testing, it is necessary to form test questions in accordance with the identified areas of professional knowledge and response to them. Test questions should be generated by analysing the QMS documentation that governs the evaluated process. The formation of such issues requires the involvement of specialists qualified in the field. Such specialists may include internal auditors, quality service personnel, etc. The testing scale is as follows: 40 % of tasks are correctly performed - 1-3 points; 41-69 % - 4-6 points; 70-89 % - 7-8 points; 90-100 % - 9 points.
The coefficients reflecting the level of professional skills (Ki3), managerial competence (K^) and communicative competence (Ki6) are determined by self-assessment. Thus, in determining Ki3, the experience of conducting analysis and solving organizational and systemic problems that arose during the operation of QMS processes of projects is evaluated. In designating Ki5, practical skills in planning, organizing, monitoring, and evaluating QMS projects are assessed. In determining Ki6, business communication skills are assessed in the course of performing QMS projects. The scale for estimating these coefficients has the following form: very superficial - 0 points; superficial - 1-3 points; not very deep - 4-6 points; deep - 7-8 points; very deep - 9 points.
To calculate the Ki4 factor, it is first necessary to determine the sources of reasoning that relate to the investigated sphere of expert evaluation. As a result of the analysis of scientific findings, point assessments of the proposed sources of argumentation in the field of study have been determined (Table 1).
The coefficient that reflects the level of the basic argumentation of the expert assessment (Ki4) is determined by self-assessment in the following way. The candidate for an expert is given a table without figures. The expert determines which source of argumentation he or she evaluates with grades B, C, and H. After applying an expert table to Table 1, the number of points is counted for all sources of reasoning.
The determination of personal qualities of the i-th expert (coefficient Ki7) is carried out on the basis of self-assessment (coefficient K'i7) and mutual evaluation by other experts (coefficient Kei7) by the formula [38]:
K,7=0.4K/7+0.6K7, (3)
Table 1
The benchmark for estimating the argumentation factor
Source of argumentation The degree of influence of the source of reasoning on the expert assessment
B (high) C (medium) H (low)
The presence of personal practical experience in the functioning of the process both within the organization and beyond its borders 0.4 0.3 0.2
Presence of personal practical experience within the organization related to the process 0.3 0.2 0.1
A theoretical analysis of both internal and external documentation that regulates the process is carried out 0.15 0.1 0.1
A theoretical analysis of the internal documentation that regulates the process is carried out 0.1 0.05 0.05
Other sources of argumentation (intuition, etc.) 0.05 0.05 0.05
Total 1.0 0.7 0.5
Among the personal qualities of the expert that determine the level of his or her competence, according to [32, 50], the accepted ones are the following:
- the desire for professional growth and continuing education in a particular and related fields;
- the ability to quickly evaluate the situation and make effective decisions;
- the ability to timely implement the decisions;
- the ability to create a normal psychological climate in the workforce;
- discipline and organization.
Besides the personal qualities of an expert in the field of study, the ones that should be added are such qualities as objectivity and analyticity, systematicity, and dynamism of thinking. An expert determines the degree of manifesting each of the identified personal qualities - both by him/her or colleagues - by using the verbal-digital scale (A;): always -1.0, almost always - 0.9, very often - 0.8, often - 0.7, oftener than on average - 0.6, on average - 0.5, rarer than on average - 0.4, rarely - 0.3, very rarely - 0.2, sometimes - 0.1, and never - 0.0. The coefficient Ki7s is determined according to the formula [32, 50]:
1 n
K'7 = N 5 A (4)
where N = 7 is the number of personal qualities of an expert.
Similarly, the coefficient of mutual evaluation of the personal qualities of the expert is determined by other experts (Ki7e).
The definition of the vector of the priority of the criteria of competence of experts in the investigated field is proposed to be carried out by the hierarchy analysis method (HAM) [60, 61]. This method determines that professional knowledge is perceived by experts as a more significant criterion of competence (P2 = 0.22765). Next, the criteria for competence were distributed as follows: an objective component of competence (P1 = 0.19752);professional skills (P3 = 0.18328); the level of basic argumentation of the
expert evaluation (P4 = 0.15347); managerial competence (P5 = 0.11553); communicative competence (P6 = 0.0676); personal qualities (P7 = 0.05495). The fourth stage. Obtaining and processing expert assessments. For this, the factors selected in the first stage of the methodology are assigned a rank in accordance with the views of experts regarding the degree of influence of a particular factor on the organizational maturity of a particular process. Ranking is based on the principle that the lower the rank, the greater the extent to which the factor affects the organizational maturity of the process. Thus, rank «1» has a factor with the highest degree of influence. If an expert recognizes several factors as equivalent, then these factors are assigned the same associated rank - the arithmetic mean rank of the places occupied by such factors. In addition, experts are invited, if necessary, to add factors that, in their opinion, have a significant effect on the organizational maturity of the investigated process and have not been included in the provided questionnaires.
Next, in order to obtain a group expert assessment, it is necessary to apply the method of arithmetic mean ranks.
n
For this purpose, the sum of the ranks ^ R (where n is
j=1
the number of experts interviewed and R is the rank of the i-th factor, assigned by the j-th expert) is calculated by each factor and divided by the number of experts n. Then, the deviation A is calculated from the average sum of the ranks for each of the factors according to the formula [68]:
n A m n
d v- m i iRr
j=i ¿=i j=i
(5)
W = -
121 ^r - 2 n(m + 1))
m lk
2(m3 - m) - nil(tj - j
(6)
at W = 1, the concordance is complete. With the value of the obtained coefficient, we can conclude that there is a chance or non-random agreement in expert assessments. Consistency is sufficient if W > 0.7.
It is then necessary to evaluate the statistical significance of the coefficient of concordance. The significance of the coefficient of concordance W is verified traditionally by the Pearson consistency criterion (%2 ). In the presence of related ranks, the calculated formula has the form [67]:
Xcalc.
1211 (T -12n(m +1))
1 m h
nm(n+1) - ii(tj- j
m -1 k=1 j=1
(7)
where Ai is the deviation of the sum of ranks of the i-th factor from the average sum of ranks; m is the number of factors;
1 m n
—^^ R is the average number of ranks.
m ¿=1 j=1
It is proved [68] that in order to obtain objective information, in addition to the method of arithmetic mean ranks, it is advisable to use the median rank method, which corresponds to a more precise averaging in the ordinal scale. Therefore, within the framework of the developed methodology, the ranking of factors is proposed to be carried out by two methods: the method of arithmetic mean and the method of median ranks.
Based on expert survey data, a composite matrix of ranks is formed and factors are ranked according to the rank ratings of expert groups.
The fifth stage. Evaluating the degree of coherence of expert opinions within the group. For this purpose, traditionally, the Kendall and Babington Smith coefficient of concordance (consistency) W is used, which is calculated by the formula [65-67]:
The calculated value of %2 is compared with the tabular value for the number of degrees of freedom (m-1) and at a given level of significance a (as a rule, it is sufficient to set the level of significance a in the range from 0.005 to 0.05). If the obtained value of x2alk is not less than of the table XtMs (%ifc - Xdahh ), then the coefficient of concordance W is statistically significant. Otherwise, it is necessary to organize an additional round of examination.
The sixth stage. Determining a generalized ranking of alternatives and analysing the results. At this stage, the arrangement of the factors of the matrix of the expert survey must be reformatted into a matrix of transformed ranks by the formula [68]:
(8)
where xmax is the maximum assigned rank. On the basis of the amount received, it is necessary to calculate the weighting features of risk-dominant factors.
The generalizing ranking of the priority of risk-dominant factors Rj in terms of competence is determined by the formula:
i Ko
R =
j=1
-, j = 1, m,
(9)
where Ik is the number of groups of equal ranks, introduced by the k-th expert; tkj is the number of small ranks in the j-th ( j = 1, lk ) group introduced by the k-th expert; t is the sum of ranks assigned to the ¿-th alternative; m is the number of comparative alternatives; n is the number of experts. The coefficient of concordance varies in the range 0 < W < 1. When W = 0, the consensus of expert opinions is absent, and
where Kcom is the coefficient of competence of the i-th expert, rj is the estimate given to the j-th alternative by the i-th expert; mj is the number of experts who rated the j-th alternative.
Next, it is necessary to sort the risk-dominant factors by the rank value from greater to smaller and to construct a diagram of the ranks. Thus, we obtain a nomenclature of ranked factors, where the serial number of a factor determines the degree of its influence on the organizational maturity of the researched process.
In the end, it is necessary to assess the degree of correlation of the assessments of the two expert groups. The tools for such an estimation may be the Kendall or Shukeni-Froly coefficients. Thus, the Kendall tau rank correlation coefficient t can be used as one of the criteria for evaluating the relative rating of two rank variables and the relationship between risk-dominant factors' ratings between the two expert groups. It is calculated by the formula [66]:
2 x (K - Q ) n x (n -1)
n
or by simplified the formulae [66, 67]:
t = 1 -
4 x Q n x (n -1)'
4 x K
T =--1,
n x (n -1)
(11)
(12)
where
n-1 n
Q= [№ < E, ] * [y, < y, ]]
i=1 ,=i+1
the number of inversions formed by the values of y,, placed in an ascending order of the corresponding Xi ; K is the sum of numbers, which shows how many ranks of the rank row Y exceed the first, second, and so on n-th rank; Q is a similar amount, which shows how many ranks of the row Y are below the first, second, and so on n-th rank.
If among the values of X and Y there is a coincidence value (Ei = Ev at i ^ v or yi = yv at j ^ v), then they are assigned average ranks. In this case, the coefficient t is corrected as follows [66]:
(K - Q )
1 n(n -1) - T^j 1 n(n -1) - U
(13)
where
the reverse dependence. If t = 0 or close to 0, this indicates a lack of connection.
Another tool for assessing the consensus of experts' opinions of two expert groups may be the Shukeni-Froly concordance coefficient [77]. Let the two groups of experts with the membership numbers m and n each set the task of ranking
k objects. Let us denote by R1.(i = 1,....m; j = 1,.....k) the ranks
proposed by the m experts of the first expert group and by R2(i = 1,....n; j = 1,.....k) the ranks proposed by the n experts of the second expert group (R1 = V R1 and R2 = V R2).
j V j V
According to these data, the Shukeni-Froly L statistics is calculated according to the formula [11, 67, 77]:
L=x RR
j=1
(16)
In this case, the value of the Shukeni-Froly L statistics is within:
mnk(k + 1)(k + 2) < < mnk(k + 1)(2k +1)
6
6
(17)
Taking into account the above, the general Shukeni-Froly coefficient of concordance W is calculated according to the formula [11, 67, 77]:
W =
L - M(L) Lmax - M(L)'
(18)
T =111, (ti -1)
2 ¿=1
and
U=1 («w -1);
2 w=1
where the average value of M(L) is calculated by the formula [77]:
M (L) =
mnk(k +1)2 4
(19)
Q is the number of combined ranks of X; t{ is the number of unified ranks in the i-th association on X; f is the number of united ranks of Y; Uw is the number of combined ranks in the w-th association on Y.
In order to check, at a given level of significance a, the zero hypothesis of the zero equation of the general coefficient of Kendall's rank correlation in the competing hypothesis that H1: t ^ 0, it is necessary to calculate the critical point by the formula [66]:
2 x (2n + 5) 9 x n x (n -1)'
(14)
where n is the sample size; Zcr is the critical point of a two-sided critical area, which is found by the table of the Laplace function by the equation:
O(20r ) = (1 -a)/2.
(15)
If < Tcr, there is no reason to refute the null hypothesis. The rank correlation between the factors is insignificant. If |t|< Tcr, the null hypothesis is denied. There is a significant correlation between the factors. In the general case, the coefficient t takes the value of the segments [-1;+1]. The value t = +1 indicates the straight linear dependence and the maximum tightness of the connection, and t = -1 indicates
In this case, of course, the variance D(L) is calculated by the formula [77]:
D(L) =
mn(k - 1)k2(k +1)2 144
(20)
If the result of calculating the value of W is close to +1, then this situation shows a high degree of consensus within both groups of experts and between groups. Close to -1 means a high degree of consensus within groups and a high degree of inconsistency between expert groups. Close to 0 means either non-consistency within expert groups or consensus within expert groups in case of inconsistency between them. Given the many advantages of this approach, it should be noted that it has its drawbacks. Thus, the limiting distribution of the Shukeni-Froly coefficient, which differs from the normal one, is not convenient to use [11].
Thus, the proposed method helps determine and organize the risk-dominant factors of influence on the organizational maturity of QMS projects, taking into account the competence of experts in the field. The application of the methodical outline of the article will help avoid a poor selection of experts and will make a transparent procedure for ranking of factors, which in turn will lead to expert-informed expert evaluations.
4. 2. Practical testing of the method of organizing risk-dominant factors
Practical testing of the developed methodology for ranking risk-dominant factors, which to the greatest extent influence the organizational maturity of the process, was carried out on the basis of the «Project Launch» process.
First, based on theoretical research, practical experience and the approach outlined in the first stage of the developed methodology, 13 factors were selected which, to a large extent, influence the organizational maturity of the «Project Launch» process.
Second, formula (1) determined the optimal number of two expert groups - 12 experts. This minimum number of experts does not contradict the amount determined by other approaches [31] - 4 and 7 experts.
Then, according to the third stage of the developed methodology, the competence of the candidates for experts was estimated according to the integrated competence indicator by formula (2), and their optimal number was selected. 12 experts were involved in the first group of experts, with the integrated competence factor Kcom > 0.83. The second group involved 12 specialists whose integrated competence coefficient was K > 0.87. As the determined level of com-
com
petence of the experts was more than the acceptable level proposed from [50] (Kcom > 0.67), it was expedient to involve the selected experts in order to rank risk-dominant factors.
At the next stage of approbating the developed method, two expert groups in the number of m=n = 12 experts conducted a ranking of m = 13 risk-dominant factors. The results of processing the rank evaluations of experts are shown in Tables 2, 3.
Table 2
The matrix of the results of processing expert assessments from the «Process Implementers»
Factors j=i i k n K ,=i j=i Rank medians A. A,2 The final rank by j=i The final ranking by the rank medians The sum of convertible ranks Weight X
mi i32.5 ii.04 ii.00 48.5 2,352.25 ii 11 23.5 0.025
m2 30.5 2.54 3.00 -53.5 2,862.25 3 3 i25.5 0.i34
m3 i7.5 i.45 i.00 -66.5 4,422.25 i 1 i38.5 0.i48
m4 7i.0 5.95 5.50 -i3.0 i69.00 6 5 84.5 0.090
m5 96.5 8.04 7.50 i2.5 i56.25 9 8 59.5 0.064
m6 ii8.0 9.79 i0.0 34.0 i,i56.00 i0 10 38.5 0.04i
m7 94.0 7.83 7.50 i0.0 i00.00 8 8 62.0 0.066
m8 80.0 6.7i 6.25 -4.0 i6.00 7 7 75.5 0.08i
mg 63.5 5.29 4.00 -20.5 420.25 4 4 92.5 0.099
mio 26.0 2.i6 2.00 -58.0 3,364.00 2 2 i30.0 0.i39
mii 70.0 5.87 5.75 -i4.0 i96.00 5 6 85.5 0.09i
mi2 i39.0 ii.58 i2.00 55.0 3,025.00 i2 12 i7.0 0.0i8
mi3 i53.5 i2.83 i3.00 69.5 4,830.25 i3 13 2.0 0.002
Total i,092 - - - 23,069.5 - - 934.5 i
Table 3
The matrix of the results of processing expert assessments from the «Consumers of the process results and stakeholders»
Factors j=i 4 k n k i=i j=i Rank medians A, A,2 The final rank by iR j=i The final ranking by the rank medians The sum of convertible ranks Weight X
mi 76.5 i2.00 i2.00 -7.5 56.25 7 7 79.5 0.084
m2 i9.0 i.58 i.50 -65.0 4,225.0 i 1 i37.0 0.i45
m3 22.0 i.83 i.75 -62.0 3,844.0 2 2 i34.0 0.i4i
m4 73.0 4.88 5.00 -ii.0 i2i.00 6 6 83.0 0.087
m5 i46.0 6.08 6.00 62.0 3,844.0 i2 12 i2.0 0.0i2
m6 i52.0 6.38 6.75 68.0 4,624.0 i3 13 6.0 0.006
m7 i24.0 i0.33 i0.50 40.0 i,600.0 ii 11 32.0 0.034
m8 ii6.0 9.67 i0.00 32.0 i,024.0 i0 10 40.0 0.042
m9 58.5 8.92 9.00 -25.5 650.25 5 5 97.5 0.i03
mi0 4i.0 8.58 8.00 -43.0 i,849.0 3 3 ii6.0 0.i23
mii 52.0 i2.5 i3.00 -32.0 i,024.0 4 4 i05.0 0.iii
mi2 i05.0 3.33 3.00 2i.0 44i.00 8 8 53.0 0.056
mi3 i07.0 4.25 4.00 23.0 529.00 9 9 49.0 0.05i
Total i,092 - - - 23,83i.5 - - 944 i
Further, during the approbation, the degree of agreement of expert assessments in each of the two expert groups was assessed. For this purpose, coefficients of concordance W were calculated according to formula (6) (relative to related ranks):
W =
w=-
12 x 23,069.5 122(133 -13)-12x 12
12 x 23,831.5 122(133 -13)-12 x 10
= 0.89,
- = 0.91.
(21)
(22)
Thus, the obtained values of the concordance coefficients W1 = 0.89 (for the first group of experts) and W2=0.91 (for the second group of experts) show a high degree of agreement of the expert opinions within each group.
At the next stage, the statistical significance of the determined concordance coefficients according to Pearson's criterion of agreement was analysed by formula (7):
Xcalc.1
12 x 23,069.5
12 x 13(13 +1)--— x 12
v 7 13 -1
- = 127.46,
Xcalc.2
12 x 23,831.5
12 x 13(13 +1)--— x 10
v 7 13 -1
- = 131.54.
(23)
(24)
m3 < m 10 < m 2 < m 9 < m ц < m 4 < m 8 < m 7 < < m5 < m6 < mi < m12 < m13.
(25)
The record m3 < m10
m2 < m 3 < m 10 < m 11 < m 9 < m 4 < m 1 < < m12 < m13 < m8 < m7 < m5 < m6.
Given the expertise of the experts, no significant differences between the two rankings are observed. The analysis is given in clause 5.
At the final testing stage, the degree of correlation between the two groups of experts was estimated by the value of the Kendall's rank correlation coefficient t, which was calculated by formulae (10)-(12), respectively:
т= 2 x (63 -15) = 0.62, 13 x (13 -1)
т = 1 4x 15 „ = 0.62,
13 x (13 -1)
т= 4 x63 -1 = 0.62.
13 x (13 -1)
Further, formula (15) was used to calculate Ф(гсг ):
Ф^сг ) = (1 -a)/2 = (1 - 0.05)/2 = 0.475.
(27)
(28)
(29)
(30)
Since the values of ¿fc, = 127.46 ^ = 21.02607 and XL.2 = 131.54 >xLfe = 21.02607 (at a given level of significance a = 0.05 and k = 12 as the number of degrees of freedom) have been calculated, the obtained coefficients of concordance Wi=0.89 and Wi=0.91 are observed as statistically significant.
Then we calculated the coefficients of weight X of risk-dominant factors, taking into account the competence of experts by formula (9).
In this case, the matrix of the experts' survey was reformatted into a matrix of transformed ranks by formula (8). The distribution of the weighting factors of the risk-dominant factors X for the two expert groups is presented in Tables 2, 3.
At the next testing stage, the risk-dominant factors were sorted by the value of the sum of the ranks received from smaller to greater. Thus, a nomenclature of the ranked factors has been obtained, where the serial number of the factor determines the degree of its affect. Thus, the risk-dominant factors by the rating of the impact on the maturity of the «Project Launch» process are estimated by the «Process Implementers» as follows:
Laplace's table was used to find zcr = 1.96. For the obtained data, a critical point was calculated according to formula (14):
T„ = 1.96
2 x (2 x 13 + 5) 9 x 13 x (13 -1)
= 0.41.
(31)
and so on indicates that the risk-dominant factor m3 has a greater influence on the maturity of the investigated process than the m1o factor.
At the same time, unlike the ranking of factors from the first expert group, according to the estimations of «Consu-mers of the results of the «Project Launch» process and stake-holders», the following ranking of the factors was obtained:
Because |t| < Tcr, there is no reason to refute the null hypothesis. The rank correlation between the estimates of the two groups of experts is significant. The conclusion is also confirmed by formula (16), calculated by the Shukeni-Froly coefficient of concordance (WW = 0.72).
6. Discussion of the results of the practical testing of the methodology for ranking risk-dominant factors
As a result of the practical testing of the developed method of ranking risk-dominant factors for the visual display of the degree of influence of isolated factors on the maturity of the «Project Launch» process, the ranks diagram was constructed. For the studied process, the diagram of the ranks is given, respectively, for the two groups of experts in Fig. 1, 2. The diagrams are constructed as follows: on the abscissa axis, the risk-dominant factors i = 1,13 are placed in order of increasing the sum of the ranks, and on the ordinate - as to the
n
sum of the ranks V Rj.
j=1
As can be seen from the diagram (Fig. 1), risk-dominant factors are estimated by the «Process Implementers» according to the degree of influence on the maturity of the «Project Launch» process in accordance with (25).
That is, to a greater extent, the maturity of the investigated process is influenced by the factor m3 «The degree of documenting the process,» and to a lesser extent, by the factor m13 «The degree of applicability of the results of evaluation of the
effectiveness of the process for its improvement». From the point of view of the process implementers, clear and transparent documentation of the process and the use of information technology in its performance are the factors that most affect the organizational maturity of the investigated process. In addition, a functioning effective system of hiring, training and remunerating in combination with the widely practiced activities of the process owner will ensure the interest of the implementers of the process in obtaining the desired results. At the same time, unlike the second group, the first group is least concerned about the degree of applicability of the results of the process efficiency for its improvement. This may be due to the fact that the experts of this group consider potentially unacceptable any change in the process functioning.
I I
The risk-dominant factors Fig. 1. A ranks diagram from the first group of experts
I
m2 m3 mlOmll m9 m4 ml ml2ml3 m8 ml m5 m6 The risk-dominant factors
Fig. 2. A ranks diagram from the second group of experts
On the other hand, as can be seen from the diagram (Fig. 2), the risk-dominant factors are estimated according to the evaluations of the «Consumers of the Process Results and Stake-holders» according to the degree of influence on the maturity of the «Project Launch» process in accordance with (26).
That is, according to the results of the expert evaluation of «Consumers of the Process Results and Stakeholders», it was established that to a greater extent the organizational maturity of the investigated process is influenced by the factor m2 «The degree of integration of the process with other internal and external processes,» and to a lesser extent,
the factor m6 «Behaviour of the process implementers». The results of the expert evaluation show that the interests of the second expert group are aimed more at the harmonious combination of the process with other processes of the system. The priority direction for consumers of the process results is transparency and adaptability of the considered process with other system processes as well as mutual understanding and established transparent relations with process implementers. At the same time, the activity and personality of the process owner, the skills and behaviour of the process implementers are not the key-priority risk-dominant factors. This result may be explained by the fact that parties that are interested in the result of the process operation are mainly concerned about the organizational maturity of the QMS of the project as a whole but not about the organizational maturity of a separate process. Besides, technical issues related to the operation of the process are not their object of interest in contrast to process implementers.
At the same time, it should be noted that, as estimated by both implementers and consumers of the investigated process, a clear documentation of the process will contribute to increasing the desired level of organizational maturity.
In addition, as a result of practical testing, attention should be paid to the fact that the rank correlation between the assessments of the two expert groups is significant. The assessments of the two groups of experts are mutually agreed (taking into account the expertise of the experts). Given this, we can conclude that in general, the ranking of risk-dominant factors takes into account the interests of the two groups of experts.
It should be noted that among a number of advantages of the proposed methodology, for example, the reformatting of experts' assessments from qualitative into quantitative, capacity and transparency of implementation in practice, there are a number of shortcomings. Among the shortcomings of the proposed methodology, it is necessary to note the subjective nature of expert assessments. However, one of the tools that will partially help overcome this disadvantage may be the involvement of qualified experts in the approach outlined in the proposed methodology.
7. Conclusion
1. Based on the theoretical analysis of standards for quality management and project management, well-known and widely applied in the practice of maturity modelling, 13 key risk-dominant factors have been identified, which potentially affect the organizational maturity of project quality management processes. It is concluded in the study that these factors are located in the plane «Process Design - Process Imple-menter - Process Owner - Infrastructure for Implementing a Process - Factors for Determining and Using a Process».
2. The methodology of ranking risk-dominant factors, which to the greatest extent influence the organizational maturity of quality management processes in projects, taking into account the competence of experts, has been developed. The basis of the developed methodology is the expert method. A distinctive feature of the methodology is to obtain an orderly list of key risk-dominant factors by involving two target groups, «Process Implementers» and «Consumers of the Process Results and Stakeholders» in expert evaluation. A systematic scientifically-based ranking of risk-dominant factors in accordance with the proposed methodology will
facilitate an objective assessment of the potential of quality management processes in projects. In addition, application of the developed methodology in practical activity will help determine the priorities among the directions of growth and organizational changes of processes for reaching the target levels of maturity. The proposed methodology can be implemented in the process of certification, self-evaluation and auditing of a QMS, as proven by the results of its practical testing using the example of the «Project Launch» process.
3. Practical testing of the proposed methodology was carried out on the example of the «Project Launch» process. According to the results of the expert evaluation of the «Pro-cess Implementers», it has been established that the factor «The degree of documenting the process» is influenced more by the maturity of the investigated process. In this case, the factor «The degree of applicability of the results of evaluation
of the effectiveness of the process for its improvement» is influential to a lesser extent. Based on the results of expert evaluation of «Consumers of the Process Results and Stakeholders», the following has been established. To a greater extent, the maturity of the investigated process is influenced by the factor «The degree of the possibility of integrating the process with other internal and external processes.» At the same time, the factor «Behaviour of the process implemen-ters» affects to a lesser extent. According to the calculated coefficient of concordance, a high degree of agreement of experts' opinions within each group was determined (0.89 for the first group of experts and 0.91 for the second group of experts). At the same time, there is a significant correlation between the estimates of the two groups of experts by the calculated coefficients of concordance according to Kendall (т = 0.62) and Shukeni-Froly (WW = 0.72).
References
1. DSTU ISO 9001:2015 (ISO 9001:2015, IDT). Systemy upravlinnia yakistiu. Vymohy. Kyiv, 2016. 22 p.
2. ISO 9001:2015. Quality management systems - Requirements. International Organization for Standardization, 2015. 29 p.
3. Harrington H. J. Project Management Excellence: The Art of Excelling in Project Management. Paton Press, 2006. 214 p.
4. Svod znaniy po upravleniyu biznes processami: BPM CBOK 3.0 / A. A. Belaychuka, V. G. Eliferova (Eds.). Moscow, 2016. 480 p.
5. Ahen D. M. CMMI: Kompleksniy podhod k sovershenstvovaniyu processov. Prakticheskoe vvedenie v model'. Moscow: MFK, 2005. 300 p.
6. Repin V. Biznes processy. Modelirovanie, vnedrenie, upravlenie. Moscow, 2013. 511 p.
7. DSTU ISO/IEC 33001:2016 (ISO/IEC 33001:2015, IDT). Informatsiyni tekhnolohiyi. Otsiniuvannia protsesu. Poniattia ta ter-minolohiya. Kyiv, 2016. 20 p.
8. ISO/IEC 33001:2015. Information technology - Process assessment - Concepts and terminology. International Organization for Standardization, 2015. 19 p.
9. Litvak B. G. Ekspertnye ocenki i prinyatie resheniy. Moscow: Patent, 1996. 271 p.
10. Hrabovetskyi B. Ye. Metody ekspertnykh otsinok: teoriya, metodolohiya, napriamy vykorystannia. Vinnytsia: VNTU, 2010. 171 p.
11. Introspektivniy analiz. Metody i sredstva ekspertnogo ocenivaniya: monografiya / Kryuchkovskiy V. V. Petrov E. G., Sokolo-va N. A., Hodakov V. E. Herson: Grin' D.S., 2011. 168 p.
12. Lavrish I. I. Samoorganizaciya ob'ektov upravleniya i mery soglasovaniya interesov sub'ekta i ob'ekta upravleniy // Avtomatizaciya i sovremennye tekhnologii. 2011. Issue 3. P. 36-41.
13. Radkevych A. V., Netesa A. M. Determination and ranging of organizational and technological factors that define the rational decisions of re-bars connection // Science and Transport Progress. Bulletin of Dnipropetrovsk National University of Railway Transport. 2017. Issue 3 (69). P. 171-181. doi: https://doi.org/10.15802/stp2017/104543
14. Hrabovetskyi B. Ye., Zianko V. V. Identyfikatsiya zmistu ta ranzhuvannia faktoriv, shcho obmezhuiut rozvytok venchurnoho biznesu, na osnovi metodu ekspertnykh otsinok Delfi // Visnyk Vinnytskoho politekhnichnoho instytutu. 2013. Issue 4. P. 46-54.
15. Hrabovetskyi B. Ye., Prytsiuk L. A. Otsinka priorytetnosti faktoriv, shcho vplyvaiut na zrostannia obsiahiv realizatsiyi avtomobiliv, zapasnykh chastyn, dokhodu vid nadannia posluh na osnovi metodu ekspertnykh otsinok Delfi // Visnyk Khmelnytskoho natsio-nalnoho universytetu. 2011. Vol. 2, Issue 5. P. 258-264.
16. Maslennikov E. V. Opportunities of use of expert knowledge as the source of concepts of development of the organizations // Moscow State University Bulletin. Series 18. Sociology and Political Science. 2017. Vol. 23, Issue 2. P. 229-249. doi: https:// doi.org/10.24290/1029-3736-2017-23-2-229-249
17. Martino Dzh. Tekhnologicheskoe prognozirovanie. Moscow: Progress, 1997. 591 p.
18. Beshelev S. D., Gurvich F. G. Matematiko-statisticheskie metody ekspertnyh ocenok. 2-e izd., pererab. i dop. Moscow: Statistika, 1980. 263 p.
19. Serenkov P. S., Romanchak V. M., Gurevich V. L. Minimizaciya pogreshnostey ocenivaniya pokazateley kachestva ekspertnymi sistemami // Pribory i metody izmereniy. 2010. Issue 1. P. 141-146.
20. DSTU ISO 10006:2005 (ISO 10006:2003, IDT). Systemy upravlinnia yakistiu. Nastanovy shchodo upravlinnia yakistiu v proek-takh. Kyiv, 2007. 27 p.
21. ISO 10006:2017. Quality management - Guidelines for quality management in projects. International Organization for Standardization, 2017. 34 p.
22. ISO 21500:2012. Guidance on project management. International Organization for Standardization, 2012. 36 p.
23. DSTU ISO/IEC 33020:2016 (ISO/IEC 33020:2015, IDT). Informatsiyni tekhnolohiyi. Otsiniuvannia protsesu. Struktura vymiriu-vannia protsesu dlia otsiniuvannia mozhlyvostei protsesu. Kyiv, 2016. 18 p.
24. ISO/IEC 33020:2015. Information technology - Process assessment - Process measurement framework for assessment of process capability. International Organization for Standardization, 2015. 18 p.
25. Hammer M. Faster Cheaper Better: The 9 Levers for Transforming how Work Gets Done. Crown Business, 2010. 302 p.
26. Askol'skaya E. A. Nekotorye aspekty metodiki otbora ekspertov na osnove ocenki stepeni doveriya k urovnyu professional'nogo suzhdeniya // Izvestiya OGAU. 2010. Issue 4 (28). P. 170-172.
27. Kryvda O. V., Voitiuk O. V. Model Delfi yak tekhnolohiya pryiniattia hospodarskykh rishen // Suchasni problemy ekonomiky ta pidpryiemnytsta. 2014. Issue 14. P. 257-262.
28. Horbatko V., Petrenko I. Metod «Delfi» ta spetsyfika yoho zastosuvannia u prohnoznykh rozrobkakh // Politychnyi menedzhment. 2008. Issue 6. P. 174-182.
29. Ekspertni otsinky. Metod «Delfi» yak tekhnolohiya pryiniattia upravlinskykh rishen / Kurtov A. I., Polikashyn O. V., Potikhen-skyi A. I., Aleksandrov V. M. // Zbirnyk naukovykh prats Kharkivskoho universytetu Povitrianykh Syl. 2017. Issue 1. P. 118-122.
30. Buharin S. N., Divueva N. A., Maryshev E. A. Vybor rezul'tiruyushchego ranzhirovaniya v processe nauchno-tekhnicheskoy eksper-tizi innovacionnyh proektov // Innovatika i ekspertiza. 2014. Issue 1. P. 114-120.
31. Ruposov V. L. Metody opredeleniya kolichestva ekspertov // Vestnik IrGTU. 2015. Issue 3 (98). P. 286-292.
32. Postnikova V. Analysis of approaches to formation of expert group membership focused on preparing and making decisions // Science and Education of the Bauman MSTU. 2012. Vol. 12, Issue 5. P. 333-346. doi: https://doi.org/10.7463/0512.0360720
33. Korobov V. B. Sravnitel'niy analiz metodov opredeleniya vesovyh koefficientov «vliyayushchih faktorov» // Sociologiya: meto-dologiya, metody, matematicheskoe modelirovanie (4M). 2005. Issue 20. P. 54-73.
34. Chernysheva T. Yu. Ierarhicheskaya model' ocenki i otbora ekspertov // Doklady TUSUR. Upravleniya, vychislitel'naya tekhnika i informatika. 2009. Issue 1 (19). P. 168-173.
35. Implementation of the hyerarchical multicriteria procedure of the evaluation of experts' quality / Putivtseva N. P., Igru-nova S. V., Beketova E. Y., Capitan S. A. // Research Result. Information technologies. 2016. Vol. 1, Issue 1. doi: https:// doi.org/10.18413/2518-1092-2016-1-1-39-47
36. Buharin S. N., Divueva N. A. Problema ocenki kompetentnosti s uchetom psihologicheskih svoystv // Innovatika i ekspertiza. 2013. Issue 1 (10). P. 108-115.
37. Korobov V. B. Organizaciya provedeniya ekspertnyh oprosov pri razrabotke klassifikacionnyh modeley // Sociologicheskie issledo-vaniya. 2003. Issue 11. P. 102-108.
38. Azgal'dov G. G. Kvalimetriya dlya vsekh: ucheb. pos. Moscow: ID InformZnanie, 2012. 165 p.
39. Velychko O., Gordiyenko T., Kolomiets L. A comparative analysis of results of the group expert assessment of metrological assurance of measurements // Eastern-European Journal of Enterprise Technologies. 2017. Vol. 6, Issue 9 (90). P. 30-37. doi: https:// doi.org/10.15587/1729-4061.2017.114468
40. Korobov V. B. Nekotorye problemy primeneniya ekspertnyh metodov na praktike // Nauchniy dialog. Estestvoznanie. Ekologiya. Nauki o zemle. 2013. Issue 3 (15). P. 94-108.
41. Velychko O. M., Hordienko T. B., Kolomiets L. V. Metodyka otsinky kompetentnosti ekspertiv z urakhuvanniam kharakterystyk nevyznachenosti danykh // Metallurh. y hornorud. prom-st. 2014. Issue 3 (288). P. 135-137.
42. Velychko O., Gordiyenko T. Evaluation of competence of the experts in field of metrology and instrumentations // XXI IMEKO World Congress «Measurement in research and industry». Prague, Czech Republic, 2015. 5 p. doi: https://doi.org/10.13140/ rg.2.1.2624.9365
43. Velychko O., Gordiyenko T., Kolomiets L. A comparative analysis of the assessment results of the competence of technical experts by different methods // Eastern-European Journal of Enterprise Technologies. 2017. Vol. 4, Issue 3 (88). P. 4-10. doi: https:// doi.org/10.15587/1729-4061.2017.106825
44. Otsiniuvannia kompetentnosti ekspertiv u sferi vyshchoi osvity / Velychko O. M., Hordienko T. B., Haber A. A., Kolomiets L. V. // Zbirnyk naukovykh prats ODATRIa. 2014. Issue 2 (5). P. 32-37.
45. Davydenko Ye. O. Formalizatsiya protsesu formuvannia skladu ekspertnoi hrupy dlia analizu ryzykiv IT-proektiv // Vestnyk KhN-TU. 2012. Issue 1 (44). P. 163-168.
46. Karatanov A. V., Druzhinin E. A. Informacionnye tekhnologii ekspertnogo ocenivaniya proektnyh resheniy pri formirovanii edi-nogo informacionnogo prostranstva // Zbirnyk naukovykh prats Kharkivskoho universytetu Povitrianykh Syl. 2014. Issue 3 (40). P. 155-160.
47. Kalinina I. O., Hozhyi O. P., Musenko H. O. Vrakhuvannia kompetentnosti ekspertiv u metodakh bahatokryterialnoho ana-lizu v zadachakh ratsionalnoho vyboru // Naukovi pratsi [Chornomorskoho derzhavnoho universytetu imeni Petra Mohyly]. Ser.: Kompiuterni tekhnolohiyi. 2013. Vol. 191, Issue 179. P. 116-123.
48. Leonov V. V., Voronych B. O. Metodyka otsinky kompetentnosti ekspertiv u protsesi rozrobky propozytsiy do prohramnykh do-kumentiv // Zbirnyk naukovykh prats Tsentru voienno-stratehichnykh doslidzhen «Natsionalnoho universytetu oborony Ukrainy imeni Ivana Cherniakhovskoho. 2014. Issue 3 (52).
49. Rayhman E. P., Azgal'dov G. G. Ekspertnye metody v ocenke kachestva tovarov. Moscow: Ekonomika, 1974. 151 p.
50. Podolianchuk S. V. Vyznachennia kompetentnosti ekspertiv z otsiniuvannia naukovoi diyalnosti u vyshchomu pedahohichnomu navchalnomu zakladi // Teoriya i praktyka upravlinnia sotsialnymy systemamy. 2014. Issue 4. P. 112-122.
51. Hordienko T. B., Velychko O. M. Metodyka otsinky kompetentnosti ekspertiv iz zastosuvanniam metoda analizu ierarkhiy // Metallurg. i gornorud. prom-st'. 2014. Issue 2 (287). P. 86-89.
52. Kadenko S. V., Tsyhanok V. V. Vyznachennia vidnosnoi kompetentnosti ekspertiv pid chas ahrehatsiyi parnykh porivnian // Reiest-ratsiya, zberihannia i obrobka danykh. 2017. Vol. 19, Issue 2. P. 69-83.
53. Polegen'ko A. F., Knyazskiy A. V. Ocenka otnositel'noy kompetentnosti ekspertov v ekspertnoy gruppe s ispol'zovaniem matric parnyh sravneniy // Ozbroiennia ta viyskova tekhnika. 2014. Issue 3. P. 49-55.
54. Kolpakova T. A. Opredelenie kompetentnosti ekspertov pri prinyatii gruppovyh resheniy // Radioelektronika, informatyka, uprav-linnia. 2011. Issue 1. P. 40-43.
55. Koczkodaj W. W., Szybowski J., Wajch E. Inconsistency indicator maps on groups for pairwise comparisons // International Journal of Approximate Reasoning. 2016. Vol. 69. P. 81-90. doi: https://doi.org/10.1016/j.ijar.2015.11.007
56. Popov G. A., Popov A. G. Rezul'tiruyushchaya ocenka pri nalichii neskol'kih variantov ocenivaniya na primere zadach informa-cionnoy bezopasnosti // Vestn. Astrahan. gos. tekhn. un-ta. Ser.: Upravlenie, vychislitel'naya tekhnika i informatika. 2017. Issue 1. P. 48-61.
57. Gucykova S. V. Metod ekspertnyh ocenok. Teoriya i praktika. Moscow: In-t psihologii RAN, 2011. 144 p.
58. Zadeh L. A. Fuzzy sets // Information and Control. 1965. Issue 8. P. 338-353.
59. Zadeh L. A. Knowledge Representation in Fuzzy Logic // An Introduction to Fuzzy Logic Applications in Intelligent Systems. 1992. P. 1-25. doi: https://doi.org/10.1007/978-1-4615-3640-6_1
60. Saaty T. L. The analytic hierarchy process. N.-Y.: McGraw Hill, 1980. 288 p.
61. Saaty T. L. Relative measurement and its generalization in decision making why pairwise comparisons are central in mathematics for the measurement of intangible factors the analytic hierarchy/network process // Revista de la Real Academia de Ciencias Exactas, Fisicas y Naturales. Serie A. Matematicas. 2008. Vol. 102, Issue 2. P. 251-318. doi: https://doi.org/10.1007/bf03191825
62. Podinovskiy V. V., Podinovskaya O. V. O nekorrektnosti metoda analiza ierarhiy // Problemy upravleniya. 2011. Issue 1. P. 8-13.
63. Podinovskiy V. V., Podinovskaya O. V. Eshche raz o nekorrektnosti metoda analiza ierarhiy // Problemy upravleniya. 2012. Issue 4. P. 75-78.
64. Korobov V. B., Tutygin A. G. Preimushchestva i nedostatki metoda analiza ierarhiy // Izvestiya Rossiyskogo gosudarstvennogo pedagogicheskogo universiteta im. A. I. Gercena. 2010. Issue 122. P. 108-115.
65. Kendall M. G., Smith B. B. The problem of m rankings // The Annals of Mathematical Statistics. 1939. Vol. 10. P. 275-287. doi: https://doi.org/10.1214/aoms/1177732186
66. Kendell M. Dzh. Rangovye korrelyacii. Zarubezhnye statisticheskie issledovaniya. Moscow: Izdatel'stvo «Statistika», 1975. 216 p.
67. Kobzar' A. I. Prikladnaya matematicheskaya statistika. Dlya inzhenerov i nauchnyh rabotnikov. Moscow: FIZMATLIT, 2006. 816 p.
68. Orlov A. I. Analiz ekspertnyh uporyadocheniy // Nauchniy zhurnal KubGAU. 2015. Issue 112 (08). URL: http://ej.kubagro.ru/ 2015/08/pdf/02.pdf
69. Orlov A. I. Noviy podhod k izucheniyu ustoychivosti vyvodov v matematicheskih modelyah // Politematicheskiy setevoy elektron-nyy nauchniy zhurnal Kubanskogo gosudarstvennogo agrarnogo universiteta. 2014. Issue 100. P. 146-176.
70. Kemeni Dzh., Snell Dzh. Kiberneticheskoe modelirovanie: Nekotorye prilozheniya. Moscow: Sovetskoe radio, 1972. 192 p.
71. Buharin S. N., Divueva N. A., Maryshev E. A. Vybor rezul'tiruyushchego ranzhirovaniya v processe nauchno-tekhnicheskoy eksper-tizy innovacionnyh proektov // Innovatika i ekspertiza. 2014. Issue 1 (12). P. 114-120.
72. Zhukov M. S., Orlov A. I. Zadacha issledovaniya itogovogo ranzhirovaniya mneniy gruppy ekspertov s pomoshch'yu mediany Kemeni // Nauchniy zhurnal KubGAU. 2016. Issue 122 (08).
73. Dvoenko S. D., Pshenichniy D. O., Popov A. V. Gruppovoe ranzhirovanie na osnove mediany Kemeni s metricheskimi svoystvami // Izvestiya Tul'skogo gosudarstvennogo universiteta. Tekhnicheskie nauki. 2017. Issue 10. P. 11-24.
74. Karatanov A. V., Druzhinin E. A. Informacionnye tekhnologii ekspertnogo ocenivaniya proektnyh resheniy pri formirovanii edinogo informacionnogo prostranstva // Zbirnyk naukovykh prats Kharkivskoho universytetu Povitrianykh Syl. 2014. Issue 3. P. 155-160.
75. Lazko I. Shaping managerial system by quality in project with use the flexible modules // Eastern-European Journal of Enterprise Technologies. 2014. Vol. 5, Issue 2 (71). P. 56-61. doi: https://doi.org/10.15587/1729-4061.2014.28336
76. Lazko I. Formation of quality management system rational model in projects // Proceedings XIV International Conference «Strategy of Quality in Industri and Education». Varna, Bulgaria, 2018. P. 355-361.
77. Schucany W. R., Frawley W. H. A rank test for two group concordance // Psychometrika. 1973. Vol. 38, Issue 2. P. 249-258. doi: https://doi.org/10.1007/bf02291117
