CC BY
7
Baltic Journal of Economic Studies tt. . „ , „„,_ -J--Vol. 5, No. 5, 2019
DOI: https://doi.org/10.30525/2256-0742/2019-5-5-136-144
ASSESSMENT SYSTEM OF THE PUBLIC PROCUREMENT EFFICIENCY
Mania Pysmenna1, Nataliia Pohribna2, Maryna Kalnytska3
Abstract. The purpose of the paper is to set out the in-depth study of how the public procurement performance can be determined through a system that considers the criteria for determining the best (ideal) supplier and the main elements of the assessment of tenders. In order to achieve the goals of the article the following tasks were set: 1) to analyze the development of scientific opinion in the field of public procurement performance assessment; 2) to provide the author's methodology for determining the criteria of the best (ideal) supplier; 3) to structure the elements of the public procurement procedures assessment system based on the criteria for selecting the best supplier. The paper addresses the need to achieve public spending efficiency by improving the public procurement assessment procedures to determine the best (ideal) provider. Methodology. The study is based on a careful analysis of the criteria for assessing the public procurement effectiveness under the evolution of scientific thought and practice of their implementation. Scientific literature expresses different and opposite views on the criteria, methods of assessment of the public procurement effectiveness by the functioning of customers. Limited research has been identified in the assessment of the procurement proposals at the stage of tendering and determining the best supplier. Results. The paper proposes a system for assessment of the most economically advantageous public procurement offer, taking into account the criterion of selecting the best supplier. This approach involves the unity of five stages of the procurement performance assessment. The characteristics of sequential execution of the assessment in five stages are given as follows: establishment of criteria for the ideal participant of public procurement, development of observations matrices for characteristics of the public procurement participants offers, determination of the vector-standard characteristics of the most economically advantageous proposal, calculation of distances between the characteristics values of the individual participants' offers and the ideal public procurement offer, the ranking of public procurement bidders, the economic advantage of the offer and the choice of its greatest advantage. The paper provides an opportunity to focus on a systematic approach to determining the public procurement effectiveness. This valuation approach provides a deeper justification for the choice of the procurement vendor, which allows expecting higher procurement performance. Practical implications. The assessment system for public procurement offers, including five stages of the use of procurement performance criteria, can be used by experts to determine the best supplier, regardless of the procurement regulation terms, as its elements provide a comprehensive description of the price and non-price criteria to achieve the effectiveness of procurement procedures. Value/originality. The paper suggests that the selection of public procurement performance indicators to be made within a criteria system based on the selection of the best (ideal) procurement provider. The authors presented their own vision of the components of the public procurement performance assessment system.
Key words: analysis, public procurement, assessment of procurement effectiveness, assessment criteria, procurement supplier.
JEL Classification: H57, H59
Corresponding author:
1 Kirovohrad Flight Academy of National Aviation University, Ukraine. E-mail: mari.pismennaya.83@ukr.net
ORCID: https://orcid.org/0000-0003-4509-8147
2 Taras Shevchenko National University of Kyiv, Ukraine. E-mail: nataliapohribna@knu.ua
ORCID: https://orcid.org/0000-0003-3867-9701
3 Uzhgorod National University, Ukraine. E-mail: kma2000@ukr.net
1. Introduction
The expectations of countries' citizens concerning the governments being able to provide them with a high social and economic standard of living are at odds because of the significant level of budget deficits. McKinsey (McKinsey 2017) estimates the deficit to be close to $ 4 trillion a year, so the need of using limited budget resources effectively requires a search for a solution.
The task of improving the ability of the public sector to use resources to achieve not only economic but also social outcomes requires productive public procurement procedures. The effectiveness of public funds being used for vital purposes needs to be measured reliably in order to spread innovation in the public sector and improve the economy. There has been limited progress in public procurement performance assessments to date. Basically, tools have been created to analyze the use of public funds in result of procurement procedures, and to test the effectiveness and efficiency of public spending. The deepening of the assessments of the expediency and productivity of public procurement, taking into account the assessments of the correct supplier selection at the stage of tendering procedures will allow achieving a considerable increase of productivity without increasing the expenses for these purposes. Thus, generalizing the scientific thought and practice of procurement regulation to analyze the public funds performance can potentially save, according to McKinsey (McKinsey, 2017), up to $ 3.5 trillion per year by 2021, which is equivalent to the entire global budget gap.
The rational organization of public procurement is directly dependent on ensuring the smooth functioning of enterprises and institutions, that is, customers of public procurement. In contemporary market there is a significant number of suppliers similar in quality properties and products characteristics, so it is important to make a reasonable choice of supplier. In such circumstances, of all possible alternatives, the supplier whose offer meets the specific needs of the consumer best should be preferred. Public procurement accounts for a large part of government spending, the amount of which depends on the standardized cases of compulsory procurement in accordance with the tender procedures. Public procurement in the EU countries averages between 34.8% and 20.3% of GDP per year, with a higher proportion in developed countries (44.5% in the Netherlands) and the lowest in Ukraine: 5.9%. Therefore, the issue of improving the productivity and efficiency assessment of the public funds usage for the needs of procurement of goods, works (services) needs to be addressed.
2. Research methodology
The main research question is a comprehensive study of public procurement performance indicators and the
interaction of analysis tools in the context of regulatory regulation in this area.
For this purpose, a deductive approach is used, which is suitable for this type of conceptual study. The dominant research methods in the literature review were analysis, synthesis, generalization and comparison. The literature review was conducted on the basis of primary sources, by querying journals from SCOPUS and the Web of Science. The term "public procurement performance" was used as a query keyword as it explored data sources and documents in the Google Scholar Search Engine. The method of component selection was used to systematize the structural relationships between the elements of public procurement, the price method was used to identify the qualitative characteristics of the assessment of procurement procedures, as well as the method of comparisons, procedures of ranking indicators, normalization of quantitative values of criteria, conversion of qualitative parameters into a numerical system of favors, matrix method. The method of taxonomic analysis was applied in determining the distances of the tender participants' characteristics from the vector-standard of the ideal proposal of the procurement object.
The results of the study were visualized using a graphical method. It allowed taking into account the multidimensionality of the public procurement performance indicators, to ensure the principle of theoretical and empirical unity.
3. Theoretical prerequisites of the study
The economic literature presents a variety oftechniques and approaches to assessing the public procurement effectiveness. Methodology of the efficiency estimation (Methodology, 2008) by calculation of indicators of absolute and relative effect from placing the state order by comparison of the prices in carrying out separately taken competition and the Lapin method (Lapin, 2016) are focused on the procurement activity assessment. In our opinion, these approaches are limited in view of the importance of the social function of public procurement in the country.
Ivanova O.V. (Ivanova, 2010) suggests a sectoral approach by determining the cost savings of public procurement, but the cost savings are calculated using the market price. And this is quite problematic in terms of access to information about signed identical agreements with homogeneous goods, works or services.
In our opinion, the main disadvantage of public procurement performance assessment methods is their justification for price alone. Thus, the quality characteristics of goods, works and services purchased on a competitive basis are ignored, and, therefore, the sense of holding such a competition is lost. Moreover, the assessment of public procurement performance is not taken into account at all. The latter is only about determining money efficiency.
Coglianese C. (Coglianese, 2012), Arykbaev R.K. (Arykbaev, 2010), and Perov K.A. (Perov, 2007) are the supporters of a slightly different position, who propose using the criteria ofcost-effectiveness, resource efficiency, taking into account the time factor, with a thorough analysis of the type, conditions and completeness of contracts, including public procurement, to assess the effectiveness of public procurement and the fulfillment of the planned public procurement indicators, which allows determining the degree of achievement for the set goals. The methodology for assessing public procurement systems (MAPS) is valuable in scientific terms. This methodology takes into account the OECD Council Recommendations on Public Procurement (OECD, 2015) and reflects the leading international fundamentals of such procurement, in particular the United Nations Model Law on Public Trade (UNCITRAL) on Public Procurement (UN, 2011), and the Directive 2014/24/EU (EU Parliament, 2014).
However, such approaches only consider the assessment from the point of view of the purchaser, not taking into account the procurement procedure as a two-way process between the procuring entity and selecting a potential supplier. Kumar A., Nair A., and Piecha J. (Kumar et al, 2015) emphasized this point but did not make any specific suggestions.
Currently, the public procurement effectiveness studies are focused on the study of intangible effects, such as reducing social tension in society. In order to fulfill the tasks of productive use of public funds
aimed at the procurement of goods, works (services), we suggest analyzing the supplier effectiveness in the complex of bidding procedures on price and non-price criteria, in order to take into account all components of procurement and to be able to perform their assessment.
4. The extended opportunities for systematic assessment
An analysis of the definition of "public procurement effectiveness" indicated the need to study the criteria for assessment of the bids of potential suppliers during the procurement process. Today, the main method of choosing the procurement item supplier is the offer price assessment method. So, in the first stage of the analysis, one can identify a better price offer and decide to cooperate with a specific supplier, which offers a lower price. However, all things being equal, and given the need for other assessment criteria to be taken into account, an in-depth analysis of the entire diversity of proposals is necessary, making it possible to formulate and make the final decision in the most objective way. Therefore, all the above points to the need to develop a methodology for assessment of the competitive bids and selecting the best bid and supplier.
We analyzed the development of the methodology and criteria for assessment of the bidders' proposals, depending on legislative changes in Ukraine (Table 1).
The analysis of the normative regulation of the price and non-price criteria for determining the successful bidder revealed the existing contradiction in the
Table 1
Evolution of assessment criteria for bidders in Ukraine
Period, year Assessment criteria Price weight, %
2000-2005 Lowest price; delivery time (execution); quality, aesthetic and functional characteristics, ecological cleanliness; after-sales service; terms of payments; the possibility of economic benefits arising from the tender offer; operating costs associated with the use of the completed facility; technology transfer and training of management, scientific and production personnel, including the use of local resources, the means of production, manpower and materials for the production of goods, performance of works, services offered by the contractor. 70
2006-2008 Lowest Price. The criterion for "rating of participant in the register of the procurement procedures participants" was introduced. The list includes participants on the basis of their request in accordance with the rating, depending on the number of procurement procedures, where the participant participated, the number of wins in the procurement procedures, the number of completed on time and in full volume public procurement contracts, other criteria. 70
2008-2010 Lowest Price; delivery time of goods, performance of works, provision of services quality and functional characteristics, ecological cleanliness; after-sales service; terms of payments; operational expenses; technology transfer and training of management, scientific and production personnel, including the use of local resources, production facilities, labor and materials for the production of goods, performance of works, provision of services offered by the participant. 70
2010-2016 Lowest price, quality of work, services, payment terms, terms of service, warranty service, operating costs, technology transfer and training of managerial, scientific and production personnel, including the use of local resources, including means of production, labor and materials for production of goods, performance of works, provision of services offered by the participant. 50
2016 -till present Lowest price, payment terms, term of service, warranty service, operating costs, technology transfer and training of managerial, scientific and production personnel. This is carried out automatically by the electronic purchasing system. 70
Source: developed by the authors using the materials of regulatory documents in the field of public procurement of Ukraine
assessment of the most economically advantageous proposal of the bidders. Thus, the successful tenderer is the supplier who has offered the minimum price, in spite of the bid determined best by the results of the assessment of tenders.
Such a paradox is conditioned by the desire of the state to provide maximum cost savings when conducting public procurement. Requirements for the subject of procurement, for which the selection of tenders is carried out, are reflected in the tender documentation. In particular, tender documentation for public procurement contains the technical, qualitative and functional characteristics of the procurement subject. It can set requirements for the efficiency, environmental friendliness and safety of products, their quality and design, the features of testing to assess compliance with the marking, packaging, etc. In order to take into account the criteria for the procurement subject other than price, the tender documents determine (if possible) their value equivalent or the proportion of those criteria as a percentage of the overall assessment of the tender proposal, as shown in Table 1. This percentage varies depending on the state's desire to promote greater efficiency in the use of public funds through the regulatory influence on determining the procurement winner.
The customer is obliged to establish one or more of the following qualifying criteria for the participants:
1) availability of equipment and material base;
2) availability of appropriately qualified workers, who have the necessary knowledge and experience; 3) the existence of a documented experience of performing a similar contract. In determining qualification requirements, the customer should avoid establishing unreasonable requirements that restrict competition and discriminate against participants, in particular: the availability of own production base, the number of own personnel, confirmation ofthe declared volume ofgoods in the warehouse of the participant, the availability of experience with budget organizations, and more.
We have found that the assessment of bids by participants in the European public procurement practices uses price and non-price criteria. The latter are used when public procurement is characterized by complexity or specificity in terms of, for example, procurement of consultancy services, research, experiments or development, and more. The share of the criterion "price" must be no lower than 70%. Accordingly, the maximum share of all price criteria cannot exceed 30% (Figure 1).
The unjustified narrowing of the grounds for the application of non-price assessment criteria (only for complex purchases) prohibits their application to purchases of goods, services or works that have a permanent market. This leads to the risk of identifying unscrupulous suppliers with products of poor quality. At the same time, the problem with identifying
procurement items that have a constantly functioning market and the lack of legislation to identify the complex or specialized nature of procurement remains. We consider that the application of non-price criteria is of the utmost importance, since other criteria for the assessment of bidders than the price, in the legislation is provided in order to maximize the approximation of the assessment method to the procurement subject, taking into account its qualitative and quantitative characteristics, as well as the needs of the customer.
If, in addition to the price, other assessment criteria are used to determine the most economically advantageous tender, then the quoted price is the price indicated by the tenderer and calculated with reference to other assessment criteria according to the formula specified by the contracting authority in the tender documentation, namely:
Quotedprice = Bidprice / correctioncoefficient, (l)
The correction coefficient is calculated by the formula:
cc^l + (Cl + C2 + C^ (2)
PW
where, CC - is the correction coefficient;
Cl... Cn - value of each non-price criterion selected by the participant;
PW - the weight of the "price" criterion.
Unlike in Ukraine, the concept of value-for-money (VfM), which is used to assess the public administration effectiveness, is quite common in internationally. The emergence of this concept is linked to public sector reform processes and the budget process towards implementing the New Public Management ideology.
The term "value-for-money" includes the optimal use of resources to achieve the outlined results. Under these conditions, it is not intended to produce the greatest savings, but to achieve the best possible result, given the existing constraints, the consistency of the costs incurred and the results obtained. According to the value for money, the final contract price cannot be the main criterion for selecting suppliers. Moreover, there are a variety of other factors that influence the decision to implement a particular contract, namely: the offer's risks and the experience of potential suppliers, financial conditions, including the volume of costs throughout the procurement cycle, contract flexibility, the availability of various alternatives to the latter, etc.
Directive 2014/24/EU allows setting of performance and functionality requirements as an alternative to technical specifications. Such specifications describe the functions and functional characteristics, the consumer properties of the equipment, the results of works and services.
The EU is now showing a tendency of rejection the lowest price indicator as the sole criterion for selecting the best tender. The most economically advantageous offer is determined not only by price, but also by such non-price criteria as: life cycle cost of goods, quality,
Grounds for the non-price criteria application
Price
In the case of procurement of goods, works and services that are produced or provided not on a separately developed specification (technical description) for which there is a constantly active market
3IT
price, along with other assessment criteria: payment
terms, term of service, warranty service, operating costs, technology transfer, and management, scientific and production personnel training
In the case of procurement of a complex or specialized nature (including consultancy, research, experimentation or development, research and development).
The specific weight of the price criterion may not be lower than 70 percent, except in the case of competitive dialogue procedure
Figure 1. Grounds for application of non-price criteria for assessment of bids of bidders
Source: developed by the authors according to Directive 2004/18/EU and Directive 2014/24/EU
including technical advantages, aesthetic and functional characteristics, accessibility for all categories of users, social, environmental and innovative characteristics; the qualification and experience of contractors if it has an impact on the level of contract performance; inservice maintenance and technical assistance, delivery conditions. It is important that contracting authorities are entitled to reject tenders if the price offered is abnormally low compared to others and such tenderer cannot support the price.
In addition to the "quoted price" method, in combination with price and non-price selection criteria, international practices also apply the point method. The latter is based on the selection of a defined set of points to assessment the selection criteria for suppliers. Considering the importance of each selection criterion, the supplier who received the highest score is recognized as the winner. In this case, linear scoring is used, for which all criteria with defined weights are added both to obtain an overall score and to score points on a parabolic scale:
PP = (Pmin / Pi)100 , (3)
where PP - the price point;
Pmin - the minimum bid price (the bid of the bidder who submits the minimum bid is awarded 100 points);
Pi - the price of the bidder's bid.
It is worth noting that it is also possible to combine scoring and rating methods for assessment of the tender proposal, namely:
Ra = ■ 100 , (4)
where, Rat - is the rating of the criterion application;
Pmax initial price;
Pi - the price of the bidder.
The degree of importance or significance of the selection criteria is taken into account for determining the integral assessment where non-price criteria are used. To this end, a specific gravity coefficient is determined for each criterion, reflecting the relative importance of the tender assessment criterion. In doing so, the coefficients of the specific gravity of the criteria are selected taking into account the possibility of meeting specific goals and objectives of competitive procurement of resources. In this regard, the proportions of the specific gravity ratios of the various criteria may vary. Thus, when determining the proportion of the
Table 2
Features of the assessment criteria significance for the competitive bids
Criterion Feature
Ranking assessment criteria are ordered on decreasing importance
Assessment weighting of criteria according to their importance
Paired comparison comparison of the criteria with one another in order to be able to clarify their ranking and give them a specific weight
Sequential comparisons including ranking operations, formulating the requirements for the criterion that is the first in the ranking, the highest specific gravity, and quantifying the weight of the following criteria, comparing them with the first.
Source: developed by the authors
criteria, we compare the significance of the criteria included in the comprehensive assessment using the procedures summarized in Table 2.
After normalized values are determined for each criterion value for all participants of the competition, their calculation is carried out taking into account the established coefficients of the specific gravity of such criteria. Then, the values obtained are summed up by all the assessment criteria for each participant.
Thus, the integral assessment of the bid of the i-th participant is determined by the formula:
Z>=tBrNt, (5)
M
where, Zi - integral assessment of the bid of the i-th participant;
Bj - the specific gravity of the j-th criterion;
Nij - the normalized value of the j-th assessment criterion of the i-th participant;
n - is the number of criteria to be assessed.
In view of the above, the participant who received the highest integrated rating and took first place in the rating is recognized as the winner.
A striking example of the use of the point method of tender selection is the application of the criteria
for assessing the participants' goodwill in public procurement, identified in Figure 2.
Considerable advantage of the European practices of qualifying participants is the establishment of requirements for their financial capacity. Financial ability is usually understood as an enterprise having sufficient financial resources to execute a contract. Smyrichinsky VV. (Smyrichinsky, 2002) connects establishing such requirements with the need to have confidence that during the term of the contract the enterprise will not be liquidated, will have sufficient resources to fulfill the contract and other obligations.
According to Directive 2014/24/EU, a tenderer must confirm that there is a turnover twice the estimated value of the contract. Financial ability can also be measured by the ratio of assets to liabilities. In such cases, customers are required to determine the method of calculation. Otherwise, if participants were allowed to make calculations on their own principles, setting a minimum allowable level would not make sense. In our opinion, in an unstable financial and economic situation, assessing the financial capacity of participants is an important criterion for their qualification selection. In view of the above, we suggest using a set of Debt to Equity Ratio,
Figure 2. The set of criteria for assessing the goodwill of participants in public procurement
Source: developed by the authors under the Supplier Assessment System
Figure 3. Public procurement performance assessment system
based on the criterion of selecting the most economically advantageous tender
Source: authors' development
Debt / EBITDA ratio, current ratio, interest coverage ratio (ICR) when selecting participants.
Generalizations of the criteria described in the tender offer assessment allow them to be applied to identify the most economically advantageous (ideal) bid of the tenderer. It comes down to determining the quantitative and qualitative characteristics of the ideal offer of the public procurement participant, after which the offer of each participant is compared with the developed ideal. Accordingly, winning the tender will belong to the bid of the tenderer whose characteristics are most closely related to the characteristics of the ideal bid.
The logical scheme of the public procurement performance assessment system is shown in Figure 3. Note that the approximation of the features of the public procurement participants to the ideal participant is suggested to be determined using the taxonomic method of analysis (Krisak, 2014).
Thus, the first stage of the logic scheme in Figure 3 is the stage of establishing the criteria for an ideal participant in public procurement. At this stage, the selection of the most important criteria for the assessment of participants is made, the relative importance of each assessment parameter is determined and the method of assessment of results is selected. It is important to take into account the objectivity and quantifiability of the tender assessment criteria. In the case of non-quantitative criteria, the opinion of experts should be used in accordance with the rules and procedures required to implement the expert methods (the expert commission establishment, outlining the
expert questionnaire procedure, processing the expert information, algorithm for constructing the general assessment).
The second stage of the system is the stage of developing the observations matrices for the proposals features of participants of public procurement:
x=
-ill ■■■ xlj x2\ x22 ■■■ x2j
xml xm2
,\mj
X1 n x2n
(6)
where, i - is the order number of the bid assessment indicator (1 to m); j - is the order number of the proposal (from 1 to n); Xij is the value of the coefficient i for the proposal j. Given that all the criteria have different dimensions, when developing the observation matrix, it is necessary to normalize the values of indicators:
1) among the indicators for which the minimum value is optimal, the unit is assigned to the minimum indicator and the rest is calculated by dividing the value of the minimum indicator by the corresponding indicator;
2) among the indicators for which the maximum value is best, the unit is assigned to the maximum value and the rest is determined by dividing their value by the maximum value.
For quality features, we suggest using Harrington's desirability function. The basis of its construction is the transformation of the natural values of the
partial parameters to a single dimensionless scale of desirability, while using standard estimates on the scale of desirability (Shutyak, 2010).
After normalization of indicators' values the observation matrix takes the following form: zu zu ... Zy ... zln z2n
Z =
z21 z22
zil zil
(7)
zml zin2 — zmj
In the third stage, using the taxonomic method, it is necessary to develop a vector-standard of characteristics of the most economically advantageous offer:
The fourth stage of selecting the most economically advantageous public procurement offer involves calculating the distances between the values of the characteristics of the individual participants' proposals and the vector-standard cm, based on the application of the Euclidean distance function:
(9)
Calculating the average distance between observations c0, the standard deviation of this distance 00, the maximum possible deviation from the standard c0, which makes it possible to determine the distance of each set of observations from the standard, and summarize the economic value of the proposals of participant's t<:
co
nl<=\
=!-£» , c0
(10)
(11) (12) (13)
where: 0 < t < 1. Hence, the public procurement participant's rating is higher; the closer Ti is to 1.
By designing alternative scenarios for the procurement of goods, works, services, contracting authorities
should analyze the benefits and losses of centralized procurement, whereby several government contracting entities can pool procurement to achieve economies of scale.
The fifth stage allows us summarizing all the performed calculations and rank the bidders based not only on the price but also on the quality indicators of their offers. In general, the public procurement efficiency system covers the stage of tendering taking into account the criteria for the selection of the best offer.
5. Findings
Thus, summing up the results of the research, we can determine the main components of public procurement effectiveness through a step-by-step assessment system: Stage 1 is establishing the criteria for an ideal public procurement participant, Stage 2 is developing the observation matrices of the tenderers' proposals features, Stage 3 is the determination of the vector-standard characteristics of the most economically advantageous proposal, Stage 4 calculates the distance between the values of the characteristics of the proposals of individual participants and the vector-standard features of ideal offer in public procurement, Stage 5 involves ranking the participants of government procurement and selection of the most advantageous offer.
6. Conclusions
The presented authors' view of the procurement performance assessment system will allow customers and other interested users to provide the comprehensive assessment of the major components that influence the decision on the best offer in the tendering process to select the successful bidder. This approach provides the possibility of productive use of public funds, taking into account the social demands and expectations of the citizens of the country.
In general, the performance of public spending on public procurement is largely determined by the effectiveness of the public procurement mechanism, in particular in the assessment of the proper fulfillment of procurement obligations, which will be addressed in future studies.
References:
Arykbaev, R. K., & Gadzhieva, V T. (2010). Formirovanie i razvitie kontraktnoj sistemy razmeshcheniya gosudarstvennogo zakaza v RF [Formation and development of a contract system for placing a state order in the Russian Federation]. Business. Education. Law. Bulletin of the Volgograd Institute of Business, 3(13), 20-27. (in Russian) Coglianese, C. (2012). Measuring Regulatory Performance: Evaluating the Impact of Regulation and Regulatory Policy, OECD, Paris. Retrieved from: https://www.oecd.org/gov/regulatorypolicy/1_coglianese%20web.pdf Directive 2004/18/EU of the European Parliament and of the Council from 31.03.2004 on the coordination of procedures for the award of public works contracts, public supply contracts and public service contracts. Official Journal of the European Union. L 134. 30.4.2004, p. 114.
Directive 2014/24/EU (2014) of the European Parliament and of the Council "On public procurement and repealing Directive 2004/18/EU" from February 26, 2014. Retrieved from: http://eurlex.europa.eu/legal-content/EN/TXT/?uri=uriserv%3A0J.L_.2014.094.01.0065.01.ENG
Baltic Journal of Economic Studies tt. . „ , -J--Vol. 5, No. 5, 2019
Ivanova, O. V. (2010). Metodika kompleksnoj ocenki effektivnosti gosudarstvennyh zakupok Orlovskoj oblasti [Methodology for a comprehensive assessment of the effectiveness of public procurement in the Oryol region]. Izvestiya TulGU. Economic and legal sciences, 2, 183-192. (in Russian)
Krisak, A. I. (2014). Taksonomichnyj analiz yak metodologichnyj pryjom ocinyuvannya efektyvnosti regulyuvannya zemelnyx vidnosyn [Taxonomic analysis as a methodological technique for assessing the effectiveness of regulation of land relations]. Economic analysis, 17(1), 66-70. (in Ukrainian)
Kumar, A., Nair, A., & Piecha, J. (2015). Measuring Efficiency in International Public Procurement. Journal of Public Procurement, 15/3, 365-389. Retrieved from: http://ippa.org/images/JOPP/vol15/issue-3/Article_5_ Kumar-et-al.pdf
Lapin, A. E., Kiseleva, O. V., & Kumundzhieva, E. L. (2016). Podhody k ocenke effektivnosti kontraktnoj sistemy v sfere gosudarstvennyh i municipal'nyh zakupok [Approaches to assessing the effectiveness of the contract system in the field of state and municipal procurement]. Business. Education. Right. Bulletin of the Volgograd Institute of Business, 1(34), 30-35. (in Russian)
McKinsey (2017). Government Productivity. Retrieved from: https://www.mckinsey.com/~/media/McKinsey/ Industries/Public%20Sector/Our%20Insights/The%20opportunity%20in%20government%20productivity/ Executive%20summary%20Government%20productivity%20unlocking%20the%2035%20trillion%20 opportunity.ashx
Metodika ocenki efektivnosti (2008). Metodicheskie rekomendacii po osushchestvleniyu ocenki effektivnosti i prozrachnosti razmeshcheniya gosudarstvennyh i municipal'nyh zakazov [Methodological recommendations for assessing the effectiveness and transparency of the placement of state and municipal orders]. Moscow: Delovoy Dvor. (in Russian)
OECD (2019). MAPS Methodology for Assessing Procurement Systems. Retrieved from: http://www.oecd.org/ gov/ethics/benchmarking-assessment-methodology-public-procurement-systems.htm
Perov, K. A. (2007). Ocenka effektivnosti razmeshcheniya zakazov: obshchie podhody [Evaluation of the effectiveness of placing orders: general approaches]. Government order: management, placement, provision, no 7 (January, March), pp. 77-83. (in Russian)
Shutyak, Y. V (2010). Vykorystannya funkciyi bazhanosti dlya ocinky ekonomichnoyi bezpeky pidpryyemstva [Using the desirability function to evaluate the economic security of an enterprise]. Scientific studios, 7, 147-154. (in Ukrainian)
Smirichinsky, V. V. (2002). Logistychnyj menedzhment derzhavnyx zakupivel Teoretychno-pravovyj ta metodologichnyj aspekt: Monografiya [Logistic management of public procurement. Theoretical and methodological aspects: monograph]. Ternopil: Carte blanche. (in Ukrainian)
Supplier Assessment System (2015). Retrieved from: http://images.myshared.ru/9/925647/slide_1.jpg
