CC BY
0
DOI: https://doi.org/10.30525/2256-0742/2021-7-2-50-56
EVALUATION OF THE REGIONAL PUBLIC AUTHORITIES' ACTIVITIES: KEY THEORETICAL AND PRACTICAL ASPECTS
Iryna Honcharenko1, Olha Honcharenko2
Abstract. Activity of public administration is one of the most evaluated aspects involving hundreds of stakeholders across Ukraine. It is an important topic, as evaluation is one of the main ways for collection of the useful evidence for policymaking. Evaluation can be defined as the collection of evidence for formulating a judgement on the merits of an intervention, programme or policy. There is an unsolved task to provide an in-depth analysis with up-to-date information on the role of the evaluation of regional public authorities and local self-government institutions' activities. The article shows a number of evaluation methods for the public authorities' activities and reveals the factors that explain the current situation. The authors propose indicators that meet current trends, in particular the concept of sustainable development, within which the world community is moving. The system of balanced indicators allows to link strategic goals and key indicators that measure the degree of their achievement in the regions. The formation of a perfect evaluation system for the public authorities' activities includes monitoring of the obtained results, their comparison with the forecast and plan under the condition of clearly defined costs in advance. This involves the use of a set of indicators, in particular: indicators that characterize economic and financial sustainability, those which assess institutional development, social transformation, environmental responsibility and energy efficiency. The influence of the public opinion makes any management process more or less public, so it is necessary to intensify the process of the public monitoring of the services provided by the public authorities, with further response or accumulation of the information for further consideration.
Key words: evaluation, management, public authorities, region.
JEL Classification: C13, H70, R58
1. Introduction
In general, citizens aspire to see concise results in exchange of their financial and development support regional public authorities and local self-government institutions. A systematic and objective assessment of the public authorities' activities is one of the main conditions for increasing their socially useful activities and responsibilities, as well as accelerating the pace of socio-economic development of the regions. Therefore, in recent decades, discussions on the use of quality and efficiency indicators for objective assessment of the performance of public administration have intensified. Evaluation is an applied inquiry process that makes a judgement on the merits of an intervention, programme or policy (Kubera, 2017). Evaluation has a variety of operational goals, namely, to measure an effect, to understand mechanisms.
The question of the evaluation of the management effectiveness in general is reflected in the works of P. Drucker, who emphasized that "These days, practically all of us work for a managed institution, large or small, business or non-business. We depend on management for our livelihoods. And our ability to contribute to society also depends as much on the management of the organization for which we work as it does on our own skills, dedication, and effort" (Drucker, 2001). T. Peters, R. Waterman reveal the concept of successful management through continuous innovation (2010).
The theory and methodology of public administration evaluation is actively developing in the United States. The country received the experience of evaluation in education, public health and hygiene before the First World War.
The most turbulent period came in the 1960s (Tanaka Hiraki, 2009). At this time, the Johnson administration
Corresponding author:
1 Mykolaiv National Agrarian University, Ukraine. E-mail: ihoncharenko305@gmail.com ORCID: https://orcid.org/0000-0001-9670-9812 ResearcherlD: D-2222-2018
2 Kyiv National Economic University named after Vadym Hetman Kyiv, Ukraine. E-mail: Honcharenko15olha@gmail.com
ORCID: https://orcid.org/0000-0002-6860-9862
of the US federal government pursued a policy using the slogan "War on Poverty." This led to the creation of the measurement of efficiency as a scientific method of governance, which was traditionally developed in the study of public administration in the United States, and its spread since the 90s on the background of the so-called "new public administration". Now this approach is being implemented in administrative institutions all over the world. Moreover, various institutions are involved in its implementation both in the central and local governments.
The evaluation of the effectiveness and efficiency of public administration is based on the fact that management activities, which focus on achieving certain strategic goals by solving tactical tasks, from the point of view of public administration refocuses on the end result in the form of individuals and society satisfaction of services consumed by them, the consequences of their activities, as well as the volume and quality of public services provided by public administration, which improve the living conditions and livelihoods of citizens (Vedung, 2003).
2. Paradigms of evaluations
The competing paradigms of evaluations include positivism, realism and constructivism (Bachtler and Wren, 2006; Hoerner and Stephenson, 2012).
Positivism considers that objective knowledge can be obtained through empirical observation. It is strongly linked with the operational goal to measure the effects or impacts of an intervention, which can serve the purposes of accountability (evaluations as proof that funds are used as intended to deliver effects) and advocacy (evaluations as arguments for debates or negotiations). Consequently, this paradigm tends to give birth to so-called impact/result-oriented evaluations, which are explicitly designed to estimate the consequences of an intervention. Positivism is also related to monitoring/progress-oriented evaluations, which track the progress of an intervention towards set targets, without necessarily distinguishing the contribution of the intervention from other factors. This paradigm is usually operationalized through predominantly quantitative methods (statistical analysis, econometrics, etc.). Realism has an intermediate position regarding how knowledge can be derived - i.e., as an interplay between objective reality and its social construction by stakeholders. Its operational goals typically involve understanding the mechanisms of an intervention and learning during design and implementation. Its primary purpose is thus to promote policy-learning (i.e., improvement of an intervention or policy; but also building the skills and capacity of policymakers, which can be transferred to other tasks and policies). Realism can be applied to all types of evaluations, but especially to process/implementation-oriented ones, which
examine how funds are managed and interventions are implemented. It rests on a combination of qualitative and quantitative methods. Constructivism rejects objective knowledge and involves several stakeholders in evaluations to understand their different views and interdependence. It emphasises the development of participation and ownership, with a clear operational goal of understanding and learning from evaluations. Its purpose is mainly to build participation and engagement among stakeholders.
Beyond these theoretical perspectives, there are also practical aspects affecting evaluation, which include the work of evaluators but also the broader organizational context. For instance, evaluations are strongly dependent on the division of responsibilities between stakeholders. Indeed, as evaluations include a normative judgement, deciding who conducts an evaluation and who benefits from its results can be challenging. The timing of an evaluation is also critical in facilitating the opportunity for it to inform the decision-making process. Delays in evaluations can represent an important problem in the process of taking stock of the generated knowledge.
3. The current trend in assessing the quality of public authority
The current trend in assessing the quality of public authority is to build quality management systems in accordance with the requirements of international standards ISO 9001:2015. The basis of the quality management systems standards is formed by seven principles: customer orientation; leadership; staff involvement; process approach; improvement; making decisions based on factual data; relationship management. The requirements of the standard on the responsibility of management, analysis and control of business processes, actions to improve this activity, development of management system documentation create a basis for the formation of local government, which is focused on the customer (citizen). In Ukraine, local governments were the first among the authorities to introduce the ISO system. Implementation and certification of quality management systems in accordance with this standard leads to increased efficiency and consistency of work, more rational use of resources, focus on consumers and, consequently, increase of the customer satisfaction (SE "Ukrmetrteststandart, 2021).
Since 2000, the Common Assessment Framework (hereinafter referred to as the CAF model) has been widely used in the EU member states as well as in the EU candidate countries (EUPAN, 2021). The CAF model is an adaptation of the well-known business model for self-assessment of the European Foundation for Quality Management - the EFQM model of excellence. The overall CAF evaluation scheme is
being developed for the public sector and the public and municipal administration of Europe under the auspices of the European Commission. The CAF model has proven to be a simple and effective tool for evaluating, analysing and improving the efficiency of the civil service, as evidenced by the experience of more than 900 organizations in the field of public and municipal government. Between 2003 and 2006, about 30 European countries included CAF in their national strategic programs to improve the quality and efficiency of public administration. In 15 countries, the application of CAF is a recommendation for public authorities, and it is mandatory in three countries - the Czech Republic, Slovakia and Romania.
The CAF model is positioned as a mechanism for examining the activities of public authorities and local governments on the basis of diagnostic self-assessment, as a tool for comparative analysis of institutional systems of European countries, which includes identification of the best practices, and as part of public and municipal governance reform programmes. The general evaluation scheme is designed specifically for the public sector, taking into account its specifics and in order to achieve the following objectives:
- to introduce the principles of quality management in the field of public administration and promote their deployment through the method of self-assessment. To facilitate the transition from the "plan - do" chain to the "plan - do - check - act" cycle (PDCA continuous improvement cycle);
- to provide a mechanism for self-assessment for diagnosing the strengths and weaknesses, identifying areas for improvement;
- to become an element between different approaches to quality-based management;
- to ensure the exchange of experiences and the study of the best management practices for the public sector.
The CAF model includes two groups of evaluation indicators: the "Opportunities" group characterizes the approaches used by the organization to achieve results and increase efficiency; group "Results" characterizes the achievements of the organization. The nine key CAF indicators, which meet the criteria of the Common Assessment Framework (CAF), combine 28 components and about 150 evaluation areas. Today, the development of CAF is facilitated by the cooperation of users of the model within the European Network of Public Administration EUPAN, created at the CAF resource center.
For example, in the countries of the European Union, the practice of introducing "Smart City" technology, which provides, in particular, governance with the broad participation of citizens, is widespread. The list of already implemented services allows citizens to monitor and control around the clock: electronic auctions, electronic market analysis, electronic bidding, electronic auction card, mayor's diary, details of the city
budget and assets, city grants, a single emergency control center (fire service, patrol police, ambulance); online processing of various appeals of citizens (European Smart Cities 4.0, 2015). Such tools as electronic opinion polls or online citizens' notes / appeals make it possible to study citizens' opinions and take them into account when planning local development. The Best Value system was developed in the UK as a programme for improving the quality of local government activities, and it is the most important aspect is the cooperation of local authorities with the public, as public consultation is a key element in many issues of improving the quality of services. Not only the quality of services is discussed with citizens, but also the list of services, their goals and quality standards, according to which services are provided. In addition to discussions, the form of citizen involvement is cooperation in the process of providing services. As a result, a significant number of services are provided to local businesses on a contractual basis (not by local authorities, but by community residents). Consulting with the local businesses is an officially approved requirement of the Best Quality programme. The need for feedback between the community and the government is identified as one of the most important aspects of success in achieving the best quality of services. Public consultation is important at the budget planning stage of the service delivery process, as the financial responsibility for the services provided rests with the local government and, consequently, with the community as a whole (Kovbasuk U., 2014). Since 2001, Canada has used a model called the Community-Based Monitoring System. Such monitoring is defined by experts as a process of cooperation between the public, government agencies, industry, academics, community groups and local institutions to adequately respond to local development processes, address existing problems and promote full cooperation between citizens and government, strengthen citizen involvement in the adoption process of the management decisions at the local level (Quinn, 2005).
Balanced Scorecard has become a promising rating system for the new generation, the result of many years of work led by Robert Kaplan, a professor at Harvard Business School, and David Norton, founder and president of Balanced Scorecard Collaborative. The Balanced Scorecard system developed for business companies is gaining popularity among public administration institutions and takes into account four "perspectives" of the organization: traditional financial (Financial) indicators and factors that directly or indirectly affect them, the success of customer service (Customer), optimality of internal business processes (Internal Business) and general competence of the company's staff in its field (Innovation and Learning Perspective). Taken together, these perspectives provide a holistic picture of the organization's current strategy and dynamics. If necessary, additional kits
of own development can be introduced and used, for example "Ecology" and others (Norton, Kaplan, 1996). Periodic measurements of indicators provide feedback and appropriate regulation of the organization's actions. The degree of achieving goals, the efficiency of business processes and the perfomance of the entire company, its departments and each employee is determined by the values of the so-called "key performance indicators" (KPI). If they are related to the employee motivation system, it is expected that the latter will be interested in achieving the company's goals on a daily basis. Thus, Balanced Scorecard becomes a kind of "framework" for transforming the strategy of the organization into a set of operational goals that determine the company's behaviour and, consequently, its financial well-being.
4. Criteria and indicative approaches for evaluating the regional government system activities
An evaluation becomes increasingly important for the activities of public administrations in Ukraine. The current monitoring practice does not provide complete unbiased information for an objective assessment of the activities of public administrations. For example, focusing on the Strategy of development of the Mykolaiv region for the period till 2020 (Strategy of the Mykolaiv region, 2015), we should note that Section 5. - Implementation and Monitoring of the Strategy - contains subsection "Monitoring of the strategy implementation". The proposed procedure has several shortcomings: there is no independent monitoring of the development strategy implementation, no participation in the monitoring carried out by government officials, experts, initiative groups, NGOs, and public councils. The imperfection of the monitoring methodology allows the authorities to formulate the results of the monitoring according to their needs and tasks. Monitoring and evaluation of the regional development strategies are carried out on the basis of information provided by the documents of central authorities. The quality of the monitoring system of the regional development strategy does not allow to manage the implementation of these documents effectively.
Using criterion and indicative approaches for assessment of the regional government system activities, based on information from the Report about the implementation of the State Strategy for Regional Development of Ukraine for 2019, for which the Mykolaiv Regional State Administration is responsible (Report, 2019), the authors calculated specific criteria (Ki, n) and generalized criterion (K) (Table 1). The specific criterion for the evaluation of the activities of public authorities in the field of regional development was determined by certain evaluation indicators
specified in the system of indicators for evaluating the activities of a particular government body.
The specific criterion for evaluating the activities of public authorities is calculated by the formula (1): Ki = Fi/Pi, (1) where Ki is a specific criterion for the evaluation of the activities of public authorities; Fi is the actual achieved value of the i-th indicator; Pi is the predicted value of the i-th indicator. If the desired result is a decrease in the value of any indicator (for example, the level of registered unemployment), the calculation of a specific criterion is carried out according to the formula (2): Ki = Pi / Fi, (2).
The final evaluation of public authorities was based on calculation generalized criterion (K). The calculation was carried out according to the formula (3): K = the sum of Ki/i, (3). After the initial processing of information and obtaining the values of the generalized criterion (K) for each of the evaluated subj ects, these values were compared with 1. When the obtained K is more than 1.1, the activity of the subject of assessment is considered highly effective, when K = 1 +/- 0.1 - effective, when K = 0.8 +/- 0.1 -inefficient, when K is less than 0.7- ineffective. The calculation of the aggregate index showed that the activity of the subject of evaluation should be considered as effective.
The analysis of the Development Strategy of the Mykolaiv region for the period 2021-2027 (Development Strategy of the Mykolaiv region, 2021) revealed the absence of a separate section where the features of monitoring and evaluation were disclosed, but the necessary information was included in Section 6. In accordance with its powers and structure, regional state administration determines consistency of the strategy with programme and strategic documents, which stipulates that monitoring reports are open documents and are used for clarification of the tasks and budget programmes of the region for the next budget year, and also that the form of the monitoring report, responsible authority for its preparation as well as submission deadlines.
When comparing the Strategy of development of the Mykolaiv region for the period till 2020 (Strategy of the Mykolaiv region development, 2020) and the Development Strategy of the Mykolaiv region for the period of 2021-2027, absence of accurately formulated purposes and indicators which would correspond to SMART parameters Specific (Concrete); Measurable; Achievable; Relevant; Time bound (Weakness of time) is traced, as well as weakness of the information collection system, lack of procedures for monitoring and analysis of the information. It is necessary to create an integrated information and communication system for region, the main components of which will be: developed infrastructure for access to information; the information systems of local self-government bodies; the informational monitoring systems; e-commerce and
Baltic Journal of Economic Studies . „ --- Vol. 7 No. 2, 2021
Table 1
The evaluation of regional public authorities' activity according to the indicators of the Mykolaiv region in 2019
Specific evaluation criteria Forecasted value Actual value Index
Increasing of the region competitiveness level
Gross regional product (actual prices) per capita, UAH 37 391 60 549 1.6
Volume of sold innovative products,% of the total volume of sold industrial products 0.6 1 1.7
Number of small enterprises per 10 thousand of the available population, units 92 98 1.1
Disposable income per person, hryvnia 37 995 55 544 1.5
The volume of foreign direct investment per capita, USD 232 223 1.0
The volume of exports per capita, USD 1 667 1 913 1.1
Territorial socio-economic integration and spatial development
Demographic burden of the population aged 16-59 per 1 thousand permanent residents, ppm 641 659 1.0
Total coefficient of the outflow of rural population (outflow from rural areas per 1 thousand of available rural population), ppm 6 6 1.0
Total mortality rate (deaths per 1,000 population), ppm 13.9 15 1.1
The unemployment rate of the population aged 15-70, determined by the ILO methodology, % 8.2 9 1.1
Density of public roads of state and local importance with a hard surface, kilometers of roads per 1 thousand square meters. kilometers 194.5 195 1.0
The share of recycled waste, % of total waste generated 5 3 0.5
The share of the area of the nature reserve fund, % of the area of the administrative-territorial unit 7 3 0.4
Generalized criterion 1.08
Source: eompiled by the authors on the basis of: (Report, 2019)
marketing systems; the informational and consulting electronic services; distance learning and retraining systems (Honcharenko, 2018). The development of informatization and telecommunications of regions will enable to achieve significant results in social and economic activity of society at the expense of a clearer organization, economical spending of all kinds of resources (material, energy, labour, financial, etc.), improvement of working conditions and life of the population.
The task of assessment of the public authorities' activities seems to require great attention from scholars and practitioners of public administration.
Currently in the world practice of strategic management, the most effective tool for implementing the strategy is a system of balanced scores. Obviously, it is impossible to achieve what cannot be measured. Therefore, in order to implement the set strategic tasks, the main target indicators of sustainable socio-economic development of the community must be identified and targeted. The system of balanced scores allows you to link strategic goals and key indicators that measure the degree of their achievements, to identify and track cause and effect relationships between key financial and non-financial indicators.
The name of the system reflects the balance or equilibrium that can be achieved between:
- long-term and short-term goals of the development strategy;
- financial and non-financial indicators;
- indicators of the upper and lower hierarchical levels of the strategy;
- internal and external sources and factors of strategy implementation. Key performance indicators in the system should be measurable and formalized in a single reporting system according to the following criteria:
- be relatively simple and unambiguous in interpretation;
- have optimal, threshold, critical values for comparison and control over their implementation;
- be able to make a comparative assessment in time dynamics;
- be updated on a regular basis;
- be representative for comparisons;
- be able to be included in economic and mathematical models, information systems and forecasting systems.
The number of key performance indicators selected should be limited. It is impossible to make effective management decisions based on the analysis of too many indicators.
To achieve the strategic goals and objectives of the regional development, the following indicators are proposed: Economic and financial stability (Growth rate of the labour productivity, Number of created and modernized work places, Index of economic innovation); Institutional development (Level of competitiveness; Terms of doing business, Quality of the local government, Development of e-government); Social transformations (The share of the population with incomes below the subsistence level, Unemployment rate, Minimum income level);
Environmental responsibility and energy efficiency (Number of pollution sources, Reducing of the energy intensity of the economy, Reducing the energy intensity of the economy).
Economic and financial indicators are quite simple to define and widely used (Posner, Fantone, 2010; Soininen, 2013). But if we focus only on them, it is not necessary to even be interested in ensuring that investments improve the welfare of citizens. Therefore, development strategies need to be assessed both by the degree of achievement of previously set goals and by the extent to which the set goals are justified in terms of the existing social problems.
5. Conclusions
Society expects from the public authorities an effective and responsible management aimed at meeting both its needs and the needs of individual citizens.
In Ukraine, achieving the progress towards the formation of a perfect system for evaluating the activities of public authorities includes monitoring of the obtained results, their comparison with the forecast and plan at clearly defined costs. It involves the use of a set of indicators, in particular: indicators that characterise economic and financial sustainability,
which assess institutional development, social transformation, environmental responsibility and energy efficiency.
Public assessment of the local self-government effectiveness is common. Such assessment includes evaluation and control, in particular on such issues as, for example, the adoption of certain management decisions by public authorities, as well as the progress of their implementation; targeted and economical spending of resources and public funds; ensuring environmental safety, as well as preserving human life and health; quality and volume of public services provided, implementation of priority national programmes and projects, etc. The impact of the public opinion and its evaluation on the process of determining the effectiveness of public administration is carried out by implementing a system of its monitoring and taking into account its effects and their consequences.
It is necessary to intensify the process of the public monitoring of the public services delivery, public activities, institutions and civil society organizations for constant, periodic or one-time monitoring of the strategies implementation by public authorities with further response or accumulation of information for further consideration.
References:
Bachtler, J., & Wren, C. (2006). Evaluation of European Union Cohesion policy: Research questions and policy challenges. Reg. Stud. 40, 143-153. doi: 10.1080/00343400600600454
Berriet-Solliec, M., Labarthe, P., & Laurent, C. (2014). Goals of evaluation and types of evidence. Evaluation, 20, 195-213. doi: 10.1177/1356389014529836
Eupan (2021). CAF - common assessment framework. Available at: https://www.eupan.eu/caf/
Development Strategy of Mykolaiv region for the period 2021-2027. Available at: https://www.minregion.gov.ua/
wp-content/uploads/2021/02/strategiya-rozvytku-mykolayivskoyi-oblasti-na-period-do-2027-roku.pdf
Drucker, Peter F. (2001). The Essential Drucker. Available at: https://1lib.eu/book/781796/a83811?dsource= recommend®ionChanged=&redirect=7203773
European Smart Cities TU Vienna European smart cities 4.0 (2015). Available at: http://www.smart-cities.eu/ ?cid=2&ver=4
Hoerner, J., & Stephenson, P. (2012). Theoretical perspectives on approaches to policy evaluation in the EU: the case of Cohesion Policy. Public Adm., 90, 699-715. doi: 10.1111/j.1467-9299.2011.02013.x Honcharenko, I., Kozachenko, L., & Moroz, T. (2018). Informational support of the rural areas' development. Baltic Journal of Economic Studies, vol. 4, no. 4, pp. 93-100. doi: 10.30525/2256-0742/2018-4-4-93-99 Kovbasuk, U. V. (2014). Local self-government in Ukraine: current state and main directions of modernization. NADU, p. 128.
Kubera, P. (2017). A new approach to policy evaluation in the European Union. Przeglqd Politol, 35. doi: 10.14746/pp.2017.22.4.3
Norton, D., & Kaplan, R. (1996). Using the Balanced Scorecard as a Strategic Management System / Harvard Business Review.
Peters Tom, & Waterman Jr. Robert (2010). The Quest for Excellence: Lessons from America's Most Successful Companies. Alpina Publishers, p. 528.
Posner, P. L., & Fantone, D. (2010). Performance Budgeting. Available at: https://books.google.com.ua/ books?hl=en&lr=&id=c9sQNdWmlD0C&oi=fnd&pg=PA92&ots=lIq0DMhZNy&sig=rv21PhPCGTksGXRT-3DgmZI133G0&redir_esc=y#v=onepage&q&f=false
Quinn Michael S., & Dubois Jennifer E. (2005). Community Based Monitoring: Engaging and Empowering / Forest Service Proceedings RMRS-P-36; Faculty of Environmental Design, University of Calgary. Calgary, Ab, Canada: Alberta Ranchers USDA, pp. 212-218.
SE "UKRMETRTESTSTANDART" (2021). Quality management systems according to ISO 9001. Available at: http://www.certsystems.kiev.ua/uk/iso-9001/sistemi-upravlinnya-yakistyu-za-iso-9001.html
Strategy of Mykolaiv region development for the period till 2020. Available at: http://smartmk.bissoft.org/uk/ page/strategiya-rozvitku-mikolaivskoi-oblasti-na-period-do-2020-roku
Soininen, T. (2013). Mechanisms of change in public management projects - SAGE Open. Available at: https://journals.sagepub.com/doi/full/10.1177/2158244013486490
Tanaka Hiraki (2009). Evaluation in Local Governments in Japan. Available at: http://citeseerx.ist.psu.edu/ viewdoc/download?doi=10.1.1.526.4606&rep=rep1&type=pdf
Vedung, E. (2003). An assessment of the state policy and programs. "Everything is wrapped", p. 350.
Report on the tasks implementation r'' ' '' r ' 11 ' ent of Ukraine for 2019
https://economy-mk.gov.ua/index.pl ^ jriamky-diialnosti/rehio-
nalnyi-rozvytok/sotsialno-ekonomichnyi-rozvytok/stratehichne-planuvannia/187-stratehichne-planuvan-nia?Itemid=437
for which Mykolaiv regional state
authority. Available at:
