CC BY-NC-ND
56
Original article
DOI: 10.17323/1999-5431-2023-0-5-165-188
EVALUATION OF PUBLIC SECTOR INNOVATION: SYSTEMATIC REVIEW
Kassia Tonheiro Rodrigues1, Lucas dos Santos Matos2, Sandra Rolim Ensslin3, Leonardo Ensslin4, Ademar Dutra5, Clarissa Carneiro Mussi6
l, 2, 3, 4 Federal University of Santa Catarina (UFSC).
Address: Roberto Sampaio Gonzaga, 274 - 88040-380. Florianopolis - SC, Brazil.
5 6 Southern University of Santa Catarina (UNISUL).
Address: Rua Trajano, 199 - 88130-100. Florianopolis - SC, Brazil.
1 Doctoring Candidate in Production Engineering;
[email protected]; ORCID: https://orcid.org/0000-0001-5738-3565
2 Doctor in Production Engineering; [email protected]; ORCID: https://orcid.org/0000-0001-6736-7227
3 Doctor in Production Engineering; [email protected]; ORCID: https://orcid.org/0000-0001-7420-8507
4 Ph.D. in Industrial and Systems Engineering; [email protected]; ORCID: https://orcid.org/0000-0001-7963-4679
5 Doctor in Production Engineering; [email protected]; ORCID: https://orcid.org/0000-0001-5289-9482
6 Doctor in Business Administration; [email protected]; ORCID: https://orcid.org/0000-0003-0367-4345
Abstract. This study aimed to understand what the literature has been approaching regarding public sector innovation and which measurement practices have been used, in addition to seeking research opportunities. The process was guided by the ProKnow-C instrument, a process of selection and critical analysis of the literature which allowed the selection of 33 articles. In general, it was found that: (i) the meaning of what innovation is has changed over the years; (ii) although there are attempts to evaluate these innovations, they are still incipient, especially in defining what is being considered as an innovation, which qualitative scale best represents what innovation is, how to transform this qualitative (ordinal) scale into a mathematical scale (cardinal); (iii) the evaluation has been promoted by the adoption of methods from the private sector, which are considered inappropriate for the public sector, since they make use of successes interpreted in organizations with divergent contexts. The results of the study make it possible to form, on the basis of institutional situational perception and needs, an instrument that meets the properties of measurement and determine the direction of managerial activity.
Keywords: public sector; innovation; evaluation.
For citation: Rodrigues, K. T., Matos, L. S., Ensslin, S. R., Ensslin, L., Dutra, A. and Mussi, C. C. (2023) 'Evaluation of public sector innovation: systematic review', Public Administration Issue, 5 (Special Issue I, electronic edition), pp. 165-188 (in English). DOI: 10.17323/1999-5431-2023-0-5-165-188.
JEL Classification: O31, H83. 1. Introduction
Innovation can be defined as an idea or practice perceived as new by an individual or an adoption unit (Choi and Chang, 2009). In this context, studies show that innovation is an important driver of potential competitiveness for countries and organizations for which the creation of competitive differentials becomes fundamental for economic development (Agolla and Van Lill, 2017; Moonen, 2017).
The design of the concept of innovation is not only given through invention, but also through differentiated combinations and the exploration of certain practices and attitudes. The globalized scenario is fertile for the implementation of novelties that generate both benefits and decision support in this increasingly complex environment (Arbix, 2010). An organization's ability to innovate is the result of a combination of external and internal factors that need to be managed to achieve the desired goals. The measurement of innovation and its management are critical issues for practitioners and researchers in this field and are fundamental to creating competitive differentials (Adams et al., 2006).
Due to globalization and increasing competition, different organizations, both private and public, strive to stimulate innovation to remain relevant in the market (Agolla and Van Lill, 2017; Moore and Hartley, 2010) by finding answers to various common demands. They also follow market trends and adhering to new technologies, which usually come from industry but are also applied in the organizational context, such as the Internet of Things (IoT) and Artificial Intelligence (AI) (de Sousa et al., 2018; Velsberg et al., 2020).
For the public sector, innovation has been increasingly cited as a driver of solutions in times of austerity and rationalization - common situations in recent years - in which, in addition to the competitive potential, innovation brings improvements in the performance of provided services while adding value and reducing costs (Agolla and Van Lill, 2017; Bello et al., 2018; Kinder, 2012).
Innovation in the public sector is seen as a creative response to the recession and also as a way to promote efficiency and effectiveness in the provision of public services and the promotion of accountability (Moore and Hartley, 2010; Schille-mans et al., 2013). Consequently, there is a direct demand from managers for innovation, mainly regarding positive risk management, so that risk acts as a helper to successful innovation rather than a barrier (Brown and Osborne, 2013). In addition, innovation is directly related to issues of human resources, organizational culture and leadership (Bernier and Hafsi, 2007; Carter et al., 2011; Grote, 2000; Kim and Yoon, 2015; Leontjeva and Trufanova, 2018; Luke et al., 2010 ).
According to Kattel et al. (2018), the growth of incentives and innovative practices in the public sector has also been accompanied by attempts to measure them in order to determine the results generated by the new actions (Vigoda-Gadot et al., 2008). Although it is possible to find studies that show the relationship between innovation and organizational performance (Fernandez and Wise, 2010; Salge and Vera, 2012; Shoham et al., 2012; Walker et al., 2010), there are few that attempt to comprehensively assess innovation in the public sector (Kattel et al., 2018). Due to the difficulty of measuring innovation and research gaps, the literature still shows flaws in its evaluation, using inappropriate indicators or even using the literature of the private sector as a basis for model replication for the public sector (Perrin, 2002; Potnis, 2010; Salge and Vera, 2009; Walker et al., 2002).
As innovation grows, the importance of innovation in the public sector also grows as do attempts to measure these innovations and their possible effects. These innovations are directly linked to the organization's performance, and their measurement still faces difficulties of a conceptual and technical nature. The measurement of innovation is considered to be in its infancy, but it can be inspired by the area of performance measurement (Kattel et al., 2018).
Therefore, this research aims to understand the approach in the literature to public sector innovation and the measurement practices that have been used, in addition to seeking research opportunities. To this end, the theme Evaluation of Innovation in the Public Sector is presented: (i) in a systematic way through the representation of its temporal evolution and literature mapping; (ii) by critical analysis of the metrics found in the selected scientific articles; and (iii) by the systemic analysis according to the adopted theoretical affiliation.
Therefore, it is necessary to select a representative fragment of the literature related to the theme, composing a Bibliographic Portfolio (BP), and, on this set of selected studies, make the proposed analyses. To enable the development of the work, researchers must use an intervention instrument supported by the Con-structivist paradigm. Thus, the Knowledge Development Process-Constructivist (ProKnow-C) instrument was selected. This instrument is already in use in qualitative studies like this as it is a structured and systematized process of selection, reflection, and critical analysis of the literature which allows the research to be guided toward achieving the defined objectives (Dutra et al., 2015; Ensslin et al., 2015; Staedele et al., 2019; Thiel et al., 2017; Valmorbida and Ensslin, 2017, Welter and Ensslin, 2022). With the development of the constituent stages of ProKnow-C, it will be possible to show the frontier of knowledge related to the theme and identify opportunities for future research.
2. Research
2.1 Process for selecting articles related to the Evaluation of Innovation in the Public Sector
The ProKnow-C was used to select articles related to the theme, which, according to Tasca et al. (2010) and Thiel et al. (2017), aims at generating researchers' knowledge on the subject under investigation due to its Constructivist bias.
This knowledge enables greater understanding of the topic which leads to a critical analysis of the literature and identification of possible research gaps.
ProKnow-C enables the researcher to carry out a structured and systematic review of the literature and select relevant scientific articles that represent the fragment of scientific publications on the desired theme (Matos et al., 2019; Staedele et al., 2019; Valmorbida and Ensslin, 2017). The instrument is operationalized through the following stages: (i) Selection of a Bibliographic Portfolio (BP) on the theme; (ii) Bibliometric Analysis of the selected BP; (iii) Literature Map development; (iv) Systemic Analysis of the BP articles; and (v) Research question and proven opportunities based on the knowledge built during the process (Welter and Ensslin, 2022). Figure 1 shows the stages of the ProKnow-C process.
Researcher knows the prominent agents ot literature fragment relating to the theme
Researcher knows the theoretical and methodological gaps and research opportunities ot literature fragment relating to the theme
Figure 1: ProKnow-C process stages
Source: Adapted from Welter and Ensslin (2022).
Generally, the first step, which is the selection of BP - starts with the identification of a set of articles that satisfy the boundaries established by the researchers. The articles then go through filtering procedures for duplicity and alignment with the subject. Finally, a cross-reference analysis is conducted, resulting in a Bibliographic Portfolio of relevant scientific articles representative of the studied literature fragment (Ensslin et al., 2015).
To start the process, researchers need to establish some delimitations and definitions for inputs to this process. These inputs concern the research axes. In this analysis, three axes were considered necessary: Performance Evaluation, Innovation and Public Sector. These terms and other keywords representing them in the academic literature formed the "search command". The delimitations refer to the
databases in which the search and the time slice were made. In this study, we decided not to make a time cut-off because the objective of this study was to find a representative fragment of the literature on the evaluation of innovations in the public sector. The searches in the Scopus and Web of Science databases were carried out on April 12, 2019 and refreshed on August 10, 2022. The selection of these databases was justified because they are broad and cover most areas of knowledge and they allow searching the title, abstract, and keywords simultaneously with Boolean commands.
The initial search of the databases resulted in a raw article database with 9,054 references. Filtering was then carried out, excluding repeated articles and publications in books and conferences. This left 6,195 articles which titles were read. It was found that studies on innovation in the public sector were mainly conducted from three perspectives: (i) evaluation of innovation in companies or institutions financed by the government; (ii) evaluation of innovation facilitators/drivers in the public sector; and (iii) evaluation of innovative practices and processes in the public sector. The selection of studies in this article followed the third perspective (876 articles), which is another delimitation established by the researchers. It is also important to highlight that for this research stage, a critical perspective on the part of the researchers was necessary regarding the identification of innovation, that is, to analyse the practices as innovations within the proposed context of public sector.
The scientific recognition of the 876 articles was searched on Google Scholar by checking the number of citations of each article. The articles were ranked in decreasing order by number of citations and it was found that 168 articles accounted for 80% of the total citations. These 168 articles with 49 or more citations thus formed the repository of non-repeated articles with a matching title and scholarly credit. The remaining 708 articles are non-repeated articles with a matching title, but their scientific recognition has not yet been confirmed. After this stage, the abstracts of 168 articles were read, of which 54 matched the research topic. Of the 54 works, three were not found in full in the databases. 51 were found and read in full to check compliance with the established delimitation, leaving 25 articles from which the Bibliographic Portfolio (BP) was compiled.
After that, in the second half of 2022, the BP was updated including 8 recent articles from 2019 to 2022. The complete process of selection, considering the updated actions in 2022, is summarized in Figure 2.
Through ProKnow-C and the Constructivist perspective used to conduct the work, it was possible to select 33 articles to compose the final BP of articles. This is the data for analysis in this study. It should also be noted that the articles from the selected BP are the basis for the theoretical foundation of the research, providing specific knowledge and concepts related to the theme. In the References section, these articles are listed in alphabetical order, numbered from 1 to 33 in square brackets "[ ]". Among these, the 28 articles included in the BP that were the subject of the systemic analysis are marked with the letter "A" after the numbering, and 5 articles that were not part of the systemic analysis are marked with the letter "B".
Research question
How has international literature approached the Evaluation of Innovation in the Public Sector? What is the future vision for the theme?
O
Scopus
Data Bases
Web of Science
Search command (keywords)
C (Performance AND (Manag* OR j-< Evaluat* OR Assess* OR Measur* I OR Indicator*) AND
„ f (Public Sector OR Public* OR jH "Public Administration" OR Government) , AND
H
(Inovat*)
Delimitations
Journal Articles; • Full access; Without time limit; • Written in english.
Until April, 2019
2019-2022
Raw Article Database Raw Article Database
9,054 References * 2,822 References
: Elimination of i
Database of non-repeated articles
6,195 articles
redundancies and references outside the boundaries
Deleting articles with misaligned titles
Non-repeated articles
2.083 articles
Database of articles with aligned titles 876 Articles Articles with aligned titles 116 Articles
............1............
representative or potential articles, with
Database of articles with aligned abstracts Articles with
representativeness and aligned representativeness and
abstracts aligned abstracts
54 Articles i 18 Articles
Selection of fully eligned articles
Bibliographic Portfolio
25 Articles
Added to Bibliographic Portfolio
6 Articles
Figure 2: BP selection process on evaluation of innovation in the public sector.
Source: Designed by the authors.
2.2 Data analysis and treatment procedure
Based on the selected BP, the Bibliometric and Systemic analysis stages were performed, in addition to the description of the temporal evolution of the theme and the development of a Literature Map. The purpose of this data treatment is to present the most relevant aspects for the scientific community interested in the topic.
Initially, the papers were categorized in electronic spreadsheets (Excel) to identify the possible basic and advanced variables (Thiel et al., 2017) to be addressed and discussed in the study in the Bibliometric Analysis stage. As basic variables for this research, we chose to investigate the countries where the studies were carried out, as well as the authors and networks of authors in the identified BP. The critical reading of the articles made it possible to identify key aspects that showed an evolution in the studies over the years. These aspects allowed the representation of the evolution of the studies over time, which is one of the advanced variables of this research.
To build the Literature Map, the innovations presented in the articles of the BP were grouped into three dimensions: (i) Management; (ii) Processes; and (iii) People. Thus, the type of innovation identified was used as a foundation for building the Literature Map in these three dimensions.
After performing the Bibliometric Analysis and building the Literature Map, the fourth phase of ProKnow-C, Systemic Analysis, was undertaken. Systemic Analysis was defined as the scientific process used, with a theoretical affiliation adopted by the researchers, to analyse a Bibliographic Portfolio of representative
articles on a particular research topic, seeking to evidence, according to the established worldview, the highlights and opportunities (shortages) of the identified knowledge (Ensslin et al., 2015).
For this work, the adopted theoretical affiliation, or established worldview, is closely linked to the Constructivist paradigm related to the theme of Performance Evaluation (PE) and based on studies such as those of Ensslin et al. (2010) and Ensslin et al. (2015). In these studies, PE is understood as a process that builds knowledge in the manager/decision-maker about the specific context it proposes to evaluate based on the manager's/decision-maker's own perception, through activities that ordinally and cardinally identify, organize, and measure to integrate and visualise the impact of actions and their management. Based on the adopted theoretical affiliation, six lenses are derived and will guide the Systemic Analysis of each of the BP articles, guiding the subsequent formulation of research opportunities (Ensslin et al., 2015; Ensslin et al., 2022). The six lenses and their concepts are presented in Table 1.
Table 1
Theoretical affiliation lenses of performance evaluation
N° LENS WHAT IS SOUGHT
1 Approach Harmonizes the model built (approach and data) with its application?
2 Singularity Recognizes the problem is unique? (Authors, context)
3 Identification process Uses the process to identify the objectives according to the perception of the decider?
4 Measurement The scales (descriptive nominal) used meet the Theory of Mensuration and its properties (measurability, operationality, homogeneity, ineligibility, and allows distinction of best and worst performances?)
5 Integration As for the determination of the constants of integration, how are the questions presented to the decider?
6 Management The knowledge generated allows the recognition of the current profile, its monitoring, and improvement?
Source: Ensslin et al. (2015, p.1000).
For each of the analysed lenses, it is possible to determine highlights regarding adherence to the adopted worldview and gaps not filled by the studies identified in the BP.
For the analysis of the measurement lens, we chose to perform an integrated analysis of two advanced variables (which are part of the bibliometric analysis). According to theoretical contributions from the Performance Evaluation area, the BP articles were examined according to the characteristics of their measurement instruments. In relation to the article by Melnyk et al. (2014), this study appropriated the concept of metric and its constituent elements and evaluated the existing metrics in BP studies. In relation to Van Camp and Braet's (2016) article, flaws in the metric level that the authors mention as the most recurrent were investigated.
Table 2 presents the concepts adopted based on the two studies to carry out the advanced bibliometric analysis of the "metric" variable.
Table 2
Advanced bibliometric analysis "Metrics" (measurement lens)
VARIABLES THEORETICAL AFFILIATION HOW WAS THE ANALYSIS PERFORMED?
Metrics Melnyk et al. (2014) Presents performance measures (yes; no)
Presents ordinal scale (yes; no; does not apply)
Presents cardinal scale (yes; no; does not apply)
Presents pattern of reference (yes; no; does not apply)
Presents numeric score and/or final evaluation (yes; no; does not apply)
Failures at the metric level Van Camp and Braet (2016) Lacks clear, unique, and transparent definition (yes; no)
Import of the metrics from a company (or norm) to another (yes; some; no)
Selected due to accessibility/availability of data collection (yes; some; no)
Unbalanced quantity between dimensions (yes; no; does not apply)
Dominant focus on financial metrics (yes; no; does not apply)
Unbalanced relationship between qualitative and quantitative metrics (yes; no; does not apply)
Difficulty to measure intangibles (yes; no; does not apply)
Incomplete set of the metrics (yes; no)
Risk of the metrics becoming targets (yes; no)
Lack of robust metrics (possibility of manipulation) (yes; no)
Lack of objective metrics (in which authors consider the primary sources as subjective data and secondary sources as objective.) (yes; no)
Uncertainty in the beginning of a project (yes; no; does not apply)
Use of deterministic metrics (yes; no; does not apply)
Source: Prepared by the authors.
It should be noted that the process of analysing the articles followed the member review protocol defined by Creswell (2014), where two of the authors are responsible for identifying the aspects raised and other authors are responsible for evaluating and validating the findings.
3. Discussion
3.1 Bibliometric analysis: Basic variables
Initially, as a BP basic variable, it was found that most studies had been done in Europe, especially in the United Kingdom. Among the 33 works that made up the BP, 15 had been done, partially or completely, in the United Kingdom.
In some of them, data was collected for researching questionnaires, while others ended up accessing some database of the sector itself [20A]. This shows that European countries, especially the UK, are increasingly concerned with innovation in the public sector and the evaluation of this process. There is a focus on new practices, especially those that bring potential savings of resources. Figure 3 illustrates in which countries the surveys were conducted and indicates, by size, where the highest concentrations were found.
None of the BP studies had been done specifically in Latin American countries, but one of the studies had been done in one of the UN member states, regarding process innovations, which the localization is not specified [18A]. No other article in the BP mentioned Brazil or other Latin American countries. This suggests that there may be a delay in studies and scientific publications on the topic of innovation in the public sector in these countries and that the topic needs to be researched in this context.
Figure 3: Countries where the studies were conducted.
Source: Developed by the authors based on Bibliographic Portfolio.
Also, the second basic variable was analysed related to the author's productivity in BP. Figure 4 follows the same pattern as the previous one, indicating through the sizes which authors appeared more often and also the collaboration networks, demonstrating in clusters the research groups responsible for BP studies.
As a result of the authors' analyses, it was evidenced that the literature on the theme does not have authors that stand out in several articles in the BP nor are
there networks of authors; rather, there is an authorship dispersion. As for highlights, Richard M. Walker was the author of two works [24A, 25A] with different co-authors, with eight years between publications. He is a professor of Public Management at the Department of Public Policy, University of Hong Kong, and has published several papers on the public sector and the relationship between innovation and organizational performance.
shoham, a. ruvio, a. vigoda gadot, e. schwabsky, n.
de melo.
bermejo, p.
de sousa. w. gomes, a. farias, r.
alahbabi, s.
gaur. s.
taylor, p.
howcroft. d.
r¡char£on. h. smifr, a. danf8rd a
sing£, s.
de vqps, h.
«Chilians, t •bek»S-V-
lem^r. v.
kat&u tönmotp. cepilovs, a.
damanpour, f.
vanhodfcierig. i. _ kea<jjs, k.
• Ü eirewie. m. devece. c. ^ ^ ,
walker, r. leontjp/a, o. beimrr. I.
kim. s. agolla,j. Iul«»>b'
yoon. g.chan& f 0Sb0rne'5' jeangs, e. cholj. brown, I.
• westereren u.
demircioglu, m.
cartgr, b. hartley. J- w
downe, j. bello, b.
r van bqyren, a.
jons^pn, k.
sa|gg, t.
duijjp, m.
matjjr. sandrews, r. vgf^a. giesfc. h.
edmypds, I. vtfcl- OVSSHfiO. p.
nadaban, m. glud^Jer, 5. tuttQn. g.
jengj, m.
lawton-^rnith, h. ton, |
kamg¿lng, r.
brujf. e.
owuiM, a.
tang. z.
eg^l». azanQgla.j.
favoflpu, c. queyfoi, y.
mau£el. c. ma*' P'cara^ns, d,
Figure 4: BP Authors.
Source: Developed by the authors based on Bibliographic Portfolio.
Torsten Oliver Salge and Antonio Vera have two mutual research studies in PB [19A, 20A]: one in 2009 and another one in 2012, both in the health sector. Salge is a professor and co-director of the Institute of Technology and Innovation Management and his most cited publications are related to the benefits of innovation in the public sector and also to hospital innovation. Antonio Vera is a professor of Organizational Management and Human Resources, and some of his most recognized work is on innovation in the public sector and the efficiency of hospital innovation. The two professors belong to different universities in Germany and they are partners in their main publications. They also have several works in the
field, which means that they are successful authors in the field of innovation in the management of public hospital organizations.
Finally, the network composed of Nitza Schwabsky, Aviv Shoham, Eran Vi-goda-Gadot and Ayalla Ruvio [7A, 17A] is a collaboration between the University of Haifa located in Israel and the State University of Michigan in the United States. The studies were published in 2008 and 2012. Among the authors, Eran Vigoda-Gadot, who is a professor of Administration and Public Management in Israel, is the author with the greatest interest in the public sector.
The analysis and research on the academic trajectory of the mentioned authors demonstrates that the researchers are active in this field, which justifies a larger number of studies in BP and can contribute significantly to the topic studied.
3.2 Bibliometric analysis: Advanced variable - Temporal Evolution
The representative literature fragment, selected by the operationalization of ProKnow-C, began in 2000 and ended in 2022. Figure 5 shows the temporal evolution of the theme based on 33 selected articles. It is important to highlight this because this is a qualitative research; thus, this evolution was built according to the authors' perceptions and interpretations of this study about the theme.
At the beginning of the time horizon, it was observed that the focus of the research work was on how to measure innovation based on the way it was evaluated in the private sector. The authors of the current study point out that the indicators used in the studies can be considered inappropriate to evaluate innovation in the public sector due to: (i) the differences between the private and public sectors; and (ii) the purpose of measuring intangible aspects now that will still occur [9B, 17A, 25A].
Beginning in 2002, it was found that the innovation treatment was related to new attitudes in governance and changes in decision making, besides to prob-lematize the scarcity of studies on this type of innovation [16B]. Also, the studies took a new look at entrepreneurship in the public sector, such as the entrepreneurial stance linked to creativity, flexibility, and willingness to adopt new ideas, which allows innovation to be evaluated through the employees' entrepreneur-ship [3A, 23A].
Figure 5. Temporal evolution.
Source: Developed by the authors based on Bibliographic Portfolio.
For the period from 2008 to 2013, the articles demonstrated that focus is on management beyond the implementation of innovations in the public sector. Examples include the implementation of e-governance [6A, 18A], activities that generate innovation and the implementation of new management practices [24A, 20A], the introduction of Lean practices [5A], and the discussion on risk management, which is a new view on the impact of risk on public innovation [4A].
Already from 2013 to 2019, there was still a focus on new management practices and process innovation; however, more attention was paid to identifying barriers, facilitators, and indicators to evaluate innovation in literature aimed at the public sector, reducing the use of practices based on the private sector as was evident in the early 2000s [1A, 7A, 10A, 11A]. Currently, the literature is not only focused on measuring innovation, but also seeks to identify other factors relevant to management and to conduct the evaluation in a way that it can serve as an instrument to support the management processes of organizations.
Finally, between 2019 and 2022, the focus of the papers was on improving employee engagement and the organization's ability to innovate, using new technologies and practices as a tool. Examples include the implementation of teleworking to engage employees more in their activities, which was a widespread practice from 2020 onwards due to the COVID-19 pandemic [26A]. Still, there is the adoption of Artificial Intelligence [27] and IoT [30A], where intelligence is created by the combination of knowledge management [28A] and people, technology and the innovative capacity of the organization [29A, 31, 32A, 33A].
3.3 Literature map
Aiming to present the topic to the scientific community synthetically and visually, the Literature Map, shown in Figure 6, was built based on the reading, analysis, and categorization of 33 articles contained in the BP in the dimensions of Management, Processes, and People. Among these dimensions, 4 papers fit in Management innovations, 8 in Process innovations, and 6 in People-related innovations. Also, 15 papers fit in the intersection between the three dimensions and are directly related to the measurement/assessment of innovation.
Thus, for the building of the Literature map, the dimensions, the aspects (in bold) were identified by the article code and the year of publication - and ramifications were found within these aspects. The ramifications were found in two themes : (i) measurement or evaluation of innovation; and (ii) electronic government.
In addition to the type of innovation, the map was developed according to what was evaluated within each dimension, even if it was not exactly the innovation. For example, in the People dimension, one of the aspects was Performance Evaluation [9B] since, in this study, the evaluation of people's performance as an innovation to evaluate the organization's performance was considered.
Performance Evaluation [1- 20171 9-2000] • Entrepreneurial Posture [3-2007] . Culture of Innovation [11-2015] • Teleworking [26-2019] • Knowledge Management [28-2019]
People
■ Governance [15-2017116-2010]
Management * Organizational Innovation [22-2012]
■ Accountability [21-2013]
......................Measurement/Evaluation
of Innovation
• Introduction of Lean techniques [5-2011]
Process
Innovation Generating Activities (C&T) [20-2012] Learning and Listening [12-2012]
Personal Innovation [8-2010] Innovation and Entrepreneurship in state enterprises [14-2010] Approaches to classification and measurement [17,25-20021 10-201817-2019] Innovations in governance [ 23-2008] Innovation management x Organizational performance [6-2009124-2010| 32,33-2022] Shared management [2-2018] Continuous improvements In the ability to innovate [29,31-2020] Risk comprehension [4-2013]
• Electronic Government Team Selection [13-2018]
Feasibility of implementation [19-2009] Evaluation [18-2010] Artificial Intelligence [27-2019] Use of loT 130-2020]
Figure 6: Literature map of studies on evaluation of innovation in the public sector.
Source: Prepared by the authors.
It is noteworthy that in this research as well as in the analysed articles, innovation was considered according to the context in which it was inserted. Therefore, Lean techniques were introduced as innovation in 2011 [5A], even though it was no longer a novelty in the manufacturing sector. Today, the techniques of lean production can also be found in health, civil construction, agriculture, and other sectors. The same can be mentioned about Artificial Interconnection applications [27] and the use of IoT [30A] that is widespread in the industrial segment, but is still perceived as a novelty in the public sector.
Even though innovation measurement and evaluation practices are directly linked to management innovations, they represent the intersection between the three dimensions under consideration and the central objective of this study, which is the reason they are located at the heart of the Figure 6. Some of these articles show how to perform innovation measurement and how to define indicators and metrics, while others not only make a proposition but also conduct the evaluation. Regarding approaches to classification and measurement, three articles [10A, 17A, 25A] reaffirmed the discussions on the measurement of innovations, previously signalled with temporal evolution.
The works of the first Portfolio phase [17A, 25A] discuss how this measurement occurred, proposing metrics to evaluate and seek a differentiated analysis and showing that averages or scores could end up disguising what is happening in an organization. For this reason, the use of performance-based indicators was not considered to be the most appropriate, as innovation is reactive and criticism of low scores could cause a disincentive to try new practices. However, in the last phase of the Portfolio [10A], the discussion on how to develop adequate indicators for the public sector remained. In this case, it was represented by a mapping of indicators used in the public sector in different countries.
The analysed BP comprises 22 years of literature on innovation evaluation in the public sector. However, during this period, it was found that there are still technical and conceptual questions to be answered: "What should be measured?" and "How should be measured?".
3.4 Systemic analysis
In this section, the analyses made in the BP articles are presented according to the six lenses shown in Table 1 and according to the theoretical affiliation adopted of the Performance Evaluation. It is important to emphasize that, for the analysis, only 28 articles from BP were used, as 5 works had no practical application, but rather had theoretical value. Except for the Measurement lens, which included advanced bibliometric analysis related to metrics, the 33 BP articles from were used, as they all discussed this element of measurement.
3.4.1 Lens 1 - Approach
The first lens to be analysed concerns the approaches used for the development of BP studies. For this, Roy (1993) and Dias and Tsoukias (2004) were used as references. They divided the approaches into Normativists, Descriptivists (both treated as realistic by Roy), Prescriptivists, and Constructivists. The harmony between the development/selection of the evaluation instrument used in the study was analysed with the proposition for which it is intended; that is, if it was developed for an organization's specific purposes or if it was generically performed for generalized applications.
Most of the studies, 22 of them, used a realistic approach (Roy, 1993) and used the literature in the private sector to replicate methods for the public sector. Only 6 of the studies used the Prescriptive approach, in which researchers set the evaluation criteria based on other studies. In terms of harmony, it was found that 19 of them, had an approach compatible with the purposes for which they were intended, particularly generic ones. However, methods with approaches that were incompatible with the purposes for which they were designed; that is, there was no harmony between the approach and the application.
3.4.2 Lens 2 - Singularity
The analysis with the Singularity lens showed a worrying aspect of the content and the approach of the studies to the development of indicators to measure innovation in the public sector. Although the studies considered that the context in which they were applied referred to the public sector, none of them recognized the managers and the institution in an ad hoc manner. In other words, the indicators, models, and tools used to evaluate a given public institution did not originate from institutional strategies and thus did not recognize the specificity of this public context, nor the demands of the manager(s), server(s), and society. Such an absence makes subsequent effective management impossible.
It was then noted that the instruments developed in these studies were not concerned with identifying information to administer the context, but only to develop the means to describe situations of innovation in the public sector. This proves the existing research gap related to the consideration of the uniqueness of the context and the actors involved in management of innovation for the public sector.
3.4.3 Lens 3 - Process for identifying objectives
For the analysis of the third lens, two questions were defined that guided the identification of the aspects and objectives that gave rise to the indicators (and
metrics) of the instruments that were used to evaluate innovation in the public sector, based on the managers' manifestation: i) How do the manager's values and preferences interfere with the identification of the aspects and objectives that form the metrics and indicators of the evaluation instrument? How were these aspects/ objectives identified? and ii) How does the process of identifying aspects and objectives deal with the manager's limits of knowledge, and its possible extension, for the development of metrics and indicators?
As a result, it is evident that the analysed studies are not concerned with expanding managers' knowledge during the process of identifying objectives and preferences in the addressed context.
The analysis of this third lens highlights the need to consider that the manager is a fundamental actor in the process of identifying the metrics and indicators used for evaluation, and it must be recognized that managers' knowledge expands and consolidates with their values and preferences. Among the BP articles, only the studies by Agolla and Van Lill (2017) were concerned with considering the manager's understanding to identify the factors required for innovation evaluation in the public sector. However, this was done in a limited way, without considering the limitations and possible expansion of knowledge during the process of building metrics and indicators.
The result of this lens analysis points to the opportunity for research that builds indicators that represent the institutional strategy from the perspective of public managers who use this instrument, configuring it as legitimate.
3.4.4 Lens 4 - Measurement
The analysis of the Measurement lens seeks to verify compliance with metrics, indicators, and the properties of Measurement Theory. Thus, a metric must necessarily satisfy three basic properties, : (i) to be "measurable", (ii) to be "operational"; and (iii) to be "understandable", according to Keeney (1992, p. 112) and Ensslin et al. (2001, p. 160-161). The properties are described in Table 3.
Table 3
Measurement theory properties for metrics
PROPERTY DESCRIPTION
Measurable The metric is measurable when it is formed by a number "x" of impact levels that detail the possibilities of occurrence in such a way as to eliminate any doubt as to what is being analysed at each of the "x" levels and as to the definition of the aspect/objective that is being analysed/evaluated using this scale.
Operational The metric is operational when it is composed of impact levels in which there is one and only one level of impact that represents/describes a possibility of the action/alternative occurring. That is, for the action/alternative analysed there is only one level that represents the real consequence of this action/alternative.
Understandable The metric is understandable/intelligible when for the various institution managers, the consequence of an action/alternative is understood by all in the same way, thus reaching the same measurement.
Source: Based on notions of Keeney (1992) and Ensslin et al. (2001).
The findings point to deficiencies related to the disregard of measurement properties in most of the analysed articles. Only 15% of the BP studies built scales that met the described properties, demonstrating the lack of commitment to the properties of measurement mainly concerns the lack of clarity in the description of the scales. This lack difficult to meet the properties of measurability and intelligibility, as well as in the use of qualitative scales for the quantification and measurement of objectives. Mathematical operations are found inappropriate to the situation, such as the mean and standard deviation in criteria evaluated according to Likert Scales.
It is emphasized that it is necessary to consider the attention to measurement properties when developing scales for innovation evaluation in the public sector.
Melnyk et al. (2014) defined metrics and the essential elements of their composition. When categorising the articles, it was observed that, in the case of evaluation of innovation in the public sector, all works had metrics. However, the selected studies dealt with metrics of different shapes and characteristics. Therefore, based on Melnyk et al.'s definition (2014), the metrics of the studies were analysed to identify the presence of the proposed elements.
The diagnosis of metrics could be performed in 31 articles of the BP, 2 papers do not use performance metrics, considering that some of them, despite being theoretical, had metrics in their content. Among them, only 1 study had an ordinal scale, a cardinal scale, and a reference standard, but did not have a final evaluation [3A]. A very critical point in this diagnosis was that the vast majority (67%) of the works did not have a cardinal scale, that is, the mathematical scales (representative of transformation of the qualitative ordinal scale into a mathematical scale). The diagnosis also demonstrates the scarcity of works that have a reference standard, that is, the scale level that is considered good (goal) and the scale level that is considered bad (Ensslin et al., 2001; Ensslin et al., 2010), referred to as the performance standard by Melnyk et al. (2014) and Van Camp and Braet (2016).
The diagnosis shows that there is no standardization in how metrics are defined and developed to evaluate and measure innovation. There is discussion about the difficulty of defining a scale for innovation [23A], which is one of the reasons why, to facilitate quantification, metrics related to scientific production and intellectual property were used [1A, 7A, 15A, 19A, 20A]. There is also a trend towards the use of descriptive metrics (qualitative/ordinal), at least in part, such as description of the innovation, type, origin, and stakeholders, among others [3A, 4A, 6A, 12A, 13A, 18A, 25A].
One of the ways to justify the use of this type of metric is the difficulty of measuring innovation because it is intangible. This difficulty is one of the 13 failures pointed out by Van Camp and Braet (2016). Figure 7 shows how the analysis of failures at the metric level of BP works, in which only 1 article did not have any of the failures described by Van Camp and Braet (2016) and met all the elements proposed by Melnyk et al. (2014) [3A]. The difficulty in measuring intangibles is the most frequent failure in BP, with 23 cases. This difficulty in measuring innovation is expected to occur due to its very nature; thus, the measure
that will represent this metric must be based on criteria that currently demonstrate or enable the development of these intangibles, rather than criteria that are believed to occur in the future [17A].
Failures at the metric level
Lack of clear, unique and transparent definition Import of the metrics from a company {or norm) to another Selected due to accessibility/availability of data collection Unbalanced quantity between dimensions Dominant focus on financial metrics Unbalanced relationship between qualitative and quantitative metrics Difficulty to measure intangibles Incomplete set of metrics Risk of the metrics become targets Lack of robust metrics Lack of objective metrics Uncertainty in the beggining of a projetct Deviation from deterministic metrics
0 S 10 15 20 25
Figure 7 Metric failures.
Source: Prepared by the authors.
The lack of objective metrics (Van Camp and Braet, 2016) and nomenclature should also be considered in this analysis, which occurred in some situations due to the inappropriate use of the Likert scale, as there was no definition and accurate explanation of what each point meant. Such a deficiency is also evidence of noncompliance with the operational and comprehensible properties. There is also a lack of explanation for the attribution/calculation of the attractiveness difference between the points, that is, the absence of a cardinal scale that later prevents aggregation of punctual evaluations. This situation occurred in the studies [9B, 11A, 13A, 20A, 26A, 28A, 32A, 33A]. It can be seen that in the relationship between the two discussed failures, one is a consequence of the other. The difficulty of measuring intangibles leads to failure in defining objective metrics and in using appropriate scales.
3.4.5 Lens 5 - Integration
According to the adopted theoretical affiliation, evaluation indicators and scales (metrics) must be integrated to allow a holistic and systemic view of the aspects considered necessary within a model.
In this regard, it was verified for the BP articles how the aspects are integrated in the worksand if they allow a complete view of an evaluation system or if they are just isolated variables.
As a result, only 4 of the BP articles manifest the connection between the evaluation variables, allowing the visualization of a global and systemic result on the managed context. The low return referring to the Integration lens characterizes an opportunity to contribute to the research field, as the holistic view of a management process is fundamental to support decisions and understand a variable reflection in a global context.
3.4.6 Lens 6 - Management
Finally, the Management lens aims to verify whether the instruments presented in the BP studies allow the identification of the current diagnosis (status quo) of the analysed situation relating to the established indicators and whether there are contributions with actions that aim at developing improvements in the management processes.
Despite presenting a diagnosis of the situation, studies often use scales that do not allow for real knowledge of the context and the meaning of the results obtained. When the studies present the results numerically, they do not present reference levels that allow identifying better possibilities. They limit themselves to pointing numerical results without a clear scale. For this reason, it is also not possible to generate improvement actions to achieve better results and promote management support.
As for the development of improvement actions for the management of innovation in the public sector, Edmunds et al. (2019) recommend good practices according to previous findings in the literature. More recent studies [26A, 28A, 29A, 32A, 33A] conclude the discussions with aspects that should be improved, but do not indicate the sequence of processes to be applied. We can also mention the study by Potnis (2010) that recommends actions to improve the measurement methods, but does not address measures to support business management and thus was not considered as improvement actions for management.
The results show that there is a gap between instruments and practices used to present relevant information to management in the provision of subsidies that support and promote the knowledge of managers.
4. Conclusions
This study aimed to understand the approach of the literature regarding public sector innovation and the measurement practices used, to seeking new research opportunities. The use of the ProKnow-C intervention instrument enabled the selection of a Bibliographic Portfolio of 33 articles from the Scopus and Web of Science databases. However, five of these studies were excluded from the sample as they did not meet the criteria for the systemic analysis. Thus, the systemic analysis was conducted on a BP of 28 articles.
In general, it was found that the meaning of innovation has changed over the years. Although there have been attempts to evaluate these innovations, these attempts are incipient, especially in defining what is considered as an innovation, which qualitative scale best represents the innovation in question, and how to transform this qualitative (ordinal) scale into a mathematical scale (car-
dinal) so that a final numerical evaluation can be made. This research extends our knowledge of efforts made by many authors to assess innovation in the public sector. However, in terms of generating information for management and alignment to theoretical contributions from the Performance Evaluation area, they are still incipient.
The Systemic Analysis through the six lenses derived from the theoretical affiliation of the authors of the present study enabled the identification of gaps that allowed the identification of future directions to fill them.
Table 4 presents the Performance Evaluation lenses and their respective gaps, as well as research opportunities.
Table 4
Summary of gaps and research opportunities
LENS GAPS AND FUTURE DIRECTIONS
Approach The development of custom templates is a differential for this area of knowledge
Singularity The recognition of specific problems is identified as a great opportunity for developments in the research area. In none of the studies was there a consideration of specific actors and contexts, which need to be taken into account to define a singular evaluation system
Process of identifying objectives Due to the non-consideration of environments and singular actors for innovation management in the identified works, the process of identifying measures is done by the authors of the analysed articles. The objectives of previous work are replicated, which may not be compatible with specific situations. It becomes essential to identify objectives closely connected to the interests and particularities of a manager responsible for innovation in a public sector organization
Measurement Most of the papers disregard the properties expected for the development of quantitative indicators that allow the identification of inferior and superior performances and that allow the identification of opportunities to improve performance. Part of the work uses qualitative measurement scales and mistakenly operates mathematical operations to identify the mean and standard deviation to build comparative metrics between different organizations
Integration Rarely are studies pointed to the consideration of integrated indicators within a system. According to the adopted theoretical affiliation, it is expected that an evaluation model will take into account different points of view that represent the organizational vision for innovation management, integrating them to promote a holistic view and present the set to support information to managers of the organization and other stakeholders
Management Although most studies present a diagnosis of their metrics, the information presented is of little use to discover deficiencies, potentialities, and to recognize actions to improve the performance of indicators. The achievement degrees of indicators are presented, but reference levels are not developed to identify performance below, within, or above what is expected for the organization. According to the literature on Performance Evaluation, it is still expected that the information promoted by an evaluation system or model will allow the achievement of continuous improvement in the evaluated management context. In this sense, the promotion of improvement actions is an opportunity related to the literature on innovation in the public sector
Source: Prepared by the authors.
Based on the ProKnow-C instrument, it can be concluded that the literature on the evaluation of innovations in the public sector has been driven by the adoption of methods adapted from the private sector. However, these practices are often shown to be inappropriate, as they draw on successes interpreted in other organizations with divergent contexts which may not deliver the expected results. In this sense, we suggest the development of an instrument that is based on the unique perceptions and needs of public managers and their specific institutions, meets the measurement properties, integrates the evaluation result of each indicator holistically, and promotes management support.
REFERENCES
1. Adams, R., Bessant, J. and Phelps, R. (2006) 'Innovation management measurement: A review', International Journal of Management Reviews, 8(1), pp. 21-47. Available at: https://doi.org/ 10.1111/j.1468-2370.2006.00119.x (accessed 6 October 2021).
2. Agolla, J.E. and Van Lill, J.B. (2017) 'Insights into Kenya's public sector innovation: The case of managers, International Journal of Innovation Science, 9(3), pp. 225-243. Available at: https:// doi.org/10.1108/IJIS-11-2016-0049 (accessed 6 October 2021). [1A].
3. Al Ahbabi, S.A., Singh, S.K., Balasubramanian, S. and Gaur, S.S. (2019) 'Employee perception of impact of knowledge management processes on public sector performance', Journal of knowledge management, 23(2), pp. 351-373. Available at: https://doi.org/10.1108/JKM-08-2017-0348 (accessed 25 August 2022) [28A].
4. Arbix, G. (2010)' Estratégias de inovaçâo para o desenvolvimento', Tempo Social, 22(2), pp. 167-185. Available at: https://doi.org/10.1590/S0103-20702010000200009 (accessed 6 October 2021).
5. Azamela, J.C., Tang, Z., Owusu, A., Egala, S.B. and Bruce, E. (2022) 'The impact of institutional creativity and innovation capability on innovation performance of public sector organizations in Ghana, Sustainability, 14(3), p. 1378. Available at: https://doi.org/10.3390/ su14031378 (accessed 25 August 2022). [33A].
6. Bello, B., Downe, J., Andrews, R. and Martin, S. (2018) 'Does austerity drive public service innovation? Evidence from shared senior management teams in local government', Public Money and Management, 38(2), pp. 131-138. Available at: https://doi.org/10.1080/09540962 .2018.1407161 (accessed 6 October 2021). [2B].
7. Bernier, L. and Hafsi, T. (2007) 'The changing nature of public entrepreneurship', Public Administration Review, 67(3), pp. 488-503. Available at: https://doi.org/10.1111/j.1540-6210.2007.00731.x (accessed 6 October 2021). [3A]
8. Brown, L. and Osborne, S.P. (2013) 'Risk and innovation: Towards a framework for risk governance in public services', Public Management Review, 15(2), pp. 186-208. Available at: https://doi.org/10.1080/14719037.2012.707681 (accessed 6 October 2021). [4A].
9. Carter, B., Danford, A., Howcroft, D., Richardson, H., Smith, A. and Taylor, P. (2011) '"All they lack is a chain": Lean and the new performance management in the British civil service', New Technology, Work and Employment, 26(2), pp. 83-97. Available at: https://doi. org/10.1111/j.1468-005X.2011.00261.x (accessed 6 October 2021). [5A].
10. Choi, J.N. and Chang, J.Y. (2009)' Innovation implementation in the public sector: An integration of institutional and collective dynamics', Journal of Applied Psychology, 94(1), p. 245. Available at: https://doi.org/10.1037/a0012994 (accessed 6 October 2021). [6A].
11. Creswell, J.W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.). SAGE Publications, Inc.
12. De Sousa, W.G., De Melo, E. R.P., Bermejo, P.H.D.S., Farias, R.A.S. and Gomes, A.O. (2019) 'How and where is artificial intelligence in the public sector going? A literature review and research agenda', Government Information Quarterly, 36(4), 101392. Available at: https://doi. org/10.1016/j.giq.2019.07.004 (accessed 25 August 2022). [27B].
13. De Vries, H., Tummers, L. and Bekkers, V. (2019) 'The benefits of teleworking in the public sector: Reality or rhetoric?', Review of Public Personnel Administration, 39(4), pp. 570-593. Available at: https://doi.org/10.1177/0734371X18760124 (accessed 25 August 2022). [26A].
14. Dias, L. C., and Tsoukiàs, A. (2004) 'On the constructive and other approaches in decision aiding', in: C.H. Antunes, J. Figueira and J. Climaco (Eds.) Aide multicritére à la décision: Multiple criteria decision aiding. CCDRC/INESCC/FEUC, pp. 13-28.
15. Dutra, A., Ripoll-Feliu, V.M., Fillol, A.G., Ensslin, S.R. and Ensslin, L. (2015) 'The construction of knowledge from the scientific literature about the theme seaport performance evaluation', International Journal of Productivity and Performance Management, 64(2), pp. 243-269. Available at: https://doi.org/10.1108/IJPPM-01-2014-0015 (accessed 6 October 2021)
16. Edmunds, L.D., Gluderer, S., Ovseiko, P.V., Kamerling, R., Ton, J., Vis, L., ... Rab, M. (2019) 'New indicators and indexes for benchmarking university-industry-government innovation in medical and life science clusters. Results from the European FP7 Regions of Knowledge HealthTIES project, Health Research Policy and Systems, 17(1), p. 10. Available at: https://doi. org/ 10.1186/s12961-019-0414-5 (accessed 6 October 2021). [7A].
17. Ensslin, L., Gonçalves, A., Ensslin, S.R. and Dutra, A. (2022) 'Bibliometric and systemic review of the state of the art of occupational risk management in the construction industry', International Journal of Occupational Safety and Ergonomics. Available at: https://doi.org/10. 1080/10803548.2022.2111893 (accessed 14 September 2022).
18. Ensslin, L., Giffhorn, E., Ensslin, S.R., Petri, S.M. and Vianna, W.B. (2010) 'Avaliaçâo do De-sempenho de empresas terceirizadas com o uso da metodologia Multicritério de Apoio à Decisâo-Construtivista', Pesquisa Operational, 30(1), pp. 125-152. Available at: https://doi. org/10.1590/S0101-74382010000100007 (accessed 6 October 2021).
19. Ensslin, L., Montibeller Neto, G. and Noronha, S.M. (2001) Apoio à Decisâo: Metodologia para Estruturaçâo de Problemas e Avaliaçâo Multicritério de Alternativas. Editora Insular.
20. Ensslin, S.R., Ensslin, L., Matos, L.D.S., Dutra, A. and Ripoll-Feliu, V.M. (2015) 'Research opportunities in performance measurement in public utilities regulation', International Journal of Productivity and Performance Management, 64(7), pp. 994-1017. Available at: https://doi. org/10.1108/IJPPM-05-2014-0067 (accessed 6 October 2021).
21. Fernandez, S. and Wise, L.R. (2010) 'An exploration of why public organizations 'Ingest' innovations', Public Administration, 88(4), pp. 979-998. Available at: https://doi.org/10.1111/ j.1467-9299.2010.01857.x (accessed 6 October 2021). [8A].
22. Gieske, H., Duijn, M. and Van Buuren, A. (2020) 'Ambidextrous practices in public service organizations: Innovation and optimization tensions in Dutch water authorities', Public Management Review, 22(3), pp. 341-363. Available at: https://doi.org/10.1080/14719037.2019.15 88354 (accessed 25 August 2022). [29A].
23. Grote, D. (2000) 'Public sector organizations: Today's innovative leaders in performance management', Public Personnel Management, 29(1), pp. 1-20. Available at: https://doi.org/ 10.1177/009102600002900101 (Accessed 6 October 2021). [9B].
24. Kattel, R., Cepilovs, A., Lember, V. and Tónurist, P. (2018) 'Indicators for public sector innovations: Theoretical frameworks and practical applications', Halduskultuur (Administrative Culture), 19(1), pp. 77-104. Available at: http://dx.doi.org/10.32994/ac.v19i1.208 (accessed 6 October 2021). [10A].
25. Keeney, R.L. (1992) Value-Focused Thinking: A Path to Creative Decision making. Harvard University Press.
26. Kim, S. and Yoon, G. (2015) 'An innovation-driven culture in local government: do senior manager's transformational leadership and the climate for creativity matter?', Public Personnel Management, 44(2), pp. 147-168. Available at: https://doi.org/10.1177/0091026014568896 (accessed 6 October 2021). [11A].
27. Kinder, T. (2012) 'Learning, innovating and performance in post-new public management of locally delivered public services', Public Management Review, 14(3), pp. 403-428. Available at: https://doi.org/10.1080/14719037.2011.637408 (accessed 6 October 2021). [12A].
28. Leontjeva, O. and Trufanova, V. (2018) 'Lean Team Members Selection for Public Administration Organizations', Public Administration Issues, (6), pp. 45-64. Available at: 10.17323/19995431-2018-0-6-45-64 (accessed 6 October 2021). [13A].
29. Luke, B., Verreynne, M.L. and Kearins, K. (2010) 'Innovative and entrepreneurial activity in the public sector: The changing face of public sector institutions', Innovation, 12(2), pp. 138-153. Available at: https://doi.org/10.5172/impp.12.2.138 (accessed 6 October 2021). [14A].
30. Matos, L.D.S., Ensslin, S.R. and Ensslin, L. (2019) 'A Review on the Performance Measurement Systems Life Cycle', Lex Localis - Journal of Local Self-Government, 17(4), pp. 939-959. Available at: https://doi.org/10.4335/17.4.939-959(2019) (accessed 6 October 2021).
31. Melnyk, S.A., Bititci, U., Platts, K., Tobias, J. and Andersen, B. (2014) 'Is performance measurement and management fit for the future?', Management Accounting Research, 25(2), pp. 173-186. Available at: https://doi.org/ 10.1016/j.mar.2013.07.007 (accessed 6 October 2021).
32. Moonen, P. (2017) 'The governance of innovation from a European perspective, social articulation and transmission of knowledge', Journal of Organizational Change Management, 30(2), pp. 243-262. Available at: https://doi.org/10.1108/JOCM-01-2017-0012 (accessed 6 October 2021). [15A].
33. Moore, M. and Hartley, J. (2010) 'Innovations in governance in The New Public Governance', in: S.P. Osborne (Ed.) Emerging perspectives on the theory and practice of public government. NY: Routledge. [16B].
34. Perrin, B. (2002) 'How to - and how not to - evaluate innovation', Evaluation, 8(1), pp. 13-28. Available at: https://doi.org/10.1177/1358902002008001514 (accessed 6 October 2021). [17A].
35. Potnis, D.D. (2010) 'Measuring e-Governance as an innovation in the public sector', Government Information Quarterly, 27(1), pp. 41-48. Available at: https://doi.org/10.1016Aj. giq.2009.08.002 (accessed 6 October 2021). [18A].
36. Queyroi, Y., Carassus, D., Maurel, C., Favoreu, C. and Marin, P. (2022) 'Local public innovation: an analysis of its perceived impacts on public performance', International Review of Administrative Sciences, 88(2), pp. 493-510. Available at: https://doi.org/10.1177/0020852320963214 (accessed 25 August 2022). [32A].
37. Roy, B. (1993) 'Decision science or decision-aid science?', European Journal of Operational Research, 66(2), pp. 184-203. Available at: https://doi.org/10.1016/0377-2217(93)90312-B (accessed 6 October 2021).
38. Salge, T.O., and Vera, A. (2009) 'Hospital innovativeness and organizational performance: Evidence from English public acute care', Health Care Management Review, 34(1), 54-67. Available at: https://doi.org/10.1097/01.HMR.0000342978.84307.80 (accessed 6 October 2021). [19A].
39. Salge, T.O. and Vera, A. (2012) 'Benefiting from public sector innovation: The moderating role of customer and learning orientation, Public Administration Review, 72(4), pp. 550-559. Available at: https://doi.org/10.1111/j.1540-6210.2012.02529.x (accessed 6 October 2021). [20A].
40. Schillemans, T., Van Twist, M. and Vanhommerig, I. (2013) 'Innovations in accountability: Learning through interactive, dynamic, and citizen-initiated forms of accountability', Public Performance and Management Review, 36(3), pp. 407-435. Available at: https://doi.org/ 10.2753/PMR1530-9576360302 (accessed 6 October 2021). [21A].
41. Shoham, A., Vigoda-Gadot, E., Ruvio, A. and Schwabsky, N. (2012) 'Testing an organizational innovativeness integrative model across cultures', Journal of Engineering and Technology Management, 29(2), pp. 226-240. Available at: https://doi.org/10.1016/j.jengtecman. 2012.01.002 (accessed 6 October 2021). [22A].
42. Staedele, A.E., Ensslin, S.R. and Forcellini, F.A. (2019) 'Knowledge building about performance evaluation in lean production', Journal of Manufacturing Technology Management, 30(5), pp. 798-820. Available at: https://doi.org/10.1108/JMTM-12-2017-0277 (accessed 6 October 2021).
43. Tasca, J.E., Ensslin, L., Ensslin, S.R., and Alves, M.B.M. (2010) An approach for selecting a theoretical framework for the evaluation of training programs', Journal of European Industrial Training, 34(7), pp. 631-655. Available at: https://doi.org/10.1108/03090591011070761 (accessed 6 October 2021).
44. Thiel, G.G., Ensslin, S.R. and Ensslin, L. (2017) 'Street lighting management and performance evaluation: Opportunities and challenges', Lex Localis - Journal of Local Self-Government, 15(2), pp. 303-328. Available at: https://doi.org/10.4335/15.2.303-328(2017) (accessed 6 October 2021).
45. Valmorbida, S.M.I. and Ensslin, S.R. (2017) 'Performance evaluation of university rankings: Literature review and guidelines for future research' International Journal of Business Innovation Research, 14(4), pp. 479-501. Available at: https://doi.org/10.1504/IJBIR.2017.087844 (accessed 6 October 2021).
46. Van Camp, J., and Braet, J. (2016) 'Taxonomizing performance measurement systems' failures', International Journal of Productivity and Performance Management, 65(5), pp. 672-693. Available at: https://doi.org/10.1108/IJPPM-03-2015-0054 (accessed 6 October 2021).
47. Van Der Wal, Z. and Demircioglu, M.A. (2020) 'Public sector innovation in the Asia-pacific trends, challenges, and opportunities, Australian Journal of Public Administration, 79(3), pp. 271-278. Available at: https://doi.org/10.1111/1467-8500.12435 (accessed 25 August 2022). [31B].
48. Velsberg, O., Westergren, U.H. and Jonsson, K. (2020) 'Exploring smartness in public sector innovation-creating smart public services with the Internet of Things', European Journal of Information Systems, 29(4), pp. 350-368. Available at: https://doi.org/10.1080/096008 5X.2020.1761272 (accessed 25 August 2022). [30A].
49. Vigoda-Gadot, E.R.A.N., Shoham, A., Schwabsky, N. and Ruvio, A. (2008) 'Public sector innovation for Europe: A multinational eight-country exploration of citizens' perspectives', Public Administration, 86(2), pp. 307-329. Available at: https://doi.org/10.1111/j.1467-9299. 2008.00731.x (accessed 6 October 2021). [23A].
50. Walker, R.M., Damanpour, F. and Devece, C.A. (2010) 'Management innovation and organizational performance: The mediating effect of performance management', Journal of Public Administration Research and Theory, 21(2), pp. 367-386. Available at: https://doi.org/10.1093/ jopart/muq043 (accessed 6 October 2021). [24A].
51. Walker, R.M., Jeanes, E. and Rowlands, R. (2002) 'Measuring innovation-applying the literature-based innovation output indicator to public services', Public Administration, 80(1), pp. 201-214. Available at: https://doi.org/10.1111/1467-9299.00300 (Accessed 6 October 2021) (accessed 12 August 2022). [25A].
52. Welter, L.M. and Ensslin, S.R. (2022) 'How do the unintended consequences of performance evaluation systems manifest themselves?', Journal of Accounting and Organizational Change, 18(4), pp. 509-528. Available at: https://doi.org/10.1108/JAOC-07-2020-0087 (accessed 14 August 2022).
The article was submitted: 07.10.2021; approved after reviewing: 23.09.2022; accepted for publication: 25.03.2023.
