CC BY-NC-ND
69
Literature Review
DOI: 10.17323/1999-5431-2022-0-5-152-181
EVALUATING DIGITAL GOVERNMENT PERFORMANCE USING SURVEYS: INTERNATIONAL LITERATURE REVIEW AND PROSPECTS FOR THE FUTURE RESEARCH
Elena I. Dobrolyubova
Ph.D. (in Economics), Deputy director of the Center for Technology Governance, Institute of Applied Economic Research, Russian Presidential Academy of National Economy and Public Administration (RANEPA), 82, Vernadskogo Av., 119571 Moscow, Russia; [email protected]; ORCID: 0000-0002-9405-4936
Abstract. The use of digital technologies in public administration has become a major trend in both developed and developing countries. Not surprisingly, the research interest to evaluating digital government performance has also increased. Since international indices and national statistics is more focused on evaluating the supply side of government digitalisation, more and more researchers are using survey methods to assess the progress determinants and barriers to digital government success from the demand side. So far, the review of such empirical studies has been quite limited. This paper aims to fill this gap and provides a comprehensive international literature review of using surveys for evaluating digital government performance published in 2011-2021. For the purpose of the study, an analytical framework including four levels of digital government performance (inputs and outputs characterising the government digitalisation process and outcomes and impacts representing public values) as well as cost and risk constructs were proposed and used.
The analysis of 216 empirical papers examining digital government performance in over 50 countries from 5 continents demonstrates that survey-based methods are widely used in both developed and developing economies, with varying levels of digital maturity. Several cross-country studies have been identified but most
of the papers address digitalisation of public authorities at the national or local level. The core focus of the research is related to the adoption of digital public services and the extent of digital participation among citizens (and, to a much lesser extent, by businesses), while outcomes and societal impacts receive less attention regardless the level of e-government development of the relevant country. Noteworthy, there is growing interest in the issue of digital government risks. It is argued that more efforts should be made to evaluate the outcomes and impacts (public value) of digital government. Prospective research areas also relate to using survey methods to evaluate the performance of digital government in regulatory and enforcement areas, exploring variation of trust in government and trust in technology, measuring perceptions of government digitalisation risks, researching the reasons for avoiding digital interactions with the government on the part of citizens and businesses, and several others.
Keywords: e-government adoption, effectiveness, public value, service satisfaction, sociological survey, outcomes, performance framework, technology acceptance model.
For citation: Dobrolyubova, E. I. (2022) 'Evaluating digital government performance using surveys: International literature review and prospects for the future research, Public Administration Issues, 5 (Special Issue II, electronic edition), pp. 152-181 (in English). DOI: 10.17323/1999-5431-2022-0-5-152-181.
Acknowledgement: The article is written based on the scientific program of the RANEPA state order.
Introduction
Digitalisation has become a major trend in public administrations around the globe, and the trend has been further accelerated by the COVID-19 pandemic which made e-government a 'necessary element of communication, leadership and collaboration between policy makers and society' (United Nations, 2020, p. 215). International organisations, national governments, and academia have expressed high expectations for the role of digital technology in improving governance quality and bringing substantial dividends to citizens and businesses (World Bank, 2016). Some scholars suggest that digitalisation creates new governance paradigms such as ICT-enabled transformational government (Heidelberg, 2009) or digital era governance (Margetts and Dun-leavy, 2013).
Others claim that while digitalisation has brought some improvements to public administrations, overall, the impact of technology has been overestimated and the initial expectations have not been met in many cases (Bannister and Connolly, 2020).
Some recent research demonstrates a correlation between the government digitalisation and the governance quality (Durkiewicz and Janowski, 2018), however, the causality between the two has not been confirmed for most gov-
ernance indicators (Dobrolyubova et al., 2019). Noteworthy, most international indicators, such as E-government Development Index (EGDI) and the UN E-participation index (United Nations, 2020) or the Digital Government Index proposed by OECD (OECD, 2020) focus on supply-side indicators and do not include measures on digital government adoption and associated positive outcomes and impacts for internal and external beneficiaries.
The data on digital government adoption is available in official statistics in some countries (notably, EU states and Russia (Dobrolyubova et al., 2017)), both cross-country and national data on digital government benefits and risks is collected and analyzed in various surveys. So far, the analysis of such surveys and the approaches used has been limited to selected studies (Perez-Morote, 2020).
This paper aims to fill this gap and provide a comprehensive international literature review on the use of surveys for evaluating performance of digital government. Based on the review, key trends and potential areas for the future research are identified.
Methodic approach
For the purposes of this paper, we define government digitalisation as a process of introducing digital technologies in public administration aimed at raising its efficiency and effectiveness. Since we aim to review the experience of countries with various levels of digital government maturity, all stages of digital government transformation ranging from digitising to electronic and then digital government, as defined by the OECD (2016) are included in the review.
The paper methodic approach is based on theory of change (Chen, 1990) and public value theory (Moore, 1995) and differentiates four main levels of digital government performance which can be evaluated using survey instruments: inputs (i.e. presence and operation of information systems and tools enabling digital governance), outputs (i.e. adoption of these tools by the target groups, especially, external beneficiaries such as citizens and businesses), outcomes (i.e. improvements in delivering public services and public goods as a result of implementation and adoption of digital technologies), and impacts of government digitalisation on citizen well-being, economic activity and business climate, and overall government effectiveness.
The first two levels (inputs and outputs) represent the process of government digitalisation, while outcomes and impacts characterize public values achieved as a result of government digitalisation. The framework is generally aligned with the digital government value chain approach (Durkiewicz and Janowski, 2021), except that the public value stage is differentiated into outcomes related to specific digital tools or government functions and broader societal impacts.
Following some previous research (Dobrolyubova, 2021), the framework used for this paper also includes cost and risk dimensions, which can either be the main focus of surveys used to evaluate digital government performance or one of the aspects considered (Figure 1).
Risk Dimension
Government digitalization process
Inputs
Outputs
Digital services (e-services, m-services)
Digital tools for policy development and regulation
Digital tools for regulatory enforcement
Internal digital processes (i.e., enterprise architecture)
Uptake of digital services
E-participation
Engagement in digital interaction with enforcement bodies
economy
efficiency
^ Public values
^ Outcomes d ^ Impacts
Satisfaction with improved service quality and accessibility Imrpoved citizen wellbeing
Improved business climate
Citizen empowerment
Effective governance
Better protection of lawful rights
Improved transparency and accountability
cost-efficiency
Cost Dimension
Figure 1. Digital government performance framework
Source: Completed by the author (- hereinafter, unless otherwise noted).
It is noteworthy that both costs and risks are relevant to each stage of the digital government performance framework. Thus, reducing costs at the input level translates into economy. Achieving better outputs at lower costs is characterized by efficiency. The ratio of the public value created and the costs of digitalisation allows for evaluating cost-effectiveness of digital solutions (i.e. value for money).
This approach differs from some recent publications where administrative efficiency and improved capability are also treated as public values (Twizeyimana and Andersson, 2019). We believe that these constructs are important for evaluating government digitalisation but, as such, they do not guarantee greater public values (outcomes and impact). For instance, increased administrative efficiency may allow for rendering more public services at a lower cost, but that does not guarantee that the quality of such services is increased. Improved capability in open government data is an important prerequisite for more active e-participation and, if such participation is perceived to be successful - for greater citizen empowerment, but, as highlighted by the recent research, the presence of e-participation tools does not always lead to high levels of adoption of these tools (Le Blanc, 2020), let alone broader societal impacts.
For structuring literature review and identifying trends, potential gaps, and prospective research areas, this paper differentiates surveys of digital government performance by type of beneficiary (i.e., citizens, businesses, government, and public servants). Various purposes of G2C and G2B digital interaction (i.e., public service delivery, e-participation) are also considered.
Based on the above performance framework and some previous research (Dobrolyubova, 2021), the following hypotheses are addressed in the literature review.
H1: Most survey-based research focuses on digital services rather than digitisation of other public administration functions.
H2: Most survey-based research evaluating digital government performance focuses on the perspective of citizens rather than other beneficiaries (i.e., businesses and government).
H3: The growth in research interest to measuring digital government performance using survey methods is associated with growing emphasis on outcomes and impacts (i.e. public value created by digitalisation) as compared to inputs and outputs. This allows for the formulation of two sub-hypotheses:
H3.1: More recent papers tend to focus more on outcomes compared to less recent studies, and
H3.2: Research focus depends on the maturity of digital government: empirical research on digital government performance in countries with higher level of EGDI tends to focus on public value rather than on the government digitalisation process, while research of government digitalisation in countries with high and medium EGDI tends to focus on inputs and outputs rather than on outcomes and impacts.
Given the significant role of government digitalisation during the pandemic and the broad discussion of related risks including privacy concerns (Rowe, 2020), as well as the issue of trust in government which has been raised before (OECD, 2017) but gained momentum in 2020-2021 (Altmann et al, 2020; Devine et al, 2021; Wong and Jensen, 2020), we can hypothesize that:
H4: Significant attention is paid to trust in government both as a predictor of successful government digitization and as a public value created by the implementation of digital tools, and
H5: The use of surveys to measure the risks of digital government has become more common during the COVID-19 pandemic.
Review sample
To identify the sample of research papers for the analysis, Scopus database was used. The search was conducted in January 2022 as follows: (e-government) OR (digital government) AND (citizen) OR (business) AND (survey) AND (benefits) OR (performance) OR (effectiveness) OR (risks). Initial search results included 399 papers, and the paper dynamics clearly demonstrated the growing research interest in the subject. While in 2001-2005 there were only 3.2 research papers on the subject per annum, in 2016-2020 the average number papers reached 30.4, and 43 research papers were published in 2021 alone.
Given the high pace of digital transformation, it was decided to focus on research papers published in 2011-2021 (i.e., in the last 11 years). The types of documents included in the review were limited to 'article', 'review', and 'conference paper'. Books, book chapters, conference reviews, notes, and documents with unidentified type were excluded from the sample to minimise the risk of duplication. After this exclusion the remaining 272 papers were reviewed manually to confirm that they correspond to the selection criteria:
(i) they contain the analysis of digital government performance: address one or more levels of performance (inputs, outputs, outcomes, or impacts) and (or) analyze risks and (or) costs related to government digitalisation; and
(ii) they use sociological surveys as a source of empirical data to support the analysis.
Website surveys and reviews that do not involve respondents were excluded because such data is available from the bi-annual UN e-government surveys and administrative sources. Since very recent literature was included in the analysis, the number of citations was not considered for paper selection.
After application of these criteria and removal of duplications (one paper was indexed twice in the original sample), a database of 216 research papers was compiled for the analysis. The distribution of research papers that met the criteria and were included in the final sample, as well as papers received from the initial search results is presented in Figure 2.
152
109 HI
i li IL a.
2000 and earlier 2001-2005 2006-2010 2011-2015 2016-2020 2021
■ Initial search results □ Papers meeting the criteria
Figure 2. The number of research papers: initial search results compared to final sample
Figure 2 demonstrates that the number of papers included in the final sample by time period is proportional to initial search results.
Geographically, the sample of research papers represents all continents and sub-regions of the world. About a half of the research papers included in the sample (115 papers or 53% of the total sample) analyse digital government in Asian countries, including the Middle East (40 papers), South Asia and East Asia (27 papers from each sub-region), and Southeast Asia (21 papers) (Figure 3).
ci Africa d Asia
■ Europe & CIS n Americas
n Australia&New Zealand d Global
Figure 3. Distribution of research papers by region, %
More than a quarter of the total sample (58 papers or 57%) contain digital government surveys conducted in Europe and CIS. About 20 papers present surveys of digital government performance in North and Latin America (14 and 6 papers, respectively), 15 papers are devoted to experiences in Africa, 5 papers evaluate digital government in Australia and New Zealand, and 3 studies include countries from various world regions. Overall, the sample covers the experiences of government digitalisation in 61 countries.
The papers included in the sample also vary by income group as defined by the World Bank1. Thus, about a half of the papers are based on the experiences of high-income countries (49% or 105 papers), 30 percent of the sample are studies of digital government in upper middle-income countries (66 papers), while studies of lower middle-income countries comprise 16 percent of the sample. The sample also includes 5 papers from low-income countries and 6 cross-country studies that include countries from various income groups (Figure 4).
7% 3%
■ High income
■ Upper middle income
■ Lower middle income
■ Low income
■ Countries from various groups
Figure 4. Distribution of papers by country income groups, %
Most of the selected papers (137 out of 216) research the experiences of countries with very high levels of EGDI, as measured by the recent UN Survey (United Nations, 2020). However, papers analysing digital government in countries with high and medium levels of EDGI, are also notable (53 and 23 papers, respectively).
Survey subject and respondents
Most studies in our sample focus on one type of digital government functions, though 19 papers (Karunasena et al, 2011; Shah and Lim, 2011; Baldwin et al, 2012; Olatubosun and Rao, 2012; Fu and Lee, 2014; Dos Santos Brito et al, 2015; Batlle-Montserrat, 2016; Maslihatin, 2016; Scott et al, 2016; Deng et al, 2018; Gómez, 2017; Ma and Zheng, 2016; Ma and Zheng, 2018; Akkaya and Krcmar, 2019; Tensina et al, 2019; Vidiasov et al, 2020; Vidiasova et al, 2020; Betala and Gawade, 2021; Zhang and Lv, 2021) focus on two or more areas of government digitalisation.
Over a half of the papers reviewed (59.3 percent) focus on the evaluation of digital services, and only 8 percent of the studies have an e-participation em-
1 URL: https://datahelpdesk.worldbank.org/knowledgebase/articles/378834-how-does-the-world-bank-classify-countries (accessed: February 2 2022).
phasis. About one-third of the papers (32.6 percent) addresses the issues of regulation, enforcement, and other aspects of digital government performance (not directly related to digital services or e-participation). Over the past decade, the structure of surveys by subject has not changed significantly (Figure 5) though there is some decrease in the share of studies evaluating digital public services (from 62.8 percent in 2011-2015 to 57.6 percent in 2016-2021) with respective increase in the percentage of studies looking at regulatory, enforcement, and other types of functions in public administration (from 29.5 to 34.2 percent).
91
2011-2015 2016-2021
■ digital services ■ e-participation ■ regulation, enforcement, etc.
Figure 5. The number of papers by subject
Note: the figures do not some up as some papers have more than one subject.
Thus, the first hypothesis of this study (H1) is confirmed: most survey papers in the international literature indeed focus on digital services rather than on other types of public administration functions, while the situation is slowly changing and more aspects are attracting researchers' attention.
Based on the literature review, 5 types of surveys can be defined depending on the respondents:
- citizen surveys (151 papers),
- business surveys (13 papers),
- public official surveys and surveys of public institutions staff, usually from universities (38 papers),
- expert surveys (2 papers), and
- comprehensive surveys that use two or more types of samples listed above (13 papers).
While citizen surveys form the largest group of studies, they vary significantly. They include (i) large cross-country randomized studies conducted in developed (Zheng and Schachter, 2017) and developing (Tassabehji et al, 2019) countries, (ii) national representative randomised studies, for instance, those carried out in the US (Nam and Sayogo, 2011), Germany (Gerpott, Ahmadi, 2016), Taiwan (Lee and Chu, 2018), Hong Kong (Venkatesh et al, 2012); Jordan (Alqaralleh et al, 2020), Pakistan (Khurshid et al, 2019), Portugal (Gonçalves et al, 2021), Russia (Revyakin and da Rocha 2021) and (iii) smaller surveys lim-
ited to selected cities or regions, for instance, Province of Quebec in Canada (Abidi et al, 2012), St. Petersburg City in Russia (Belyi, Chugunov, 2021; Vidias-ova et al, 2020), Madina City in Saudi Arabia (Weerakkody et al, 2013), Surabaya city in Indonesia (Susanto et al, 2017); special target population groups, both potentially vulnerable such as the elderly (Choudrie et al, 2017), persons with disabilities (Chu et al, 2011), citizens residing in rural or remote areas (Roy et al, 2015; Dawadi and Shakya, 2016) and those potentially in a better position to engage in digital interaction, i.e., onliners (Niehaves et al, 2012), netizens (Harun et al, 2018), or youth in general (Alomari, 2021).
The literature review also helped identify examples of non-conventional approaches to citizen surveys which are not widespread but could be useful in supporting further digital transformation efforts. One example is related to using a controlled trial method to assess citizen perceptions of regulatory sandboxes conducted in Taiwan (Huang et al, 2021). The survey results demonstrate that respondents who had received more information about regulatory sandboxes before the survey tend to perceive related risks lower than those who had not received similar information. Another example is related to using survey methods to assess the actual citizen experience in applying for public services, i.e., 'citizen journeys' (Sholta et al, 2020), which then help to adapt public services to the specific user needs.
Noteworthy, the second most frequent type of survey conducted to evaluate government digitalisation is a public official survey. The objectives of these surveys vary from evaluating ICT adoption and introduction of certain technological solutions, such as e-document systems in Portugal (Afonso et al, 2012), enterprise architecture, for instance in South Korea (Lee et al, 2016) and Finland (Ba-naeianjahromi, 2018), public clouds in South Korea (Lee et al, 2020), to assessing e-readiness and adoption of e-government practices at the municipal level mostly in developed countries (Liste and Sorensen, 2015; Batlle-Montserrat et al, 2016; Chen and Kim, 2019; Madytinos, Sidiropoulou, 2020; Tangi et al, 2021) and also in developing (Batara et al, 2017) countries. Some researchers go further and use surveys of public officials to evaluate potential outcomes and impacts of e-government (Torres and Pina, 2011; Abu-Shanab, 2017; Abutabenjeh et al, 2021).
Thus, the internal perspective of the public officials on government digitali-sation has a higher influence on international digital government literature than perspective of other external beneficiaries, i.e. businesses. The business surveys that we found focus primarily on the adoption of e-invoicing (Haag et al, 2013; Lian, 2015; Poel et al, 2016; Qi and Che Azmi, 2021), e-procurement (Gupta and Narain, 2012; Seo et al, 2018), and e-filing (Kastik, 2019; Sobotovicova and Blechova, 2021), while the outcomes and impacts for businesses are rarely assessed. An important exemption from this trend is the study (Vashist et al, 2021) evaluating the correlation between e-governance development and profitability of medium and small enterprises in India.
Comprehensive surveys are also not common, but they are usually based on a broader perspective and allow comparing the views of citizens and public officials (Alssbaiheen and Love, 2015; Elamin and Abushama, 2016; Charbonneau and Doberstein, 2020; Vidiasova and Cronemberger, 2020), businesses and experts (Sorum and Fagerstrom, 2015), or public officials and experts (Guntur et al,
2018; Purwandari et al, 2019). There are also several examples of papers based on surveys of all digital government stakeholders: citizens, businesses, and public officials (Alhyari et al, 2013; Misra, 2014; Wang et al, 2021).
The least frequently used approach involves expert surveys of digital government performance. We found only two such studies in the recent international literature: one on organizational change inspired by introducing enterprise architecture in South Korean public sector (Nam et al, 2016) and another one on critical factors of smart city development (Vidiasova et al, 2019).
Based on the above, the second hypothesis of this study (H2) is also confirmed: some 69.9 percent of the surveys of digital government performance are citizen surveys.
Inputs, outputs, outcomes, or impacts: What is in the spotlight?
Diverse subjects of survey-based international research in digital government demonstrate that studies vary in both objectives and scope. Not surprisingly, the international studies also differ in terms of research focus. About a half of the studies reviewed (51.4%) focus on digitalisation outputs (i.e., uptake of digital public services and the factors that influence the use of such services). Noteworthy, the focus on outputs is especially high in studies of upper middle-income countries.
More than one-third of studies in our sample (36.6%) are fully or partially focused on the outcomes of government digitalisation (issues of citizen satisfaction with improved service delivery, transparency and accountability, empowerment, and the like). About 21.8% of the studies focus on inputs, i.e., information systems deployed as part of public administration digitalisation. Finally, societal impacts and risks are quite rarely addressed in survey-based evaluations of digital government performance. However, the risk dimension has gained more attention in the recent years and among researchers analyzing government digitalisation experience in high-income countries and countries with very high EGDI scores (Table 1).
Overall, the qualitative data from our review does not support the third hypothesis of the study. The percentage of empirical survey research addressing digital government outcomes has not increased (on the contrary, the share of relevant papers dropped from 48.6 percent in 2011-2015 to 30.6 percent in 2016-2021). Increased digital maturity also does not yield more attention to the public value dimensions of the government digitalisation. Therefore, hypotheses H3.1 and H3.2 are to be rejected.
Table 1
Distribution of research papers by main research focus
Research papers group Research Foci, % of total papers of respective group
Inputs Outputs Outcomes Impacts Risks
All papers 21.8 51.4 36.6 6.0 7.4
Papers published in 2011-2015 20.8 45.8 48.6 6.9 5.6
Papers published in 2016-2021 22.2 54.2 30.6 5.6 8.3
Research papers group Research Foci, % of total papers of respective group
Inputs Outputs Outcomes Impacts Risks
Papers by country group
High income countries 22.9 45.7 35.2 5.7 12.4
Upper middle income countries 16.7 65.2 36.4 3.0 3.0
Lower middle income and low-income countries 28.2 46.2 43.6 10.3 2.6
Very high EGDI 21.9 49.6 38.0 5.8 10.2
High and Medium EGDI 25.0 55.3 36.8 5.3 2.6
Note: The figures do not sum up since some papers have more than one focus (i.e., for instance focus both
on inputs and outputs or both on outputs and outcomes).
Sources: Completed by the author (- hereafter, unless otherwise noted).
From theoretical perspective, input-oriented papers are usually based on DeLone and McLean's information systems success model (DeLone and McLean, 1992) linking system and information quality to individual and organizational impact, and its further developments (Seddon and Kiew, 1996) suggesting that user satisfaction is related to information and system quality and usefulness. More comprehensive models of organisational change, such as Burke and Litwin's model (Burke and Litwin, 1992) are used to explain transactional and transformational effects of digital technologies at the organisational level (Nam et al, 2016).
Evaluating outputs of government digitalisation calls for going beyond organizational boundaries and considering the motives of external e-government users. Hence, output-oriented digital government surveys are based on a broad range of theories and models, including Rogers's diffusion of innovation theory (Rogers, 1962), the Technology Acceptance Model (Davis, 1989), the Unified Theory of Acceptance and Use of Technology, UTAUT (Venkatesh et al, 2003) and its various extensions, such as the Technology Readiness and Acceptance Model, TRAM (Lin et al, 2007) and the Unified Model of E-Governance Acceptance, UMEGA (Dwivedi et al, 2017). Psychological theories, such as the Theory of Planned Behavior (Ajzen, 1991) and Social Cognitive Theory (Bandura, 1986) are also utilized to enhance the explanatory power of acceptance models.
Most output-oriented digital government surveys aim at testing whether the acceptance factors proposed by the above models (such as performance expectancy, effort expectancy, social influence, facilitating conditions, perceived risk, perceived usefulness, prior experience, trust in government and technology and some others) influence citizen attitudes toward e-government and public intent to engage in digital interactions with public authorities in various country contexts, for various population groups and in relation to diverse specific digital technologies, services, or applications.
In most cases, the selected theoretical frameworks are empirically confirmed (Table 2). Thus, quite a few studies confirmed the importance of performance expectancy, facilitating conditions, perceived usefulness, user satisfaction, or positive prior experience, as well as ease of use for greater adoption of digital government tools by citizens and businesses.
Table 2
Key factors determining adoption of digital public services and e-participation:
Synthesis of empirical literature
u>
Factor Digital Services E-participation
Significance confirmed Significance rejected Significance confirmed Significance rejected
Performance expectancy Turkey (Kurfali et al, 2017), Mauritius (Lallmahomed et al, 2017), Thailand (Bhuasiri et al, 2016), Jordan (Rabaa'i, 2017), Indonesia (Adiyarta et al, 2018), Oman (Alraja et al, 2016), Uzbekistan (Avazov and Lee, 2020), Syria (Saleh and Alyaseen, 2021), Colombia (Sánchez-Torres et al, 2021), Pakistan (Khurshid et al, 2019) Portugal (Naranjo-Zolotov et al, 2019), South Africa (Hobololo and Mawela, 2017)
Effort expectancy Jordan (Rabaa'i, 2017), Oman (Alraja et al, 2016), Syria (Saleh and Alyaseen, 2021), Taiwan (Lian, 2015) Turkey (Kurfali et al, 2017), Thailand (Bhuasiri et al, 2016) Portugal (Naranjo-Zolotov et al, 2019)
Social influence Turkey (Kurfali et al, 2017), Thailand (Bhuasiri et al, 2016), Jordan (Rabaa'i, 2017, Alomari, 2021), UK (Al-Muwil et al, 2019), India (Kumar and Sachan, 2017), Taiwan (Lian, 2015), Greece (Chatzoglou et al, 2015) Jordan (Almaiah and Nasereddin, 2020) South Africa (Hobololo and Mawela, 2017) Portugal (Naranjo-Zolotov et al, 2019)
Facilitating conditions Turkey (Kurfali et al, 2017), Mauritius (Lallmahomed et al, 2017), Thailand (Bhuasiri et al, 2016), Jordan (Rabaa'i, 2017; Alomari, 2021), UK (Al-Muwil et al, 2019), Saudi Arabia (Alharbi, 2021), Syria (Saleh and Alyaseen, 2021), Indonesia (Jacob et al, 2018) Portugal (Naranjo-Zolotov et al, 2019) South Africa (Hobololo and Mawela, 2017)
Perceived usefulness, user satisfaction, prior experience USA (Nam and Sayogo, 2011), Canada (Abidi et al, 2012; Roy et al, 2015), Greece (Chatzoglou et al, 2015), Mauritius (Lallmahomed et al, 2017), UAE (AL Athmay et al, 2016), UK (Al-Muwil et al, 2019), Jordan (Alqaralleh et al, 2020), UAE (Eid et al, 2020), India (Kumar and Sachan, 2017), Malaysia (Qi and Che Azmi, 2021), India (Vincent and Sengupta, 2019) Indonesia (Susanto et al, 2017), China (Wangetal, 2021), EU (Zheng and Schachter, 2017), Germany (Winkler et al, 2012)
Ease of use Canada (Roy et al, 2015), UK (Al-Muwil et al, 2019), Jordan (Alqaralleh et al, 2020), UAE (Eid et al, 2020), India (Kumar and Sachan, 2017), Canada (Abidi et al, 2012) Indonesia (Susanto et al, 2017), Germany (Winkler et al, 2012)
ft
s
ft g
I
S Q
S
r-
o
0
1
I ft
2;
s
c
5o
ï
S
c:
I
O §
s
•K
Factor Digital Services E-participation
Significance confirmed Significance rejected Significance confirmed Significance rejected
Trust in technology (Internet) Turkey (Kurfali et al, 2017), India (Kumar and Sachan, 2017), Indonesia (Assegaffet al, 2021), China (Tang et al, 2021) UK (Al-Muwil et al, 2019) Taiwan (Fu and Lee, 2014)
Trust in government UK (Al-Muwil et al, 2019), Indonesia (Jacob et al, 2017), UAE (Eid et al, 2020), Saudi Arabia (Alharbi, 2021), India (Kumar and Sachan, 2017), Malaysia (Qi and Che Azmi, 2021), China (Tang et al, 2021), Jordan (Alomari, 2021), Colombia (Sánchez-Torres et al, 2021), Pakistan (Khurshid et al, 2019), Taiwan (Lian, 2015), Canada (Roy et al, 2015), Greece (Chatzoglou et al, 2015), Canada (Abidi et al, 2012) Turkey (Kurfali et al, 2017), Indonesia (Assegaff et al, 2021) Taiwan (Fu and Lee, 2014)
Perceived risk UAE (Eid et al, 2020), Indonesia (Fakhruzzaman and Dimitrova, 2020), Taiwan (Lian, 2015), Canada (Abidi et al, 2012; Roy et al, 2015), Greece (Chatzoglou et al, 2015) Thailand (Bhuasiri etal, 2016), UK (Al-Muwil et al, 2019) Taiwan (Fu and Lee, 2014)
Age USA (Nam and Sayogo, 2011) Saudi Arabia (Choudrie et al, 2017), Germany (Gerpott and Ahmadi, 2016), Sudan (Elamin and Abushama, 2016), India (Vincent and Sengupta, 2019) South Africa (Hobololo and Mawela, 2017)
Gender Saudi Arabia (Choudrie et al, 2017), Germany (Gerpott and Ahmadi, 2016), Brazil (MacAya et al, 2021), Sudan (Elamin and Abushama, 2016), Pakistan (Arshad and Khurram, 2021), India (Vincent and Sengupta, 2019), Slovenia (Cestnik and Kern, 2017), Finland (Taipale, 2013) South Africa (Hobololo and Mawela, 2017)
E-literacy, education Jordan (Nawafleh, 2018), UK (Al-Muwil et al, 2019), Saudi Arabia (Choudrie et al, 2017), Germany (Gerpott and Ahmadi, 2016), Sudan (Elamin and Abushama, 2016), Finland (Taipale, 2013)
Income Finland (Taipale, 2013), Germany (Gerpott and Ahmadi, 2016), Sudan (Elamin and Abushama, 2016)
Urbanization Germany (Gerpott and Ahmadi, 2016), Finland (Taipale, 2013)
IS quality and maintenance Jordan (Nawafleh, 2018; (Almaiah and Nasereddin, 2020), Indonesia (Jacob etal, 2017) EU (Ma and Zheng, 2018)
!-d
>
o 3
c
ft
to o to to
en 'TS
ft
C
ft
However, some findings are quite unexpected. For instance, L. Ma and Y. Zheng (2018) find a negative relation between the quality of government websites' design and maintenance in the EU countries, on the one hand, and the extent of e-participation - on the other.
Different studies come to opposite conclusions regarding the importance of trust in government and trust in technology, perceived risk, social influence, and effort expectancy for digital government uptake. Some empirical results support the importance of social and demographic factors such as age, gender, education, and urbanization, but this conclusion is questioned by other authors.
Most outcome-focused surveys of digital government performance are based on public value theory (Moore, 1996), new approaches to public management approaches and especially balanced scorecards (Kaplan and Norton, 1992). Several papers use a citizen-centric COBRA model (Osman et al, 2014) linking cost, benefits, risks, and opportunities to citizen satisfaction with e-government. Some papers also note other factors influencing citizen perception of e-government success such as well-informedness, trust, participation in decision-making (Scott et al., 2016).
The studies vary in defining outcomes and impacts of government digitalisation. While some authors focus on user satisfaction (Chen and Zhang, 2012; Javaid and Arfeen, 2017; Alkraiji, 2021), others propose a broad range of possible public values. For instance, Karunasena et al. (2011) suggest 4 types of public values created by e-government: the delivery of public services, the achievement of outcomes, the development of trust, and the effectiveness of public organizations. In their later works the list of public values was extended to include 'openness of public organisations, equity, citizens' self-development, and environmental sustainability' (Deng et al, 2018). Another survey identified democracy, reflexivity, and productivity as the key values that citizens expect from the government digi-talisation (Agbabiaka, 2018). Some studies focus on public officials' perspectives of government digitalisation outcomes, but improvements in time savings (Abu-Shanab, 2017) and overall administrative efficiency are noted more frequently than better citizen engagement (Rose et al, 2015).
To identify government digitalisation priorities from the beneficiary perspective, some surveys employ willingness to pay approach to identify (Tassabe-hji et al, 2019; Poel et al, 2016), but in general direct cost - benefit analysis is still not very common in government digitalisation evaluation. Interesting examples include papers based on citizen-centric COBRA-methodology (Lee et al., 2015; Al-Yafi et al., 2016), activity-based costing (Miyata, 2021), and measuring financial gains from government digitalisation for businesses (Kastik, 2019; Vashist et al, 2021).
Trust in government is an important construct considered in all types of empirical studies, and the relationship between this construct and public administration digitalisation is examined from various viewpoints. While some papers evaluate the impact of digital government performance on increased trust in government (Sharma et al, 2018; Mahmood et al, 2019), others use trust in government as a factor influencing the adoption of certain digital tools or satisfaction with government digitalisation (see for instance Nam and Sayogo, 2011; Chatzoglou
et al., 2015; Jacob et al., 2017; Eid et al., 2020; Sanchez-Torres et al., 2021). Therefore, while the correlation between trust in government and digital government performance has been confirmed, the causality of the two still present a promising research area.
Overall, the issue of trust (including trust in government and trust in technology) plays a significant role: 51 surveys in our sample analyze trust (23.6 percent of the total) which allows us to confirm the fourth hypothesis of this paper (H4).
Outcome and impact-oriented papers tend to explore predictors of certain positive outcomes or impacts, rather than estimate impacts of specific government digitalisation initiatives. The latter studies demonstrate some positive effects of government digitalisation, but these effects are not always very significant (Liang, 2011). Therefore, measuring the actual outcomes and impacts of government digitalisation beyond e-government adoption seems to be a prospective research area.
Finally, our review suggests that the risks related to government digitalisation gain more and more research attention. While many surveys take into account perceived risk, a recent cross-country survey provides estimates of the actual risk frequency for identity theft: the authors claim that "25-30 percent of the adult population in the surveys countries experienced some form of misuse or attempted misuse of their personal information within the past three years" (Kalvet et al, 2019, p. 660). Risk perceptions are also the focus of recent research on adoption of digital COVID-19 tracing tools (Kozyreva et al., 2021; Lin et al., 2021; Padyab and Kavrestad, 2021), perception and the effects of state surveillance (Tanriverdiand Chen, 2018; Westerlund et al., 2021).
Overall, 8 out of 16 papers primarily addressing government digitalisation risks have been published in the past two years, i.e., during the pandemic. Thus, the fifth hypothesis of this study is confirmed.
Discussion: Gaps and prospects of using surveys for evaluating digital government performance
The review of international empirical survey-based literature focused on evaluating the digital government performance allows for the identification of several gaps which may guide further research.
Firstly, given that more and more countries accept digital-by-default principle for G2B, G2C, and G2G interactions, the adoption of technology is no longer optional for both businesses and citizens. Though some authors (Al-Muwil et al., 2019; Alkraiji, 2021) claim that extended acceptance models are applicable in the case of mandatory digital services, the need and policy relevance of exploring psychological factors driving, for instance, adoption of digital services if the digital communication channel is mandatory (or near-mandatory) is questionable. In this context changing the overall research focus from the outputs and factors driving adoption of technology to the outcomes and impacts of government digitalisation on the society at large seems a logical step forward.
To pursue this path, more empirical data should be collected on the actual (rather than expected) benefits that all stakeholders (especially citizens and
businesses) get from government digitalisation and their perceptions on the impact that digital technologies make on governance quality. While most of the surveys analyzed are based on the perspective of citizens and governments, few of them address the impact of government digitalisation effects on businesses. In this regard, it is worth paying more attention to evaluating digital government outcomes from the business sector lenses. Comprehensive surveys combining the perceptions of various stakeholders identified in this review (see for instance Vidiasova and Cronemberger, 2020; Wang et al., 2021) also provide a promising model for evaluating the outcomes and impacts of government digitalisation.
The review demonstrated that most papers are concentrated on evaluating public services and e-participation while the outcomes of digitalisation of other types of government functions (regulation, enforcement) lacks empirical validation.
The current literature tends to pre-suppose that greater adoption of such technologies is positive for the users. But this may not be the case. Even user satisfaction with certain digital tool that government employs does not mean that the quality of respective public service or function has improved. For instance, business satisfaction with tax e-filing procedures does not automatically mean that the overall efficiency of tax administration has improved, citizen satisfaction with a digital application form for a national passport does not necessarily mean that the administrative procedure has improved significantly, and user-friendliness of an e-participation portal does not guarantee that the issues raised by citizens have been resolved. Addressing this issue, would call for combining the data on perceptions of governance quality with evaluations of public administration digitalisation.
The review demonstrates the growing interest in the issues of government digitalisation risks and related issues of trust in government and trust in technology. The empirical studies on linking these factors with digital government adoption have yielded mixed results. One of the reasons for these mixed research findings may be related to the varied level of trust one citizen might have to various digital tools used by governments for various purposes. For example, one may fully trust the authorities when it comes to applying for social benefits and at the same time distrust the authorities when it comes to digital surveillance tools. The success of ongoing efforts to introduce data-based and data-driven decisions in the public sector, based on applying advanced technologies such as artificial intelligence, would require a better understanding of citizens perceptions of the impact of such technologies on government performance.
One of the promising areas of research in this respect is related to identifying patterns of and reasons for resistance to change and possible protective steps that citizens and businesses can take to avoid the undesired digital surveillance and tracing. A recent paper on self-censorship and the Snowden effect (Tanriverdi and Chen, 2018) presents an example of such an approach.
Noteworthy, digital avoidance challenges some acceptance models which use citizen satisfaction (or prior experience) as a predictor for digital government adoption (see for instance Alotaibi and Roussinov, 2017; Zheng and Schachter,
2017). This factor automatically excludes current non-users of e-government. However, from a practitioner's point of view, it is often more important to find out why some citizens or businesses resist using digital tools rather than why others use them.
Finally, the ongoing COVID pandemic has created natural experimental conditions for researching not only the benefits and costs, but also the limits of digital technology in raising the quality of public administration. While in some areas the use of digital technologies has helped to improve access to public services, notably, in healthcare (United Nations, 2020), there is a growing concern that online learning has decreased school satisfaction (Kirsch et al, 2021) and student motivation (Rahiem, 2021). Therefore, using survey methods to analyze the public value created by government digitalisation from a marginal utility perspective forms another promising area for future studies.
Conclusions
The review of international literature clearly demonstrates growing academic interest in evaluating digital government performance based on survey data. This research area is of interest in both developed and developing countries with varying levels of digital maturity. Increased pace of adopting digital technologies during the pandemic has further stimulated the research, especially on issues related to the risks of digitalisation and the role of trust both as a factor promoting adoption of digital technology and as an impact of digital government performance.
Overall, international studies conclude that importance of performance expectancy, facilitating conditions, perceived usefulness, user satisfaction, or positive prior experience, as well as ease of use influence the adoption of digital government tools by citizens and businesses. The influence of trust, risk perception, and social and demographic factors is debated.
Our review has demonstrated that so far, more research effort has been invested in identifying determinants and barriers to digital governance adoption rather than measuring the actual public value created (or facilitated) by advanced technology. Overall, the papers tend to focus more on inputs, outputs, and administrative efficiency issues rather than on outcomes and impacts. This conclusion is counter-intuitive because the data on outcomes and impacts cannot be easily derived from administrative sources and therefore requires special collection efforts.
While much attention is paid to the perspective of citizens and public officials, the business perspective of digital government performance is less frequently analysed. The same observation applies to the regulatory and enforcement functions of public administration. Further studies on variation of trust in government and trust in technology, measuring perceptions of government digitalisation risks, researching the reasons for digital avoidance, and evaluating public value of digital transformation from marginal utility viewpoint could provide a better theoretical and empirical basis for practitioners engaged in the digital transformation of public governance around the globe.
REFERENCES
1. Abidi, O., Chartier, A., Roy, M.-C. and Poulin, D. (2012) 'Determinants of the use of e-government services in outlying regions: A multi-method approach', in: Gasco M. (ed) Proceedings of the European Conference on e-Government, ECEG. Reading: Academic Publishers International Limited, pp. 1-10.
2. Abu-Shanab, E.A. (2017) 'E-government contribution to better performance by public sector', International Journal of Electronic Government Research, 13(2), pp. 81-96.
3. Abutabenjeh, S., Nukpezah, J.A. and Azhar, A. (2021) 'Do Smart Cities Technologies Contribute to Local Economic Development?', Economic Development Quarterly. DOI: 10.1177/08912424211053599
4. Adiyarta, K., Napitupulu, D., Nurdianto, H., Rahim, R. and Ahmar, A. (2018) User acceptance of E-Government Services Based on TRAM model. IOP Conference Series: Materials Science and Engineering, 352(1), art. No.012057. DOI: 10.1088/1757-899X/352/1/012057
5. Afonso, C., de la Gonzalez, M., Roldán, J. and Sánchez-Franco, M. (2012) 'Determinants of user acceptance of a local eGovernment Electronic Document Management System (EDMS)', in: Gasco M. (ed) Proceedings of the European Conference on e-Government, ECEG. Reading: Academic Publishers International Limited, pp. 19-28.
6. Agbabiaka, O. (2018) 'The public value creation of eGovernment: An empirical study from citizen perspective', in: Kankanhalli A. et al (Eds) ICEGOV '18: Proceedings of the 11th International Conference on Theory and Practice of Electronic Governance. ACM International Conference Proceeding Series, New York: Association for Computing Machinery, pp. 143-153.
7. Ajzen, I. (1991) 'The theory of planned behavior', Organizational Behavior and Human Decision Processes, 50(2), pp. 179-211
8. Akkaya, C. and Krcmar, H. (2019) 'Potential use of digital assistants by governments for citizen services: The case of Germany', in: Chen Y.-C. et al (Eds) Proceedings of the 20th Annual International Conference on Digital Government Research. ACM International Conference Proceeding Series. New York: Association for Computing Machinery, pp. 81-90.
9. Al Athmay, A.A.A., Fantazy, K. and Kumar, V. (2016) 'E-government adoption and user's satisfaction: an empirical investigation', EuroMed Journal of Business, 11(1), pp. 57-83.
10. Alharbi, F. (2021) 'The use of digital healthcare platforms during the COVID-19 pandemic: The consumer perspective', Acta Informatica Medica, 29(1), pp. 51-58.
11. Alhyari, S., Alazab, M., Venkatraman, S., Alazab, M. and Alazab, A. (2013) 'Performance evaluation of e-government services using balanced scorecard: An empirical study in Jordan', Benchmarking, 20(4), pp. 512-536.
12. Alkraiji, A.I. (2021) 'An examination of citizen satisfaction with mandatory e-gov-ernment services: comparison of two information systems success models', Transforming Government: People, Process and Policy, 15(1), pp. 36-58.
13. Almaiah, M.A. and Nasereddin, Y. (2020) 'Factors influencing the adoption of e-government services among Jordanian citizens', Electronic Government, 16(3), pp. 236-259.
14. Al-Muwil, A., Weerakkody, V., El-haddadeh, R. and Dwivedi, Y. (2019) 'Balancing Digital-By-Default with Inclusion: A Study of the Factors Influencing E-Inclusion in the UK', Information Systems Frontiers, 21(3), pp. 635-659.
15. Alomari, M.K. (2021) 'M-government trust framework: deployment of an empirical study amongst Jordanian youth', Transforming Government: People, Process and Policy, DOI: 10.1108/TG-04-2020-0062
16. Alotaibi, S. and Roussinov, D. (2017) User acceptance of M-government services in Saudi Arabia: An SEM approach. Paper presented at 17th European Conference on Digital Government, Lisbon, Portugal, 12/06/17-13/06/17.
17. Alqaralleh, B.A.Y., Al-Omari, A.H. and Alksasbeh, M.Z. (2020) 'An integrated conceptual model for m-government acceptance in developing countries: The case study of Jordan', International Journal of Interactive Mobile Technologies, 14(6), pp. 115-136
18. Alraja, M.N., Hammami, S., Chikhi, B. and Fekir, S. (2016) 'The influence of effort and performance expectancy on employees to adopt e-government: Evidence from Oman', International Review of Management and Marketing, 6(4), pp. 930-934.
19. Alssbaiheen, A. and Love, S. (2015) 'M-government adoption in Saudi Arabia: Challenges and opportunities', International Journal of Technology and Human Interaction, 11(3), pp. 51-68.
20. Altmann S., Milsom L., Zillessen H., Blasone R., Gerdon F., Bach R., Kreuter F., Nosen-zo D., Toussaert S. and Abeler J. (2020) 'Acceptability of App-Based Contact Tracing for COVID-19: Cross-Country Survey Study', JMIR Mhealth Uhealth, 8(8), e19857
21. Al-Yafi, K., Hindi, N.M. and Osman, I.H. (2016) 'A user-centric evaluation of e-gov-ernment services in the GCC region: Case of state of Qatar', International Journal of Electronic Government Research, 12(4), pp. 15-34.
22. Arshad, S. and Khurram, S. (2021) 'Gender difference in the continuance intention to e-file income tax returns in Pakistan', Information Polity, 26 (2), pp. 147-155.
23. Assegaff, S., Andrianti, A. and Astri, L.Y. (2021) 'Evaluation of the factors influencing the trust of millennial citizens in e-government', Journal of Physics: Conference Series, 1898 (1), Art. No. 012009
24. Avazov, S. and Lee, S. (2020) 'E-government adoption in Uzbekistan: Empirical validation of the unified model of electronic government acceptance (UMEGA)', in: Eom S.-J., Lee J. (Eds) dg.o '20: The 21st Annual International Conference on Digital Government Research. ACM International Conference Proceeding Series. New York: Association for Computing Machinery, pp. 338-339.
25. Baldwin, J.N., Gauld, R. and Goldfinch, S. (2012) 'What public servants really think of e-government', Public Management Review, 14(1), pp. 105-127.
26. Banaeianjahromi, N. (2018) Where enterprise architecture development fails a multiple case study of governmental organizations. Paper presented at International Conference on Research Challenges in Information Science, Nantes, France, 29/05/2018.
27. Bandura, A. (1986) Social foundations of thought and action: a social cognitive theory. Englewood Cliffs, N.J.: Prentice-Hall
28. Bannister F. and Connolly R. (2020) 'The Future Isn't What It Used to Be: Forecasting the Impact of ICT on the Public Sphere', Government Information Quarterly, 37(1), art. 101410
29. Batara, E., Nurmandi, A., Warsito, T. and Pribadi, U. (2017) 'Are government employees adopting local e-government transformation? The need for having the right attitude, facilitating conditions and performance expectations', Transforming Government: People, Process and Policy, 11(4), pp. 612-638.
30. Batlle-Montserrat, J., Blat, J. and Abadal, E. (2016) 'Local e-government Benchle-arning: Impact analysis and applicability to smart cities benchmarking', Information Polity, 21 (1), pp. 43-59.
31. Belyi, V.A. and Chugunov, A.V. (2021) E-government services introduction effects in the Covid-19 pandemic: 2020-2021 surveys results. CEUR Workshop Proceedings, 3066, pp. 147-155. Available at: http://ceur-ws.org/Vol-3066/spaper3.pdf (accessed: February 4 2022).
32. Betala, R. and Gawade, S. (2021) 'Usability improvements in e-governance applications with simple and usable interface (design stage)', in: Luhach A.K., Jat D.S., Bin Ghazali K.H., Gao XZ., Lingras P. (eds) Advanced Informatics for Computing Research. ICAICR 2020. Communications in Computer and Information Science, Singapore: Springer, pp. 170-180
33. Bhuasiri, W., Zo, H., Lee, H. and Ciganek, A.P. (2016) 'User Acceptance of e-govern-ment Services: Examining an e-tax Filing and Payment System in Thailand', Information Technology for Development, 22(4), pp. 672-695.
34. Burke, W.W. and Litwin, G.H. (1992) 'A Causal Model of Organizational Performance and Change', Journal of Management, 18(3), pp. 523-545.
35. Cestnik, B. and Kern, A. (2019) 'Supporting decisions with fast and frugal trees for improving trust in public housing services', in: Vassilev T., Smikarov A. (Eds). Proceedings of CompSysTech '19: 20th International Conference on Computer Systems and Technologies, Ruse Bulgaria June 21-22, 2019. ACM International Conference Proceeding Series, New York: Association for Computing Machinery, pp. 118-123.
36. Charbonneau, É. and Doberstein, C. (2020) 'An empirical assessment of the intru-siveness and reasonableness of emerging work surveillance technologies in the public sector', Public Administration Review, 80 (5), pp. 780-791.
37. Chatzoglou, P., Chatzoudes, D. and Symeonidis, S. (2015) Factors affecting the intention to use e-Government services. Proceedings of the 2015 Federated Conference on Computer Science and Information Systems, FedCSIS 2015, pp. 1489-1498, available at: https:// annals-csis.org/proceedings/2015/index.html (accessed on: February 4, 2022).
38. Chen, H.-T. (1990) Theory-driven evaluations. Newbury Park, CA: Sage.
39. Chen, Y.-C. and Kim, Y. (2019) Adoption of e-government services by small municipalities', International Journal of Organization Theory and Behavior, 22(2), pp. 174-190.
40. Chen, Y.-C. and Zhang, J.-C. (2012) 'Citizen-centric e-government performance: Satisfaction with e-information', Electronic Government, 9(4), pp. 388-402.
41. Choudrie, J., Alfalah, A. and Spencer, N.H. (2017) Older adults adoption, use and diffusion of e-government services in Saudi Arabia, Hail City: A quantitative study. Proceedings of the Annual Hawaii International Conference on System Sciences, 2017-Jan-uary, pp. 2953-2962 Available at: http://hdl.handle.net/10125/41513 (accessed on: February 4, 2022).
42. Chu, P.-Y., Huang, T.-Y. and Huang, N.-W. (2011) 'Measuring performance of eGo-vernment to the disabled: Theory and practice in Taiwan', in: Proceedings of the European Conference on e-Government, ECEG. Reading: Academic Publishers International Limited, pp. 158-167.
43. Davis, F. D. (1989) 'Perceived usefulness, perceived ease of use, and user acceptance of information technology', MIS Quarterly, 13(3), 319-340.
44. Dawadi, B.R. and Shakya, S. (2016) 'ICT implementation and infrastructure deployment approach for rural Nepal', Advances in Intelligent Systems and Computing, 463, pp. 319-331
45. DeLone, W.H. and McLean, E.R. (1992) 'Information systems success: The quest for the dependent variable', Information Systems Research. 3(1), pp. 60-95.
46. Deng, H., Karunasena, K. and Xu, W. (2018) 'Evaluating the performance of e-gov-ernment in developing countries: A public value perspective', Internet Research, 28(1), pp. 169-190.
47. Devine, D., Gaskell, J., Jennings, W. and Stoker, G. (2021) 'Trust and the Coronavirus Pandemic: What are the Consequences of and for Trust? An Early Review of the Literature', Political Studies Review, 19 (2), pp. 274-285.
48. Dobrolyubova E., Alexandrov O. and Yefremov A. (2017) 'Is Russia Ready for Digital Transformation?', in: Alexandrov D. et al (edsj Digital Transformation and Global Society. DTGS 2017. Communications in Computer and Information Science, vol. 745, Cham: Springer, pp. 431-444.
49. Dobrolyubova E., Klochkova E. and Alexandrov O. (2019) 'Digitalization and Effective Government: What Is the Cause and What Is the Effect?', in: Alexandrov D., Boukhanovsky A., Chugunov A., Kabanov Y., Koltsova O., Musabirov I. (eds) Digital Transformation and Global Society. DTGS 2019. Communications in Computer and Information Science, vol. 1038. Cham: Springer, pp. 55-67. https://doi. org/10.1007/978-3-030-37858-5_5
50. Dobrolyubova, E. (2021) 'Measuring outcomes of digital transformation in public administration: Literature review and possible steps forward', The NISPAcee Journal of Public Administration and Policy, 14 (1), pp. 61-86.
51. Dos Santos Brito, K., Da Silva Costa, M.A., Garcia, V.C. and De Lemos Meira, S.R. (2015) 'Assessing the benefits of open government data: The Case of meu congresso nacional in Brazilian Elections 2014', in: Mosberger K. et al (Eds) dg.o '15: Proceedings of the 16th Annual International Conference on Digital Government Research. ACM International Conference Proceeding Series, New York: Association for Computing Machinery, pp. 89-96.
52. Durkiewicz, J. and Janowski, T. (2021) 'Chain Action - How Do Countries Add Value Through Digital Government?', in: Proceedings of the 54th Hawaii International Conference on System Sciences. Available at: https://scholarspace.manoa.hawaii.edu/ bitstream/10125/70897/1/0229.pdf (accessed: January 31, 2022).
53. Durkiewicz, J. and Janowski, T. (2018) 'Is Digitalization Improving Governance Quality? Correlating Analog and Digital Benchmarks', in: Bouzas-Lorenzo, R. et al. (Eds) Proceedings of the 18th European Conference on Digital Government ECDG 2018. Reading: Academic Conferences and Publishing International Limited, pp. 48-56.
54. Dwivedi, Y. K., Rana, N. P., Janssen, M., Lal, B., Williams, M. D., and Clement, M. (2017) 'An empirical validation of a unified model of electronic government adoption (UMEGA)', Government Information Quarterly, 34(2), 211-230.
55. Eid, R., Selim, H. and El-Kassrawy, Y. (2020) 'Understanding citizen intention to use m-government services: an empirical study in the UAE', Transforming Government: People, Process and Policy, 15(4), pp. 463-482.
56. Elamin, R. and Abushama, H. (2016) 'E-readiness assessment for e-government in Sudan', in: Babiker S. and K. Badawi (Eds) Proceedings of 2016 Conference of Basic Sciences and Engineering Studies, SGCAC 2016. Red Hook, NY: Curran Associates, pp. 50-56.
57. Fakhruzzaman, M.N. and Dimitrova, D.V. (2020) 'Factors influencing e-government adoption in Indonesia: The importance of perceived risk', Journal of Advanced Research in Dynamical and Control Systems, 12 (6 SI), pp. 125-131.
58. Fu, K.-J. and Lee, C.-P. (2014) 'The role of trust in the prioritization of channel choices, in: Janssen M., Scholl H.J., Wimmer M.A., Bannister F. (eds) Electronic Government. EGOV 2014. Lecture Notes in Computer Science, vol 8653. Berlin: Springer, pp. 96-105.
59. Gerpott, T.J. and Ahmadi, N. (2016) 'Use levels of electronic government services among German citizens: An empirical analysis of objective household and personal predictors', Transforming Government: People, Process and Policy, 10(4), pp. 637-668.
60. Gomez, E.A.R. (2017) 'Perceptions about the concept and benefits of open government in local governments in Spain', in: Baguma R., De',R., Janowski, T. (Eds). ICEGOV '17: Proceedings of the 10th International Conference on Theory and Practice of Electronic Governance. ACM International Conference Proceeding Series, New York: Association for Computing Machinery, pp. 594-597.
61. Gonçalves, G., Pineiro-Naval, V. and Toniolo, B.P. (2021) 'Who Do the Portuguese Trust? Government Communication Management in the Covid-19 Pandemic [Em Quem Confiam os Portugueses? A Gestäo da Comunicaçâo Governamental na Pandemia Covid-19]', Comunicacao e Sociedade, 40, pp. 169-187.
62. Guntur, M., Purwandari, B., Raharjo, T., Solichah, I. and Kumaralalita, L. (2018) 'Critical Success Factors for information systems development: A case study in e-government', in: ICBIM '18: Proceedings of the 2nd International Conference on Business and Information Management. ACM International Conference Proceeding Series, New York: Association for Computing Machinery, pp. 29-33.
63. Gupta, M. and Narain, R. (2012) 'A study on some aspects of E-Procurement in Indian organizations', Advanced Materials Research, 433-440, pp. 1612-1619.
64. Haag, S., Born, F., Kreuzer, S. and Bernius, S. (2013) 'Organizational resistance to e-invoicing-results from an empirical investigation among SMEs, in: Wimmer M.A., Janssen M., Scholl H.J. (eds) Electronic Government. EGOV 2013. Lecture Notes in Computer Science, vol 8074. Berlin, Heidelberg: Springer, pp. 286-297.
65. Harun, A.F., Adzman, S.N.A., Saman, F.M. and Suliman, S.I. (2018) 'Architecting virality: Information sharing from government FB page to netizens', Indonesian Journal of Electrical Engineering and Computer Science, 9 (3), pp. 660-666.
66. Heidelberg, C.A. (2009) 'Citizens, not Consumers', in: Weerakkody, V., Janssen, M., Dwivedi, Y. (eds) Handbook of research on ICT-enabled transformational government: A global perspective. Hershey, PA: IGI Global.
67. Hobololo, T.S. and Mawela, T. (2017) 'Exploring the use of mobile phones for public participation in the Buffalo City metropolitan municipality', Agris On-line Papers in Economics and Informatics, 9(1), pp. 57-68.
68. Huang, H., Pin-Hsuan Chang, B., Zhou-Peng Liao, C. and Chen, D.-Y. (2021) 'A matter of risk management: The effects of the innovation sandboxes on citizens' risk perceptions', in: DG.O'21: DG.02021: The 22nd Annual International Conference on Digital Government Research. ACM International Conference Proceeding Series, New York: Association for Computing Machinery, pp. 281-285.
69. Jacob, D.W., Md Fudzee, M.F., Salamat, M.A., Kasim, S., Mahdin, H. and Ramli, A.A. (2017) 'Modelling end-user of electronic-government service: The role of information quality, system quality and trust', IOP Conference Series: Materials Science and Engineering, 226 (1), article No. 012096.
70. Jacob, D.W., Yanto, I.T.R., Md Fudzee, M.F. and Salamat, M.A. (2018) 'Maximum attribute relative approach of soft set theory in selecting cluster attribute of electronic government data set', Advances in Intelligent Systems and Computing, 700, pp. 473-484.
71. Javaid, M.A. and Arfeen, M.I. (2017) 'Impact of e-government on citizen satisfaction: A case of federal government agencies in Pakistan, in: Themistocleous M., Morabito V. (eds) Information Systems. EMCIS 2017. Lecture Notes in Business Information Processing, vol. 299. Cham: Springer, pp. 221-237.
72. Kalvet, T., Tiits, M. and Ubakivi-Hadachi, P. (2019) 'Risks and societal implications of identity theft', in: Chugunov A., Misnikov Y., Roshchin E., Trutnev D. (eds) Electronic Governance and Open Society: Challenges in Eurasia. EGOSE 2018. Communications in Computer and Information Science, vol. 947. Cham: Springer, pp. 67-81.
73. Kaplan, R.S. and D.P. Norton (1992) 'The balanced scorecard-measures that drive performance', Harvard Business Review, 70(1), 71-79.
74. Karunasena, K., Deng, H. and Singh, M. (2011) 'Measuring the public value of e-government: A case study from Sri Lanka', Transforming Government: People, Process and Policy, 5(1), pp. 81-99.
75. Kastik, T. (2019) 'The impact of digital governance on the business environment: The case of Estonian tax and customs board', in: Dhaou S.B., Carter L., Gregory M. (Eds) ICEG0V2019: Proceedings of the 12th International Conference on Theory and Practice of Electronic Governance. ACM International Conference Proceeding Series, New York: Association for Computing Machinery, pp. 472-474.
76. Khurshid, M.M., Zakaria, N.H., Rashid, A., Ahmed, Y.A. and Shafique, M.N. (2019) 'Adoption of Transactional Service in Electronic Government - A Case of Pak-Iden-tity Service', in: Pappas I.O., Mikalef P., Dwivedi Y.K. et al (eds) Digital Transformation for a Sustainable Society in the 21st Century. I3E 2019. Lecture Notes in Computer Science, vol. 11701. Cham: Springer, pp. 439-450.
77. Kirsch C., Engel de Abreu P., Neumann S. and Wealer C. (2021) 'Practices and experiences of distant education during the COVID-19 pandemic: The perspectives of six- to sixteen-year-olds from three high-income countries', International Journal of Educational Research Open, 2, article No. 100049.
78. Kozyreva, A., Lorenz-Spreen, P., Lewandowsky, S., Garrett, P.M., Herzog, S.M., Pachur, T. and Hertwig, R. (2021) 'Psychological factors shaping public responses to COVID-19 digital contact tracing technologies in Germany', Scientific Reports, 11 (1), article No.18716.
79. Kumar, R. and Sachan, A. (2017) 'Empirical study to find factors influencing e-Filing adoption in India', in: Baguma R. et al. (Eds) ICEGOV '17: Proceedings of the Special Collection on eGovernment Innovations in India. ACM International Conference Proceeding Series, New York: Association for Computing Machinery, pp. 52-57.
80. Kurfali, M., Arifoglu, A., Tokdemir, G. and Paçin, Y. (2017) 'Adoption of e-govern-ment services in Turkey', Computers in Human Behavior, 66, pp. 168-178.
81. Lallmahomed, M.Z.I., Lallmahomed, N. and Lallmahomed, G.M. (2017) 'Factors influencing the adoption of e-Government services in Mauritius', Telematics and Informatics, 34 (4), pp. 57-72.
82. Le Blanc, D. (2020) E-participation: a quick overview of recent qualitative trends. UN DESA Working Paper. No. 163. Available at: https://www.un.org/development/ desa/publications/working-paper/wp163 (accessed: February 4, 2022).
83. Lee, H., Sivarajah, U., Molnar, A., Weerakkody, V. and Irani, Z. (2015) 'A user satisfaction study of London's Congestion Charge e-service: A citizen perspective', International Journal of Electronic Government Research, 11 (2), pp. 35-50.
84. Lee, L.-W. and Chu, P.-Y. (2018) 'A user approach to open government data impact assessment', in: Proceedings of the European Conference on e-Government, ECEG, 2018-0ctober, pp. 111-118.
85. Lee, S., Oh, S.W. and Nam, K. (2016) 'Transformational and transactional factors for the successful implementation of enterprise architecture in public sector', Sustain-ability (Switzerland), 8 (5), article No. 456
86. Lee, S., Choi, Y., Ra, J., Kim, J. and Ashihara, K. (2020) 'Impact of public cloud computing service in Korean government organizations', ICIC Express Letters, Part B: Applications, 11(3), pp. 313-318.
87. Lian, J.-W. (2015) 'Critical factors for cloud based e-invoice service adoption in Taiwan: An empirical study', International Journal of Information Management, 35 (1), pp. 98-109.
88. Liang, T.-H. (2011) 'Internet Service Cognition and Use, and Their Promotion of Quality of Life in Taiwan', Social Indicators Research, 102 (1), pp. 99-110.
89. Lin, C. H., Shih, H. Y., and Sher, P. J. (2007) 'Integrating technology readiness into technology acceptance: The TRAM model', Psychology and Marketing, 24(7), 641-657.
90. Lin, J., Carter, L. and Liu, D. (2021) 'Privacy concerns and digital government: exploring citizen willingness to adopt the COVIDSafe app', European Journal of Information Systems, 30 (4), pp. 389-402.
91. Liste, L. and Sorensen, K.H. (2015) 'Consumer, client or citizen? How Norwegian local governments domesticate website technology and configure their users', Information Communication and Society, 18 (7), pp. 733-746.
92. Ma, L. and Zheng, Y. (2016) '"Good wine needs bush": A multilevel analysis of national E-Government performance and citizen use across European countries' In: Kim Y., Liu M. (Eds) dg.o '16: Proceedings of the 17th International Digital Government Research Conference on Digital Government Research. ACM Conference Proceedings Series. New York: Association for Computing Machinery, pp. 184-193.
93. Ma, L. and Zheng, Y. (2018) 'Does e-government performance actually boost citizen use? Evidence from European countries', Public Management Review, 20 (10), pp. 1513-1532.
94. MacAya, J.F.M., Ribeiro, M.M., Jereissati, T., Dos Reis Lima, C. and Cunha, M.A. (2021) 'Gendering the digital divide: The use of electronic government services and implications for the digital gender gap', Information Polity, 26(2), pp. 131-146.
95. Madytinos, D. and Sidiropoulou, N.N. (2020) 'Incentives for the adoption of e-gov-ernment by Greek municipalities', International Journal of Economics and Business Administration, 8 (1), pp. 298-326.
96. Mahmood, M., Weerakkody, V. and Chen, W. (2019) 'The influence of transformed government on citizen trust: insights from Bahrain', Information Technology for Development, 25 (2), pp. 275-303.
97. Margetts, H. and Dunleavy, P. (2013) 'The second wave of digital-era governance: a quasi-paradigm for government on the Web', Philosophical Transactions of the Royal Society, 371(1987)
98. Maslihatin, T. (2016) 'Balanced scorecard: Performance measurement for e-govern-ment', Journal of Theoretical and Applied Information Technology, 90 (2), pp. 116-123.
99. Misra, H. (2014) 'Convergence in rural e-governance services for sustainable development: A model driven study in Indian context', in: Estevez E., Janssen M., Soares Barbosa, L. (Eds) ICEGOV '14: Proceedings of the 8th International Conference on Theory and Practice of Electronic Governance. ACM International Conference Proceeding Series, New York: Association for Computing Machinery, pp. 162-165.
100. Miyata, M. (2021) 'Measuring impacts of e-government support in least developed countries: A case study of the vehicle registration service in Bhutan', Information Technology for Development, 17(2), pp. 133-152.
101. Moore, M. (1995) 'CreatingPublic Value - Strategic Management in Government'. Cambridge: Harvard University Press.
102. Nam, K., Oh, S.W., Kim, S.K., Goo, J. and Sajid Khan, M. (2016) 'Dynamics of enterprise architecture in the Korean public sector: Transformational change vs. transactional change', Sustainability (Switzerland), 8 (11), Article No.1074
103. Nam, T. and Sayogo, D.S. (2011) 'Who uses e-government? Examining the digital divide in e-government use', in: Estevez E. et al (Eds) ICEGOV '11: Proceedings of the 5th International Conference on Theory and Practice of Electronic Governance. ACM International Conference Proceeding Series, New York: Association for Computing Machinery, pp. 27-36.
104. Naranjo-Zolotov, M., Oliveira, T. and Casteleyn, S. (2019) 'Citizens' intention to use and recommend e-participation: Drawing upon UTAUT and citizen empowerment', Information Technology and People, 32(2), pp. 364-386.
105. Nawafleh, S. (2018) 'Factors affecting the continued use of e-government websites by citizens: An exploratory study in the Jordanian public sector', Transforming Government: People, Process and Policy, 12 (3-4), pp. 244-264.
106. Niehaves, B., Gorbacheva, E. and Plattfaut, R. (2012) 'The digital divide vs. The e-Government divide: Do socio-demographic variables (still) impact e-Government use among onliners?', in: ECIS 2012 - Proceedings of the 20th European Conference on Information Systems. Available at: https://aisel.aisnet.org/ecis2012/199 (accessed: February 4, 2022)
107. OECD (2016) Digital Government Strategies for Transforming Public Services in the Welfare Areas. Paris: OECD Publishing
108. OECD (2017) Trust and Public Policy: How Better Governance Can Help Rebuild Public Trust. Paris: OECD Publishing.
109. OECD (2020) Digital Government Index: 2019 results. OECD Public Governance Policy Papers, No. 3. Paris: OECD Publishing.
110. Olatubosun, O. and Rao, K.S.M. (2012) 'Empirical study of the readiness of public servants on the adoption of e-government', International Journal of Information Systems and Change Management, 6 (1), pp. 17-37.
111. Osman I.H., Anouze A.L., Irani Z., Al-Ayoubi B., Lee H., Balci A., Medeni T.D. and Weerakkody V. (2014) 'COBRA framework to evaluate e-government services: A citizen-centric perspective', Government Information Quarterly, 31(2), pp. 243-256.
112. Padyab, A. and Kävrestad, J. (2021) 'Perceived Privacy Problems Within Digital Contact Tracing: A Study Among Swedish Citizens', in: J0sang A., Futcher L., Hagen J. (eds) ICT Systems Security and Privacy Protection. SEC 2021. IFIP Advances in Information and Communication Technology, vol 625. Cham: Springer, pp. 270-283.
113. Pérez-Morote, R., Pontones-Rosa, C. and Núñez-Chicharro, M. (2020) 'The effects of e-government evaluation, trust and the digital divide in the levels of e-government use in European countries', Technological Forecasting and Social Change, 154(March), article No.119973.
114. Poel, K., Marneffe, W. and Vanlaer, W. (2016) 'Assessing the electronic invoicing potential for private sector firms in Belgium', International Journal of Digital Accounting Research, 16, pp. 1-34.
115. Purwandari, B., Adawati, I.R., Sandhyaduhita, P.I. and Solichah, I. (2019) 'Strategies to improve quality of data dissemination website: A case study of the Central bureau of statistics, in: Nunes M.B. et al. (Eds) Proceedings of the 12th IADIS International Conference Information Systems 2019, IS 2019, IADIS Press, pp. 155-162.
116. Qi, Y. and Che Azmi, A. (2021) 'Factors affecting electronic invoice adoption and tax compliance process efficiency', Transforming Government: People, Process and Policy, 15 (1), pp. 150-168.
117. Rabaa'i, A.A. (2017) 'The use of UTAUT to investigate the adoption of e-government in Jordan: A cultural perspective', International Journal of Business Information Systems, 24 (3), pp. 285-305.
118. Rahiem, M.D.H. (2021) 'Remaining motivated despite the limitations: University students' learning propensity during the COVID-19 pandemic', Children and Youth Services Review, 120, article No.105802. Available at: https://doi.org/10.1016/ j.childyouth.2020.105802.
119. Revyakin, S.A. and da Rocha, A. (2021) 'An Empirical Investigation of E-Govern-ment Adoption in Russia: Access, Rights, Trust and Citizens' Experience', Public Administration Issues, 5, pp. 137-160.
120. Rogers, E.M. (1962) Diffusion of innovations (1st ed.). New York: Free Press of Glencoe.
121. Rose, J., Persson, J.S. and Heeager, L.T. (2015) 'How e-Government managers prioritise rival value positions: The efficiency imperative', Information Polity, 20 (1), pp. 35-59.
122. Rowe, F. (2020) 'Contact tracing apps and values dilemmas: A privacy paradox in a neo-liberal world', International Journal of Information Management, 55, article No. 102178.
123. Roy, M.-C., Chartier, A., Crête, J. and Poulin, D. (2015) 'Factors influencing e-gov-ernment use in non-urban areas', Electronic Commerce Research, 15 (3), pp. 349-363.
124. Saleh, A.A., Alkhuwaylidee, A.R. and Thangiah, M. (2021) 'Successful Factors Determining the user behaviour in information system', Journal of Physics: Conference Series, 1963 (1), article No. 012161. DOI: 10.1088/1742-6596/1963/1/012161
125. Sánchez-Torres, J.A., Arroyo-Cañada, F.J., Varon-Sandoval, A. and Sánchez-Alzate, J.A. (2021) 'Adoption of e-government in Colombia: The importance of government policy in citizens' use of e-government', Electronic Government, 17(2), pp. 220-236.
126. Scholta, H., Halsbenning, S., Distel, B. and Becker, J. (2020) 'Walking a mile in their shoes - a citizen journey to explore public service delivery from the citizen perspective', in: Viale Pereira G. et al. (eds) Electronic Government. EGOV 2020. Lecture Notes in Computer Science, vol. 12219. Cham: Springer, pp. 164-178.
127. Scott, M., Delone, W. and Golden, W. (2016) 'Measuring eGovernment success: A public value approach', European Journal of Information Systems, 25 (3), pp. 187-208.
128. Seddon, P. and Kiew, M.-Y. (1996). 'A Partial Test and Development of Delone and Mclean's Model of IS Success', Australasian Journal of Information Systems, 4(1). Available at: https://doi.org/10.3127/ajis.v4i1.379
129. Seo, D., Tan, C.-W. and Warman, G. (2018) 'Vendor satisfaction of E-government procurement systems in developing countries: an empirical research in Indonesia', Information Technology for Development, 24(3), pp. 554-581.
130. Shah, B.P. and Lim, N. (2011) 'Using social media to increase e-government adoption in developing countries, in: Estevez E. et al (Eds) ICEGOV '11: Proceedings of the 5th International Conference on Theory and Practice of Electronic Governance. ACM International Conference Proceeding Series, New York: Association for Computing Machinery, pp. 205-213.
131. Sharma, P.N., Morgeson, F.V., III, Mithas, S. and Aljazzaf, S. (2018) 'An empirical and comparative analysis of E-government performance measurement models: Model selection via explanation, prediction, and parsimony', Government Information Quarterly, 35(4), pp. 515-535.
132. Sobotovicová, S. and Blechová, B. (2021) 'Perception of business entities towards digitization of tax administration in the Czech Republic', E a M: Ekonomie a Management, 24 (3), pp. 208-223.
133. S0rum, H. and Fagerstr0m, A. (2015) 'How to evaluate investments in website quality within eGovernment? Exploring the webmaster's perception of benefits', in: Kuro-su M. (eds) Human-Computer Interaction: Users and Contexts. HCI2015. Lecture Notes in Computer Science, vol. 9171. Cham: Springer, pp. 102-111.
134. Susanto, T.D., Diani, M.M. and Hafidz, I. (2017) 'User Acceptance of e-Government Citizen Report System (a Case Study of City113 App), Procedia Computer Science, 124, pp. 560-568.
135. Taipale, S. (2013) 'The use of e-government services and the Internet: The role of so-cio-demographic, economic and geographical predictors', Telecommunications Policy, 37(4-5), pp. 413-422.
136. Tang, Z., Zhou, Z., Xu, F. and Warkentin, M. (2021) 'Apps within apps: predicting government WeChat mini-program adoption from trust-risk perspective and innovation diffusion theory', Information Technology and People (in press). DOI: 10.1108/ ITP-06-2020-0415.
137. Tangi, L., Soncin, M., Agasisti, T. and Noci, G. (2021) 'Exploring e-maturity in Italian local governments: empirical results from a three-step latent class analysis' International Review of Administrative Sciences. DOI: 10.1177/00208523211012752.
138. Tanriverdi, H. and Chen, H. (2018) 'Government's digital surveillance and citizens' self-censorship of technology use', in: International Conference on Information Systems 2018, ICIS 2018. Available at: https://aisel.aisnet.org/icis2018/impact/Presenta-tions/7 (accessed: February 4, 2022).
139. Tassabehji, R., Hackney, R. and Maruyama, T. (2019) 'Evaluating digital public services: A contingency value approach within three exemplar developing countries', Information Technology and People, 32(4), pp. 1021-1043.
140. Tensina, I., Vidiasova, L. and Bershadskaya, E. (2019) 'Information technologies in G2C communications: Cybersocial trust survey', in: Alexandrov D. et al (eds) Digital Transformation and Global Society. DTGS 2019. Communications in Computer and Information Science, vol. 1038. Cham: Springer, pp. 107-119.
141. Torres, L. and Pina, V. (2011) 'E-participation perspectives. A Delphi study applied to climate change policies', in: Proceedings of the IADIS Int. Conf. ICT, Society and Human Beings 2011, Proceedings of the IADIS International Conference e-Democ-racy, Equity and Social Justice 2011, Part of the IADIS, MCCSIS 2011, Rome; Italy; 20.07.2011 to 26.07.2011, pp. 19-26.
142. Twizeyimana, J.D. and Andersson, A. (2019) 'The public value of E-Government -A literature review', Government Information Quarterly, 36(2), 167-178.
143. United Nations (2020) E-government survey 2020. Digital government in the decade of action for sustainable development. New York: United Nations.
144. Vashist, R.P., Arya, A. and Dhiman, A. (2021) 'An analysis of e-governance and its impact on the profitability of MSMEs: A structural equation modelling approach', FIIB Business Review. DOI: 10.1177/23197145211032729.
145. Venkatesh, V., Morris, M. G. Davis, G.B. and Davis, F.D. (2003) 'User acceptance of information technology: Toward a unified view', MIS Quarterly, 27 (3), pp. 425-478.
146. Venkatesh, V., Chan, F.K.Y. and Thong, J.Y.L. (2012) 'Designing e-government services: Key service attributes and citizens' preference structures', Journal of Operations Management, 30 (1-2), pp. 116-133.
147. Vidiasov, E., Tensina, I. and Vidiasova, L. (2020) 'Institutionalization of C2G new communication forms: Trends and dynamic in Saint Petersburg, in: Chugunov A. et al (eds) Electronic Governance and Open Society: Challenges in Eurasia. EGOSE 2020. Communications in Computer and Information Science, vol. 1349. Cham: Springer, pp. 195-204.
148. Vidiasova, L., Cronemberger, F. and Vidiasov, E. (2019) 'Risk factors in smart city development in Russia: A survey', in: Chugunov A. et al (eds) Electronic governance and open society: Challenges in Eurasia. EGOSE 2018. Communications in Computer and Information Science, vol. 947, Cham: Springer, pp. 26-37.
149. Vidiasova, L. and Cronemberger, F. (2020) 'Discrepancies in perceptions of smart city initiatives in Saint Petersburg, Russia', Sustainable Cities and Society, 59, article No. 102158
150. Vincent, F.M. and Sengupta, A. (2019) 'Factors affecting the use of mobile payments among educated urban population: An Indian perspective', International Journal of Innovative Technology and Exploring Engineering, 8(6), pp. 531-537.
151. Weerakkody, V., El-Haddadeh, R., Al-Sobhi, F., Shareef, M.A. and Dwivedi, Y.K. (2013) 'Examining the influence of intermediaries in facilitating e-government adoption: An empirical investigation, International Journal of Information Management, 33(5), pp. 716-725.
152. Wang, C., Teo, T.S.H., Dwivedi, Y. and Janssen, M. (2021) 'Mobile services use and citizen satisfaction in government: Integrating social benefits and uses and gratifications theory', Information Technology and People, 34(4), pp. 1313-1337.
153. Westerlund, M., Isabelle, D.A. and Leminen, S. (2021) 'The acceptance of digital surveillance in an age of big data', Technology Innovation Management Review, 11(3), pp. 32-44.
154. Winkler, T.J., Ziekow, H. and Weinberg, M. (2012) 'Municipal benefits of participatory Urban sensing: A simulation approach and case validation', Journal of Theoretical and Applied Electronic Commerce Research, 7(3), pp. 101-120.
155. Wong, C.M.L. and Jensen, O. (2020) 'The paradox of trust: Perceived risk and public compliance during the COVID-19 pandemic in Singapore', Journal of Risk Research, 23 (7-8), pp. 1021-1030.
156. World Bank (2016) Digital dividends. World development report. Washington D.C.: World Bank.
157. Zhang, A. and Lv, N. (2021) 'Research on the impact of big data capabilities on government's smart service performance: Empirical evidence from China', IEEE Access, 9, pp. 50523-50537.
158. Zheng, Y. and Schachter, H.L. (2017) 'Explaining citizens' e-participation use: The role of perceived advantages', Public Organization Review, 17, pp. 409-428.
The article was submitted: 04 February 2021; approved after reviewing: 20 March 2021; accepted for publication: 14 April 2022.
