CC BY-NC-ND
11
WHAT DRIVES NATIONAL RESEARCH AND DEVELOPMENT PERFORMANCE? -FACTORS IN COMPETITIVE GOVERNMENT FINANCING OF SCIENCE AND TECHNOLOGY1
DongwookSeoh
Ph.D. Candidate, Graduate School of Public Administration, Seoul National University. Address: Bldg. 57-1, 1 Gwanak-ro, Gwanak-gu, Seoul, Korea 08826. E-mail: [email protected]
Tobin Im
Professor, Graduate School of Public Administration, Seoul National University. Address: Bldg. 57-1, 1 Gwanak-ro, Gwanak-gu, Seoul, Korea 08826. E-mail: [email protected]
Abstract
This research mainly explores factors that may contribute to national performance on science and technology research and development (R&D). The 17-year panel data regression analysis on 35 OECD countries and Russia provides evidence to potential causal relationships between value-added of industry and several investment variables. As a proxy variable for research and development performance, value-added of industry indicates a statistically significant positive relationship with business-performed expenditure of government-funding on R&D. It indicates negative relationships with government-performed expenditure of government-funding on R&D and regulatory burdens. The results suggest that national R&D policy should guarantee less regulatory burden and more autonomy of government-funded R&D projects.
Keywords: research and development; regulatory burden; public investment; government competitiveness; performance management.
Citation: Seoh, D. & Im, T. (2020). What Drives National Research and Development Performance? - Factors in Competitive Government Financing of Science and Technology. Public Administration Issues, no 5, (Special Issue I, electronic edition), pp. 191-215 (in English); DOI: 10.17323/1999-5431-2020-0-5-191-215
1 This research is supported by National Research Foundation of Korea (NRF-2017S1A3A2065838).
Introduction
At the center of technological innovation are significant investments on research and development (R&D). Increasingly aware of the importance of innovation, governments and private industries of developed and developing countries have allocated parts of their budgets to research and development. By spending on research and development, firms expect innovations to result in a productivity increase at the industry level (Griffith et al., 2001). Governments often subsidize innovative activities to further promote research and development (Ko et al., 2015; Song et al., 2009).
The South Korean government in the developing era, for example, realized that economic development based mainly on adopting and imitating developed countries' technology is unsustainable. Thus, it established the Korea Institute of Science and Technology (KIST) in 1966 to invite Korean researchers overseas to promote R&D. Such an attempt was devised to bring the manufacturing-driven Korean economy2 to the next level. Public investments in research and development have also been accompanied by increasing private investments by firms. In 2018, records show that R&D expenditures accounted for 4.53% of South Korean gross domestic product.
Table 1
R&D Expenditures as % of GDP, Select OECD countries, Russia and China, 2018
Select Countries Share of R & D expenditure in GDP, %
Austria 3.17
Belgium 2.76
Canada 1.56
Czech Republic 1.93
Denmark 3.03
Finland 2.75
France 2.20
Germany 3.13
Greece 1.18
Hungary 1.53
Iceland 2.03
Ireland 1.15
Italy 1.39
2 The main sources of South Korean economic development included cement, chemicals, automobile, shipbuilding, and electronics, and modernization of these industries contributed to even more rapid growth of the economy (Im, 2014; Kim, 2008).
Select Countries Share of R & D expenditure in GDP, %
Japan 3.26
Korea 4.53
Luxembourg 1.21
Mexico 0.31
Netherlands 2.16
Norway 2.07
Poland 1.21
Portugal 1.35
Slovakia 0.84
Spain 1.24
Sweden 3.31
Great Britain 1.71
United States 2.83
China 2.19
Estonia 1.40
Israel 4.94
Romania 0.51
Russia 0.99
Slovenia 1.95
Taiwan 3.46
EU 28 Countries 2.03
OECD Total 2.40
Latvia 0.64
Lithuania 0.88
Sources: Research and development expenditure (% of GDP). UNESCO Institute of Statistics.
However, whether research and development expenditures guarantee noticeable accomplishments is questionable. Both supporting and opposing discussions prevail on the effectiveness of such expenditures, and quite a few focus on public sector funding. Government investments are sometimes considered ineffective in terms of research and development capability over different sectors, despite prevailing positive perceptions (Goolsbee, 1998). Some of the literature even claims that government funding on R&D would rather decrease private firms' investments on R&D and cause a decrease in the total amount of R&D activities (Wall-sten, 2000). In reality, despite 4.53% of its GDP spent on R&D, South Korea still does not have a Nobel Prize winner in the science and technology sector. While
the size of gross domestic product and other unknown factors may determine the gross expenditure, spending more than 4% of GDP on research and development is rare among OECD countries. While the number of Nobel Prize winners may not be a sole indicator of research and development performance, its small figure juxtaposed with a large amount of expenditure suggests there is a problem.
The main purpose of this paper is to address a factor that may cause effectiveness or ineffectiveness of research and development expenditures at the country level. While much of the existing literature has tried to show positive / negative relationships between public and private R&D spending and R&D performance of individual firms, not many have shown the aggregate effect of R&D expenditures at the national level. Because governments are identified as one of the major sources of R&D funding, it is important to determine how government competi-tiveness3 may play a role in assuring research and development performance.
Another question arises from measurement issues. While it is not challenging to identify costs of R&D, benefits from these costs may be represented by various variables. Composite variables such as Gross Domestic Product is a popular measure of a country's economic development, but they are not specific enough to capture the R&D effect. The present paper thus proposes a more specific variable to measure R&D performance and makes it more convenient to perform cross-country comparison. Once performance of R&D expenditures is measurable, a longitudinal cross-country analysis may provide ideas on which factors contribute to higher performance.
This paper thus aims at addressing impacts of public support and obstacles to research and development on science and technology by analyzing the relationship between government expenditure and performance of the industry. The introductory section of the paper will be followed by a multifaceted literature review and a research question. The research will proceed with results and analysis, to be concluded by its contribution to policy-making processes.
Literature Review
This chapter will introduce potential factors that may affect performances of research and development activities. First, it will begin with discussions on defining and measuring research and development. Then the chapter will analyze previous research on different factors: government expenditure, regulations, intellectual properties (patents) and human capital.
Research and Development (R&D)
Before proceeding to any detailed discussion, it is important to first describe what research and development is. According to OECD, research and development (R&D) is "creative and systematic work undertaken in order to increase the stock of knowledge - including knowledge of humankind, culture and society - and to de-
3 Government competitiveness is defined as "the power of government to, in light of various constraints, take resources from in and outside of the country and improve social, economic and cultural conditions of the nation in order to sustainably enhance citizens' quality of life" (Ho & Im, 2012; Im & Hartley, 2019).
vise new applications of available knowledge" (OECD, 2015, p. 44). The International Accounting Standards Committee (IASC) defines research as a unique planned investigation aimed at obtaining newer knowledges on science and technology; IASC's definition on development is: application of new research outcomes on materials, production, system or service design before commercial production (Kim et al, 2013, p. 697). Based on OECD (2015)'s typology of research and development, the Korea National Science & Technology Commission (NSTC) and the Korea Institute of Science and Technology Evaluation and Planning (KISTEP) propose further detailed definitions of different classes of research and development activities.
Table 2
Basic research, applied research, and commercial research / experimental development
Basic research Applied research Commercial research / Experimental development
Definition Experimental or theoretical work undertaken primarily to acquire new knowledge of the underlying foundations of phenomena and observable facts, without any particular application or use in view Original investigation undertaken in order to acquire new knowledge; directed primarily towards a specific, practical aim or objective Systematic work, drawing on knowledge gained from research and practical experience and producing additional knowledge, which is directed towards producing new products or processes or to improving existing products or processes
Key features Mainly based on experiments and observations; individual research Application of results from basic research; basis of commercial research and production Development of commercial products; sales matter
Timeline 10+ years 5-10 years 1-5 years
Source: Modified from OECD (2015). Frascati Manual 2015: Guidelines for Collecting and Reporting Data on Research and Experimental Development. The Measurement of Scientific, Technological and Innovation Activities, OECD Publishing, Paris and Korea Ministry of Science and ICT & KISTEP (2015, p. 6).
Research and development activities can be further categorized into three types: basic research, applied research, and commercial research. Basic research is essentially an experimental or theoretical work undertaken primarily to acquire new knowledge of underlying foundations of phenomena and observable facts, without any particular application or use in view. Industries and governments have especially emphasized the importance of R&D on basic science. Competitive industries may be achieved by securing original basic technologies, which may be the result of continued research and development activities (Yoon, 2014). These factors may contribute even more in advanced industries including, but not limited to, information and communication industries and medical industries (Yoon, 2014). Therefore, this type of research mainly depends on individual research that focuses on experiments and observations, rather than simply investing in capital (i.e. expanding production lines). The duration of research may differ, but the general expectation is more than 10 years to obtain significant results.
Applied research is defined as an original investigation undertaken, aimed at the acquisition of new knowledge, and usually directed towards a rather specific and practical aim or objective. Applied research often develops on results from basic researches, arousing more practical use as a basis of commercial research and production. It is usually expected to run for 5-10 years, which is shorter than the timeline for basic research.
The last phase of research is commercial research or experimental development. It is a compilation of systematic works, drawing on knowledge gained from research and practical experience and producing additional knowledge, which is directed towards producing new products or processes or to improving existing products and processes. At this stage development of a tangible product comes up, and sales do matter. How customers react to this new product may be a credible measurement of success.
Some authors prefer to primarily differentiate research from development. Im (2015) defines research as "long-term academic research that deals with fundamentals and aims for Nobel Prizes," while he defines development as producing industrial and commercial products based on research products (Ibid.). In general, research activities of R&D are carried out by universities and government research agencies, while development activities are often conducted by research centers of private firms. Applied research and commercial research from the OECD's categorization count as development activities.
To rephrase these descriptions, research and development are creative works that are expected to add to sophistication, quality, and practicality of the current set of knowledges. What distinguishes research from development is that the former will be the underlying foundation of the latter, offering advanced knowledge. On the other hand, achievements from development can provide feedback for improvements, requiring further research.
Figure 1: Research and development
FEEDBACK (ROOM FOR IMPROVEMENTS)
Source: Author's original image.
Means of measuring R&D performance
On the other hand, performance of research and development activities is a rather complex indicator that can be measured in multiple ways. Its definition and means of measurement depend on the purpose, scope and depth (complexity) of an individual R&D activity. R&D performances may generally be categorized as: outputs, outcomes, and impacts. Output is "a return to expenditure in a rather direct and preliminary form" (Jang, 2003) including, but not limited to, research papers and patents. Outcome consists of actual benefits a service provider offers to its customers; these benefits may include total sales, market share, cost reduction, et cetera (Jang, 2003). Impact is another measure of research and development activities, and it may be measured by a GDP growth rate, value added of industry and other indices. Kim et al. (2013) defines impact as "originally unintended by researchers but affects human life in general". These alternatives to measuring R&D performance may lead to arguments over which indicators best represent effects of research and development activities.
A few existing research studies have presented ideas on measurement of R&D performance. According to Lundvall (2010), effective and efficient production-diffusion-application process of economically profitable knowledge is an appropriate measure of domestic innovation. Previous researches have usually measured the ratio of total R&D expenditure to gross domestic product (GDP), but its major drawback is unbalanced emphases on inputs and outputs (greater emphasis on inputs): R&D expenditure may not play the biggest role in the innovation process. Griffith et al. (2001) investigated a causal relationship between TFP (Total Factor Productivity) and OECD's BERD (Business Expenditure on Research and Development) indicator for respective countries. In this research, TFP was a measure of R&D performance. Hall et al. (2010) in a UNU-Merit working paper went through a comprehensive literature review on R&D and discovered social spillover of private R&D activities. These authors insist that output from a conventional productivity equation (output/input) can be measured by total output, total value added, or total sales; however, they argue that the total value added may be the most appropriate measurement of R&D performance (Hall et al, 2010). Total value added is an output of two types of inputs - labor and capital - minus intermediate products produced by other producers; in contrast, total output includes all intermediate goods, and thus is not suitable for inter-firm comparison mainly because of different vertical integration of respective firms.
Accordingly, several researches incorporate total value added as a performance indicator of R&D. Soltmann et al. (2015) constructed a panel data of 12 OECD countries to test causal relationships between industry-specific environment-friendly research and development activities and each country's total value added. Woo (2013) also conducted a panel data analysis on 12 OECD countries to explore the effect of intellectual property rights and patents on industry-level value added.
While there are many researches that propose patents as a key indicator of R&D performance, patents may also be considered as an intermediate product of final products. This is because patents do not necessarily need to be realized in an actual tangible product. The World Intellectual Property Organiza-
tion (WIPO) defines patent as "an exclusive right granted for invention, which is a product or a process that provides, in general, a new way of doing something, or offers a new technical solution to a problem" (WIPO website, 2018). A patent is just a set of new ideas that may or may not have been put into a final product, yet it gives a legal right to the entity who first filed the patent. Therefore, patents may not fully represent the R&D performance.
There also are a few other researches that assess effectiveness of technological transfer. While technological transfer may not always be part of research and development activities, Bozeman's (2000) research hints at the measurement of effectiveness of technology transfer. Bozeman assesses effectiveness in terms of transfer's impact on firms' sales and profitability, regional economic development, political benefits, opportunity costs of resources, and capacity of scientific and technical human capital (Bozeman, 2000, p. 645). These references suggest that R&D effectiveness or performance may be measured by economic benefits that involve economic development and human capital.
While not every researcher would agree on which measure best represents research and development performance, it is at least reasonable to state that value added is a final economic product of research and development activities. Patent and technology transfer, proposed in some of the literature as an indicator of R&D performance, is rather an intermediate than final product. There may also be other performance indicators that are not economic, such as the number of Nobel Prizes in a country. This paper will emphasize economic aspects of R&D performance that are considered more conventional and available for international comparison.
Effects of government investment on R&D performance
The previous section about measuring R&D performance hinted at a potential involvement of economics of production. A universal and conventional yet simple production function from economics defines the fundamental relationship of inputs and outputs in production of goods and services.
7 = A * f(L, K) (1)
Y, or yield, in the Equation (1) may represent the output of an R&D activity. L stands for labor, and K stands for capital; both could be inputs of an R&D activity. Lastly, A is technology involved in the R&D activity. Most economic theories generally define labor as unskilled labor force, and capital as dollars used for acquiring physical capital (e.g. production facilities). This section reviews the public sector's role in advancing technology to maximize products out of labor and capital.
Regarding the government's role as a source of capital investments, a number of academic papers have presented the relationship between public spending and economic growth. In R&D-themed papers, different authors claim that government support instigates a firm's innovation activities (Yoon, 2006; Guellec & Pot-terie, 2003; Jung, 2013). Some research concluded that (both private and public) firms' expenditures on research and development lead to an upward convergence
of productivity at the industry level, through innovation and technology transfer within industry (Griffith et al, 2001). Ko et al. (2015) and Song et al. (2009) found out that governments provide subsidies and tax cuts to private firms in order to promote research and development.
Kim (2006) estimated a causal effect of innovation activities on trades and economic growth. Innovation activities such as investing in research and development improves productivity of respective industries, differentiates products and thereby promotes exports in a skilled labor-intensive high-tech division. On the other hand, Shin (2004) discovered a causal effect of basic research on R&D investments. Specifically, Shin's research shows that the Korean government's expenditure on basic research potentially relates to firms' intentions to invest in research and development.
Figure 2: Technology, labor, capital and productivity
Source: Author's original image.
However, most of these authors focused on Korean cases, leaving room for international comparison regarding the matter. Moreover, many articles talk about R&D in general rather than pinpoint different sectors (i.e. public sector, private sector and higher education) that may contribute to and perform on R&D investments. A few studies have conducted cross-country comparisons on public R&D expenditure and performance. Cho et al. (2005) analyzed 1970-2004 panel data on OECD countries to explore investment efficiency of numerous inputs toward R&D outputs. In particular, inputs included the ratio of government expenditure in total domestic R&D expenditure, government support for industry-higher education collaboration and commercial application of developments. Cho et al. conclude their research with two findings: there are positive relationships between research and development performance (measured by number of filed patents) and ratio of public R&D expenditure; efficiency improvement under increasing government support to industry-education collaboration and commercial application and densely populated human capital (2005). These evidences highlight the significance of government expenditure, while authors fail to argue for a positive relationship between human capital (and/or knowledge basis) and productivity.
Goolsbee (1998) argues against a popular claim that American public sector investments promote research and development in general. In reality, increases in public R&D expenditures are mainly partitioned as labor force salaries; on the other hand, actual research and development capability turned out to be inelastic to the total amount of government R&D investments (Goolsbee, 1998). This may imply a negative relationship between government-funded R&D investments and a will to do R&D, for both firms and researchers. The author also critiques the nature of the R&D sector that only a narrow span of labor force is eligible for such activities; this is due to the importance of skills and specialty in science and technology research (Goolsbee, 1998). It thus comes as natural for government funds to be allocated to a limited number of people and it may be regulatory.
Wallsten (2000) conducted empirical research on the U.S. Federal Small Business Innovation Research Program. Findings show that the positive relationship between fiscal R&D expenditure dedicated to private sector firms and their additional investments in patents, employment and general R&D activities is statistically insignificant. The study claims that government funds may cause firms to potentially reduce usage of their own funds dedicated to research and development; the total amount of R&D activities may stay the same in the end (Wallsten, 2000, p. 98). Similarly, Mamuneas and Nadiri (1996) showed that public R&D funds do not substitute firm's own R&D funds in R&D-centric industries. In R&D-cen-tric industries, an increase in government R&D funding does not negatively affect firms' research and development; in contrast, industries that put less emphasis on R&D experience reduced R&D activities in the identical condition (Mamuneas & Nadiri, 1996). However, less R&D activities contributed to more efficient production cost-wise.
In addition, some of the previous literature has presented potential positive spillover effects of government R&D funding over business R&D efforts (Levy & Terleckyj, 1983; Hu, 2001). Interestingly, while many research papers have analyzed the effectiveness of public R&D funding and private R&D funding on private sector performance (Archibald & Pereira, 2003), we have not been able to find literature that has compared the public sector and private sector performances when the government provides R&D funding. It would be meaningful to compare effectiveness of government funding on public and private sectors, because governments often operate research and development projects on their own, while still subsidizing and promoting private projects.
Literature reviews on effects of government R&D investment show both positive and negative aspects of public investment in innovative activities. It is important to discover which factors may interfere with public investment despite trending increases worldwide. The next section discusses one potential factor that may negatively affect R&D performance under government investment.
Effects of government regulation on R&D performance
Along with various inputs that lead to R&D performance, government regulations may play a significant role in developing certain economic values. Lundvall (2010) have proposed strict budget auditing procedures as an explanatory variable of research and development actions. As government funds towards R&D policies
become a priority, more money will come, and more budget reviews and audits will follow. Then these strict reviews and audits may affect research and development activities.
Regulation in many cases refers to "implementation of rules by public authorities and governmental bodies to influence market activity and behaviour of private sectors in the economy" (Blind, 2012, p. 392). Regulations are often created to deal with certain problems associated with a market-driven economy. According to Veljanovski (2010), an efficient outcome is possible when inputs such as "resources, goods, and services" are arranged so that whatever available technology is used most productively (20). However, asymmetric information between the seller and the buyer of a product may cause a Pareto-inefficient allocation of different resources. Due to this information asymmetry, many transactions are susceptible to moral hazard and adverse selection by actors from either side. This is often considered a market failure to be cured by regulation, and resources are supposed to be reallocated based on each actor's need for welfare.
Therefore, most government programs involve some forms of regulation. As long as these programs involve direct financial transfers from the government, they usually come at certain costs. Even social security programs require paperwork and time from service recipients. Government funding to R&D is exposed to even more government intervention, because the government may favor or control advanced technology in select areas. Baldwin, Cave and Lodge (2010) claim that "regulation has become a central feature not just in the debates regarding the control of new or changing technologies, but also in the context of new technologies that change the frontiers of existing regulatory regimes" (6). Government funding of research and development projects are usually need-based, implying that public officials will have to investigate whether submitted applications comply with requirements. They will not simply give away free money that recipients may spend inappropriately. Even before debating about regulations promoting or discouraging new technology, government-subsidized R&D projects are exposed to potentially burdensome orders.
Regulations on research and development may be categorized into different types. Different authors have suggested economic regulation on innovation, social regulation on innovation, and institutional regulation on innovation (Blind, 2012, pp. 393-394). Economic regulation includes, but is not limited to, "competition policies, price regulation, market entry regulations, and the regulation of natural monopolies and public utilities" (Blind, 2012, p. 394). The second type, social regulation, may be exemplified by environmental regulations.
Another type of regulation is institutional, which may be "implemented by administrative regulations... [with link to the] legal framework to innovation" (Blind, 2012, p. 394). While the institutional regulation brought up by Blind (2012) focuses on the liability issue regarding the new innovative product, it does present a significant implication for further research. Innovators are often afraid of safety issues from their new product, and thus they do not risk introducing it. Institutional regulation may also be applied in cases where too much restriction on budget allocation requires an excessive amount of paperwork and proof of expenditures.
While government regulations aim at adjusting market inefficiency, they also cause another inefficiency. Moynihan et al. (2015) coined a more concrete form of government regulation: the "administrative burden" in the context "in which the state regulates private behavior or structures how individuals seek public services" (44). It is defined as "an individual's experience of policy implementation as onerous" (Burden et al., 2012; Moynihan et al., 2015). Administrative burden is thus a form of regulatory costs involved in receiving public services, consisting of three different types of costs: learning costs, psychological costs, and compliance costs.
As Moynihan et al. (2015) builds on prior works, learning costs mainly deal with citizens learning about their eligibility and how to access certain public services. Then psychological costs arise when people are no longer autonomous as they participate in a new public service. Lastly, compliance costs come by when doing paperwork and providing documentation as required by a service provider. These costs may exist in any form of public services provided by the government, including R&D expenditures funded by the government. Among these three types of costs, compliance costs have been identified to reduce citizens' participation in government programs as shown in several empirical researches (Bartlett, Burstein & Hamilton, 2004; Brien & Swann, 1999; Wolfe & Scrivner, 2005; Moynihan et al., 2015).
The simple production model previously mentioned involves component A, which may include any technology-related variables affecting input-output ratio. With a certain amount of technological improvements, the same amount of inputs may result in larger outputs. However, there may or may not be a room for a government regulatory environment around technological development and application. As far as countries differ in regulatory environments, similar levels of inputs may lead to dissimilar R&D performances. Blind (2012) discusses the impact of legal and regulatory framework on firms' innovation: the author differentiates economic, social and institutional regulations - economic regulations are mainly about product market competition and price controls; social regulations are about environmental compliance; and institutional regulations mostly deal with intellectual property rights and general legal and regulatory framework (Blind, 2012, p. 396). According to Im (2015), some governments look for more regulation while other governments such as that of the United States depend on more innovation. In addition, tools of regulation may be different from government to government.
The research by Cho et al. (2005) brings up this issue at the end. Among different policy implications, authors specifically mention that government regulations in government-funded research and development projects potentially deter R&D performance from significant improvements. For example, administrative tasks for fulfilling regulatory requirements take a certain amount of time. However, too many of these tasks could negatively affect research outputs and thus make further application obsolete. Regarding this matter, Nicoletti & Scar-petta (2003) have provided empirical evidence of excessive European regulation resulting in low entrepreneurial productivity. Their result implies that too much regulation keeps firms from adopting breakthrough technology and adapting
to changing business environments; in the long-run, not only production outputs but also other output variables including sales may be negatively affected (Nicoletti & Scarpetta, 2003).
Intellectual properties
Basberg (1987) addresses a few questions regarding the significance of patent information as a representation of innovative activities. The author's questions include whether patent information is valuable as an indicator of technological change and whether patent data is universal and comparable across industries and countries. Not all innovations or inventions are patented, so intellectual properties may or may not be a sole indicator of technological changes. However, innovations that are patented and actually produced into a commercial product may contribute to value added in the economy. Among the patented ideas, important major innovations are likely to be patented in multiple countries, assuring cross-referencing across countries (Basberg, 1987, p. 136). This suggests that the number of triadic family patents, including those patented in the United States, Europe and Japan, may count as an important indicator of innovative activities.
Lanjouw & Schankerman (2004) points out that research productivity may be measured by the ratio of number of patents to R&D investments. Using the U.S. patent data, the research assesses potential determinants of research productivity, including patent quality index represented by the size of the patent family, number of citations and number of patent claims. Claims describe novel features of an invention, while the number of citations demonstrate how much new inventors rely on a particular patent (Lanjouw & Schankerman, 2004, p. 446). Aside from these two factors, the size of the patent family is an important part of the patent quality because not all patents gain equal presence by patent offices in the U.S., Europe and Japan. The family size of a patent illustrates the number of patent markets that protect a patent. Triadic families of patents generally means that an invention is protected by the U.S., European and Japanese patent offices (Dernis, 2003). Thus the number of triadic patent families is a viable international measure of technological change and innovation.
In addition to previous discussions on patents, Dosi et al.'s (2015) comparative study at both firm-level and country-level shows that a technology gap results in a gap in total amount of exports. Patent filing and new technology through research and development, unlike cost reduction in labor and other input factors, have a positive causal relationship to a country's and firm's level of exports (Dosi et al., 2015, pp. 1801-1809). Unlike labor-intensive industries from the past, a modern industrial system values promoting a country's innovation and competitiveness through technology developments based on human capital; therefore, cutting labor costs and other preliminary costs does not improve a country's competitiveness. In the end, a simple expansion of capital investments and expenditures do not solely guarantee positive R&D performance improvement. Rather, these factors should be accompanied by a continuum of knowledge accrual.
Lastly, Popp (2005) emphasizes the reason a number of patents may be an appropriate measure of research and development activity. While he notes that not all successful innovations may be patented, patents provide "a detailed record
of each invention" while R&D expenditure data only offer aggregate information (Popp, 2005, p. 3). Also, the fact that patent data are available in many different countries helps researchers compare R&D activities across countries. Therefore, because patent data is widely available and comparable while offering information about research and development activities, it is important to consider it as part of this research. However, as mentioned in one of previous sections in this paper, patents are rather intermediate than final products of research and development activities. Patents do pave new ways for developing and producing actual products, but actual products do not always happen. Patents are more innovative ideas than tangible output.
Human capital and R&D performance
As previously mentioned, R&D investment or expenditure is not the only major contributor to R&D performance. Human capital is another major factor that is increasingly important. A large pool of researchers can continuously suggest new ideas and form a growing knowledge base. In the process of knowledge growth or knowledge accrual, products of R&D may occur at certain points. Therefore, knowledge base naturally accompanies human capital when we analyze causal factors of research and development performance.
O'Mahony & Vecchi (2009) analyzed an entrepreneurial panel data of five OECD countries and stated that knowledge-based and human capital-centric industries are prone to show direct positive effects of monetary investments on research and development achievements. If this is the case, a spillover from research and development from only a part of the industry can improve productivity of the entire industry (O'Mahony & Vecchi, 2009, p. 42). If this explanation extends to the country level, a firm's research and development may indirectly improve productivity of other industries and the whole country's productivity as well. It is thus important to include human capital as part of the research model assessing factors that contribute to R&D performance of a country.
Research Design
Variables of interest and research model
Based on the literature review of factors that may contribute to R&D performance, most R&D's consist of R&D related capital and human capital investments. Factors that have been brought up in the previous chapter are: government investment, regulatory burden, intellectual properties (patents) and human capital. The research is mainly targeted at analyzing the significance of government investment on R&D and regulatory burden in the national R&D performance, measured by the value added of industry. Thus, all other variables that have been identified as potential factors of R&D performance will be control variables. It is important to consider the significant gaps among these 36 countries. The gap may include, but not limited to, economic gap, technology gap, and human capital gap. For example, countries that are OECD members are not necessarily at the same stage of knowledge base development or previous achievements on R&D projects. Control variables will take these gaps into account.
Specifically, government investment may be divided into government expenditure of the investment, business expenditure of investment, and other expenditures of investment4. In other words, this research will assess government as a source of funding (investment) and as a performer (expend) of R&D activities. It will also assess business enterprises as a performer of R&D activities, funded by the government. These activities are represented by the following indicators: Government-performed R&D expenditure of government funding on R&D and Business-performed R&D expenditure of government funding on R&D. Government investments on Control variables include the gross expenditure on R&D, number of triadic patent families, number of researchers, gross domestic product and population, all of which are counted at national level. Figure 3 demonstrates the proposed research model.
Figure 3: Research Design
Independent variable
Government-Performed R&D Expenditure of Government Funding on R&D (gov_gov)
Business-Performed R&D Expenditure of Government Funding on R&D (bus_gov)
Regulatory burden (regul_burden)
Control variables
Gross expenditure on R&D (gross_exp) Triadic patent families (patent) Number of researchers (researcher) Gross Domestic Product (gdp) Population (pop)
□
Dependent variable
Value added of industry
(value_added)
value_ added = a + f1 * gov_gov + f2 * bus_gov + f3 * regul_burden (2)
Equation (2) demonstrates a linear equation involving value added of industry, government performance of government funding on R&D, business performance of government funding on R&D and regulatory burden. The extended model, presented in Equation (3), incorporates the following control variables: gross expenditure on R&D, number of triadic patent families, number of researchers, gross domestic product (GDP) and population.
value_added = a + f1 * gov_gov + f2 * bus_gov + f3 * regul_burden + f4 * gross_exp + f5 * patent + f6 * researcher + f7 * gdp + f8 * pop (3)
4 Other expenditures of R&D investment include those by the higher education sector and private non-profit sector. These other types of expenditures are not variables of main interest, so they will be controlled in part by a gross domestic expenditure on R&D.
Research hypotheses
The first hypothesis will test the potential positive relationship between total value added of industry and the government-performed expenditure of government funding on research and development. Whether a government is good at spending its own money on R&D will be assessed through this test.
Hypothesis 1: An increase in government-performed expenditure of government-funding on R&D leads to an increase in total value added of industry.
The second hypothesis will test the potential positive relationship between total value added of industry and the business-performed expenditure of government funding on research on development. Unlike the first hypothesis, non-government performance of the same source of funding on R&D will be assessed.
Hypothesis 2: An increase in business-performed expenditure of government-funding on R&D leads to an increase in total value added of industry.
The last hypothesis will test whether regulatory burden has a significant negative relationship with the total value added of industry. Whether less regulatory burden would result in more value added will be discovered through this test.
Hypothesis 3: Less regulatory burden leads to an increase in total value added of industry.
Data and research methods
The main source of data is OECD's Main Science and Technology Indicators (MSTI). Indicators include, but are not limited to, R&D-related variables such as number of total researchers, gross domestic expenditure on R&D (GERD), government-performed expenditure on R&D (GOVERD), business-performed expenditure on R&D (BERD), and other various variables related to research and development plus science and technology. The dataset is an annual assortment of data from 35 OECD countries (as of 20 1 65) and 1 non-OECD country (Russia).
The authors established a panel dataset of 36 countries over 17 years (2000 to 2016), but the missing data came up as a problem. Normally, when data is missing for certain variables and for certain countries, a computer statistical package like STATA eliminates that item in the analysis. Some researchers prefer to replace missing data with mean values if they do not want to eliminate any item. However, substituting missing values with their respective mean values may create biases in variables. Instead, STATA supports missing data imputation, which is based on a random yet simulated projection of missing data. This research utilizes imputation methods in order to keep valid information useful, instead of deleting items that miss any one data.
5 As of 2016 (the final year for the dataset), there are 35 OECD countries: Australia, Austria, Belgium, Canada, Chile, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Israel, Italy, Japan, Korea, Latvia, Luxembourg, Mexico, Netherlands, New Zealand, Norway, Poland, Portugal, Slovak Republic, Slovenia, Spain, Sweden, Switzerland, Turkey, United Kingdom, and United States.
The panel data include original data from OECD MSTI for following indicators:
- Value added of industry (in million US Dollars, adjusted at 2015 prices).
• Definition: Value of output minus value of intermediate consumption, for industry divisions. It excludes "real estate activities" or "imputed rent of owner-occupied dwellings. that represents a significant share of total GVA [Gross Value Added] and has no R&D counterpart (OECD, 2015).
- Expenditure on R&D by sector of performance and source of funds (in million US Dollars, adjusted at 2015 prices).
• Sectors: Government and Business enterprise.
• Source of funds: Government.
• Definitions:
a) Government-performed expenditure of government funding on R&D: gross amount of expenditure on R&D, spent by the government and funded by the government;
b) Business-performed expenditure of government funding on R&D: gross amount of expenditure on R&D, spent by business enterprises and funded by the government.
- Gross domestic expenditure R&D (in million US Dollars, adjusted at 2015 prices).
• Definition: all expenditures on R&D domestically.
- Number of triadic patent families.
• Definition: a set of patents "protecting a single invention across various jurisdictions"; triadic patent families apply if and only if a patent is filed at the European Patent Office (EPO), the Japan Patent Office (JPO), and the US Patent & Trademark Office (USPTO) (OECD, 2015).
- Total [number of] researchers (FTE: full-time employed).
• Definition: R&D personnel within a national territory, counted by the number of full-time workers.
- Gross domestic product (GDP).
- Population.
Additionally, the 'regulatory burden' data have been adopted from the World Governance Indicators. The original indicator is the burden of government regulation, from a survey question asking, "In your country, how burdensome is it for businesses to comply with governmental administrative requirements (e.g., permits, regulations, reporting)?"6. This indicator measures the government's ability to support and regulate the private sector's intention to do research and development. It is also a combination of different variables that are not disclosed in public. The original data ranges from 1=extremely burdensome to 7=not burdensome at all, requiring reverse measurements for intuitive analyses. Therefore the reversed measurements for regulatory burden now range from 1=not burdensome at all to 7=extremely burdensome to represent higher numbers for higher perceived regulatory burdens.
With all these variables in the panel data, STATA computer statistical package software has been used to conduct a panel regression analysis. Specifically, a random effect GLS regression with multiple-imputation estimates has been conducted.
6 Survey question retrieved from World Bank TCdata360, burden of government regulation.
Table 3
Descriptive statistics: initial data without multiple imputation
Variable Obs. # Mean Std. Dev. Min Max
Country_ID 612 18.5 10.39679 1 36
Year 612 2008 4.902987 2000 2016
Value_added 612 800300.3 1598085 5217.85 1.20e+07
Gov_gov 528 3824.992 9235.367 10.552 58278.41
Bus_gov 523 1928.183 5548.354 0.889 43834
Regul_burden 513 3.729313 0.6715117 1.7 5.09942
Gross_exp 532 27770.62 68652.5 78.92 502893
Patent 576 1473.061 3795.079 0.42 18703.1
Researcher 495 136610.2 242539.7 1645.6 1400000
Gdp 612 1219632 2483644 8294.96 1.90e+07
Pop 612 37944.1 58330.66 281.15 323926
Note: Value_added: value added of industry; gov_gov: gross amount of expenditure on R&D, spent by the government and funded by the government; bus_gov: gross amount of expenditure on R&D, spent by the business enterprises and funded by the government; regul_burden: regulatory burden; gross_exp: gross domestic expenditure on R&D; patent: number of triadic patent families; researcher: total number of researchers (full-time equivalent); gdp: gross domestic product; pop: population.
Sources: OECD Main Science and Technology Indicators & World Bank World Governance Indicators.
Variables in bold type have missing data, thus requiring multiple imputation of missing values. Multiple-imputation random effect GLS regression as well as pooled OLS regression will be conducted for comparison.
Results
Results from pooled OLS regression without imputation
We first ran pooled OLS regression on the initial data with missing values. Results build following inferences: it is statistically significant at the 95% confidence level that the increase in government-performed research and development funded by the government is associated with a decrease in the value added of industry. It is also statistically significant that the increase in the business-performed research and development funded by the government is associated with an increase in the value added of industry. Regulatory burden, of which higher numbers represent higher perceived burdens from regulation, shows a negative relationship with the value added of industry. In other words, less perceived regulatory burden is positively associated with an increase in the value added. Therefore, Hypothesis 1 has not been endorsed, while Hypotheses 2 and 3 have been endorsed.
Table 4
Correlation of variables
Value _added Gov _gov Bus _gov Regul _burden Gross _exp Patent Researcher Gdp Pop
Value _added
Gov _gov 0.9825
Bus _gov 0.9112 0.9314
Regul -burden 0.0011 -0.0210 0.0062
Gross _exp 0.9814 0.9815 0.8841 -0.0622
Patent 0.7913 0.7751 0.5684 -0.0709 0.8217
Researcher 0.9649 0.9486 0.8680 -0.0099 0.9558 0.8626
Gdp 0.9982 0.9850 0.9184 0.0015 0.9823 0.7700 0.9569
Pop 0.9569 0.9241 0.8770 0.1010 0.8967 0.7510 0.9391 0.9484
Note: Value_added: value added of industry; gov_gov: gross amount of expenditure on R&D, spent bythe government and funded by the government; bus_gov: gross amount of expenditure on R&D, spent by the business enterprises and funded by the government; regul_burden: regulatory burden; gross_exp: gross domestic expenditure on R&D; patent: number of triadic patent families; researcher: total number of researchers (full-time equivalent); gdp: gross domestic product; pop: population.
Source: Authors' calculations.
Table 5
Pooled OLS regression results, with missing values
Value_added Coef. t P>|t| [95% Conf. Interval]
Gov -gov -16.65429 (2.842802 -5.86 0.000*** -22.24388 -11.06469
Bus -gov 10.24652 (2.457726) 4.17 0.000*** 5.414069 15.07896
Regul -burden -13638.71 (5399.89) -2.53 0.012*** -24256.11 -3021.299
Gross -exp 2.488614 (0.4997892) 4.98 0.000*** 1.505915 3.471313
Patent 22.90567 (2.826763) 8.10 0.000*** 17.34851 28.46462
Researcher -0.1704345 (0.0780332) -2.18 0.030*** -0.3238654 -0.0170035
Value_added Coef. t P>|t| [95% Conf. Interval]
Gdp 0.5211972 (0.0150594) 34.61 0.000*** 0.4915871 0.5508073
Pop 3.646355 (0.2987214) 12.21 0.000*** 3.059001 4.233709
_cons -55323.9 (18708.65) -2.96 0.003 -92109.33 -18538.46
# of obs F(8,380) Prob > F R-squared Adj R-sq Root MSE 389 36763.86 0.0000 0.9987 0.9987 65201
Note: Standard errors are presented in parentheses. * indicates significance of coefficients at the 90% level (p<0.10). ** indicates significance of coefficients at the 95% level (p<0.05). *** indicates significance of coefficients at the 99% level (p<0.01).
Value_added: value added of industry; gov_gov: gross amount of expenditure on R&D, spent by the government and funded by the government; bus_gov: gross amount of expenditure on R&D, spent by the business enterprises and funded by the government; regul_burden: regulatory burden; gross_exp: gross domestic expenditure on R&D; patent: number of triadic patent families; researcher: total number of researchers (full-time equivalent); gdp: gross domestic product; pop: population.
Source: Authors' calculations.
Results from panel data regression with multiple imputation
The result of random effects GLS regression on the panel data (after multiple imputation of missing values) is as follows:
Table 6
Random effects GLS regression on panel data, with multiple imputation of data
Value _added Coef. t P>|t| [95% Conf. Interval]
Gov _gov -10.79665 (2.454329) -4.40 0.000*** -15.60964 -5.983658
Bus _gov 5.728293 (1.986161) 2.88 0.004*** 1.834961 9.621625
Regul _burden -11286.02 (5221.305) -2.16 0.032*** -21594.96 -977.0843
Gross _exp 0.5078297 (0.4118732) 1.23 0.218 -0.3012625 1.316922
Patent 23.51355 (2.631214) 8.94 0.000*** 18.35093 28.67618
Researcher -0.2171915 (0.0738897) -2.94 0.003*** -0.3623621 -0.0720208
Value _added Coef. t P>|t| [95% Conf. Interval]
Gdp 0.5974206 (0.0113243) 52.76 0.000*** 0.5751979 0.6196434
Pop 2.265917 (0.3081016) 7.35 0.000*** 1.661116 2.870718
_cons -45850.06 (19220.83) -2.39 0.018 -83721.38 -7978.747
sigma_u sigma_e rho 32231.77 50186.866 0.29201835 (fraction of variance due to u_i)
Note: Standard errors are presented in parentheses. * indicates significance of coefficients at the 90% level (p<0.10). ** indicates significance of coefficients at the 95% level (p<0.05). *** indicates significance of coefficients at the 99% level (p<0.01).
Value_added: value added of industry; gov_gov: gross amount of expenditure on R&D, spent by the government and funded by the government; bus_gov: gross amount of expenditure on R&D, spent by the business enterprises and funded by the government; regul_burden: regulatory burden; gross_exp: gross domestic expenditure on R&D; patent: number of triadic patent families; researcher: total number of researchers (full-time equivalent); gdp: gross domestic product; pop: population.
Source: Authors' calculations.
Based on the panel regression GLS model, the following statistical inferences have been made7. It is statistically significant at the 95% confidence level that the increase in government-performed research and development funded by the government is associated with a decrease in the value added of industry. It is also statistically significant that the increase in the business-performed research and development funded by the government is associated with an increase in the value added of industry. Regulatory burden, of which higher numbers represent higher perceived burdens from regulation, shows a negative relationship with the value added of industry. To make it simple, less perceived regulatory burden is positively associated with an increase in the value added.
In addition, control variables, while they are not of main interest, show certain directions of relationships with the dependent variable. First of all, the gross expenditure on research and development that encompasses all the national expenditure regardless of funding has not shown a statistically significant relationship with the value added of industry. This is a departure from the previous pooled-OLS regression analysis, in which statistical significance has been discovered. Second, the number of triadic patent families that include U.S. patents, EU patents and Japan patents shows a statistically significant positive relationship with the dependent variable. On the other hand, the number of researchers that are full-time equivalent has shown a statistically significant negative relationship with the value added of industry. The gross domestic product and the population both have shown positive relationships with the dependent variable, both of which are statistically significant.
7 No lagged dependent variable has been included as controls in this statistical inference.
Interpretation
The regression results provide us with some meaningful interpretations. While funding of research and development has for decades been considered a crucial investment of governments, what really matters is who uses the money. Governments may have acted as a main source of R&D investment, but they may not have been effective in spending on their own. In contrast, businesses, or the private sector, seem to have been more successful in generating value added by spending on research and development, funded by governments. In addition, countries where people perceive regulations to be of less burden perform better in generating value added of industry.
Conclusion
The main purpose of this research was to address factors that may affect national performance of research and development. Since governments are major funding sources of research and development in many countries, government competitiveness of making the most out of public investment should be a critical issue. However, most previous research has tried to demonstrate relationships between R&D spending and R&D performance of individual firms, or relationships between public R&D expenditure and the national performance on R&D in narrower perspectives. Therefore, this research aimed at a multi-faceted analysis of government investment in research and development, with particular interests in government performance, business performance, and regulatory burdens.
From the 17-year panel data regression analysis on 35 OECD countries and Russia, the R&D performance measured by value added of industry was discovered to be negatively related with the government-performed R&D funded by the government. On the other hand, it was positively related with business-performed R&D funded by the government and less regulatory burdens. These results offer some meaningful interpretations. Despite governments' crucial role in funding R&D investments to different sectors, its spending for its own R&D projects have not been found to generate more value added than private enterprises' spending on R&D. This suggests that the government may be a decent source of funding R&D projects, but it may not be as effective as business enterprises in generating economic values. These findings are followed by the discovery that less regulatory burdens (proxy measured by perceptions on regulatory burdens) would increase the value added.
To sum up, a competitive government research and development investment framework may involve less government spending, more private business spending, and less regulatory burden. Governments may be effective in financially supporting research and development projects, but it would be even more effective to let private industries use this financial resource to conduct research. In addition, regulations, which could range from burdensome paperwork to major accounting audits, could be reduced for better performance on research and development. It is important to provide enough support while removing excessive obstacles to research activities.
REFERENCES
1. Amable, B., Ledezma, I. & Robin, S. (2016). Product Market Regulation, Innovation, and Productivity. Research Policy, no 45, pp. 2087-2104.
2. Archibald, R.B. & Pereira, A.M. (2003). Effects of Public and Private R&D on Private-Sector Performance in the United States. Public Finance Review, vol. 31, no 4, pp. 429-451.
3. Arora, A. & Gambardella, A. (1990). Complementarity and External Linkages: The Strategies of the Large Firms in Biotechnology. The Journal of Industrial Economics, vol. 38, no 4 (June), pp. 361-379.
4. Basberg, B.L. (1987). Patents and the Measurement of Technological Change: A Survey of the Literature. Research Policy, vol. 16, no 2-4, pp. 131-141.
5. Blind, K. (2012). The Influence of Regulations on Innovation: A Quantitative Assessment for OECD Countries. Research Policy, vol. 41, no 2, pp. 391-400.
6. Blind, K., Petersen, S.S. & Riillo, C.A.F. (2017). The Impact of Standards and Regulation on Innovation in Certain Markets. Research Policy, no 46, pp. 249-264.
7. Bozeman, B. (2000). Technology Transfer and Public Policy: A Review of Research and Theory. Research Policy, no 29, pp. 627-655.
8. Brown, G.M. & Svenson, R.A. (1988). Measuring R&D Productivity. Research Technology Management, vol. 31, no 4, pp. 11-153.
9. Cho, Y., Kim, W., Nam, J. & Oh, J. (2005). Policy Directions to Improve the Efficiency of the R&D Investment for Reinforcing the Innovative Capability. Seoul: Korea Institute for Industrial Economics & Trade (in Korean).
10. Chung, E., Lee, K. & Choi, M. (2013). The Structural Relationship between R&D Resources and Innovation Performance in Manufacturing Industry: With a Special Emphasis on Internal R&D Capability, External R&D Collaboration, and Governmental Support. POSRI Business Review, vol. 13, no 1, pp. 100-124 (in Korean).
11. Cozzarin, B.P. (2008). Data and the Measurement of R&D Program Impacts. Evaluation and Program Planning, vol. 31, no 3, pp. 284-298.
12. Dernis, H. (2003). OECD Triadic Patent Families: OECD Methodology: An Overview. WIPO-OECD Workshop on Statistics in the Patent Field.
13. Dosi, G., Grazzi, M. & Moschella, D. (2015). Technology and Costs in International Competitiveness: From Countries and Sectors to Firms. Research Policy, no 44, pp. 1795-1814.
14. Frenz, M. & Ietto, G.G. (2007). Does Multinationality Affect the Propensity to Innovate? An Analysis of the Third UK Community Innovation Survey. International Review of Applied Economics, vol. 21, no 1, pp. 99-117.
15. Goolsbee, A. (1998). Does Government R&D Policy Mainly Benefit Scientists and Engineers? NBER Working Paper, No. 6532.
16. Griffith, R., Redding, S. & Van Reenen, J. (2001). Mapping the Two Faces of R&D: Productivity Growth in a Panel of OECD Industries. The Institute for Fiscal Studies Working Paper.
17. Guellec, D. & Van Pottelsberghe de la Potterie, B. (2003). The Impact of Public R&D Expenditure on Business R&D. Economics of Innovation and New Technology, vol. 12, no 3, pp. 225-243.
18. Hall, B.H., Mairesse, J. & Mohnen, P. (2010). Measuring the Returns to R&D. UNU-MERIT Working Papers, ISSN 1871-9872.
19. Ho, A.T.K. & Im, T. (2012). Defining a New Concept of Government Competitiveness. The Korean Journal of Public Administration, vol. 50, no 3, pp. 1-34.
20. Hu, A.G. (2001). Ownership, Government R&D, Private R&D, and Productivity in Chinese Industry. Journal of Comparative Economics, no 29, pp. 136-157.
21. Im, T. (2014). The Goal and Direction of Park Chung-Hee Administration. Major Policies and Governance of Korean Administration. Edited by the Korea Institute of Public Administration. Seoul, Korea: Daeyoungmunhwasa (in Korean).
22. Im, T. (2015). Why is the Korean Government the 16th? 2015 Government Competitiveness Report. Goyang, Korea: Moonwoosa (in Korean).
23. Im, T. & Hartley, K. (2019). Aligning Needs and Capacities to Boost Government Competitiveness. Public Organization Review, no 19, pp. 119-137.
24. Irwin, A. & Vergragt, P. (1989). Rethinking the Relationship Between Environmental Regulation and Industrial Innovation: The Social Negotiation of Technological Change. Technology Analysis & Strategic Management, vol. 1, no 1, pp. 57-70.
25. Kim, B. (2006). R&D Activities and International Trade: A Study on the Effect R&D to Exports in Korean Economy. Seoul, Korea: Science and Technology Policy Institute (in Korean).
26. Kim, H., Yoo, J. & Ryu, J. (2013). A Study on the Relationship Between Cooperative Factors of National R&D Projects and Performance: Focusing on the Moderating Effect of Projects' Characteristics. The Korean Journal of Business Administration, vol. 26, no 3, pp. 695-718 (in Korean).
27. Kim, Y. (2015). An Analysis of Factors Leading into the Failed Commercialization After the Technology Transfer from Public R&D to Private Enterprises. Seoul, Korea: Korea Institute of S&T Evaluation and Planning (in Korean).
28. KISTEP (2014). 2014 Composite Science and Technology Innovation Index. Seoul, Korea: Korea Institute of S&T Evaluation and Planning (in Korean).
29. KISTEP (2015). 2015 Composite Science and Technology Innovation Index. Seoul, Korea: Korea Institute of S&T Evaluation and Planning (in Korean).
30. Ko, D., Moon, H. & Lee, S. (2015). A Study on the Effect of Government R&D Supports on Firm's Innovation Activities. Korean Journal of Business Administration, vol 28, no 5, pp. 1325-1344 (in Korean).
31. Ko, Y. (2015). 'Selecting and Focusing' Strategy and Policy Culture: An Analysis of the Large-Scale National R&D Programs. Korean Policy Studies Review, vol. 24, no 3, pp. 413-435 (in Korean).
32. Lanjouw, J.O. & Schankerman, M. (2004). Patent Quality and Research Productivity: Measuring Innovation with Multiple Indicators. The Economic Journal, vol. 114, no 495, pp. 441-465.
33. Levy, D. M., & Terleckyj, N. E. (1983). Effects of Government R&D on Private R&D Investment and Productivity: A Macroeconomic Analysis. The Bell Journal of Economics, vol. 14, no 2, pp. 551-561.
34. Lundvall, B.A. (2010). National Systems of Innovation: Toward a Theory of Innovation and Interactive Learning. London and New York: Anthem Press.
35. Michelson, E.S. (2006). Approaches to Research and Development Performance Assessment in the United States: An Analysis of Recent Evaluation Trends. Science & Public Policy, vol. 33, no 8, pp. 546-560.
36. Ministry of Science and ICT & KISTEP (2015). 2014 National Research and Development Program Report. Seoul: Ministry of Science and ICT & Korea Institute of S&T Evaluation and Planning (in Korean).
37. Moynihan, D., Herd, P. & Harvey, H. (2015). Administrative Burden: Learning, Psychological, and Compliance Costs in Citizen-State Interactions. Journal of Public Administration Research and Theory, vol. 25, no 1, pp. 43-69.
38. Nicoletti, G. & Scarpetta, S. (2003). Regulation, Productivity and Growth: OECD Evidence. Economic Policy, April 2003, pp. 9-72.
39. OECD (2015). Frascati Manual: Proposed Standard Practice for Surveys of Research and Experimental Development. Head of Publications Service, OECD Publications Service, pp. 77-79.
40. O'Mahony, M. & Vecchi, M. (2009). R&D, Knowledge Spillovers and Company Productivity Performance. Research Policy, no 38, pp. 35-44.
41. Park, S. & Kwon, O. (2011). The Joint Impact of R&D Intensity and Marketing Intensity on Performance. The Korean Journal of Business Administration, vol. 24, no 6, pp. 3295-3312 (in Korean).
42. Popp, D. (2005). Using the Triadic Patent Family Database to Study Environmental Innovation. OECD Working Paper.
43. Shin, T. (2004). Economic Impact of Basic R&D. Seoul: Science and Technology Policy Institute (in Korean).
44. Song, J. & Kim, H. (2009). The Effectiveness of Fiscal Policies for R&D Investment. Journal of Technology Innovation, vol. 17, no 1, pp. 1-48 (in Korean).
45. Soltmann, C., Stucki, T. & Woerter, M. (2015). The Impact of Environmentally Friendly Innovations on Value Added. Environmental and Resource Economics, no 62, pp. 457-479.
46. Vega-Jurado, J., Gutierrez-Gracia, A. & Fernandez-de-Lucio, I. (2009). Does External Knowledge Sourcing Matter for Innovation? Evidence from the Spanish Manufacturing Industry. Industrial and Corporate Change, vol. 18, no 4, pp. 637-670.
47. Woo, S. (2013). Role of Intellectual Property Rights and Patent Propensity in R&D and Industry Value-added. Master's Thesis. Seoul National University.
48. Yoon, J. (2014). Evolution of Science and Technology Policy in Korea. The Korean Journal of Policy Studies, vol. 29, no 1, pp. 147-172.
