CC BY-NC-ND
35
Grant Research Support in Russia: What Can We Learn from the Russian Science Foundation's First Grant Competition?
Yuri Simachev a
Director for Economic Policy, [email protected]
Liudmila Zasimova a
Assistant Professor, Department of Applied Economics at the Faculty of Economic Sciences, [email protected]
Timur Kurbanov a;b
Post-Graduate Student; and Chief Specialist at the Evaluation Organisation Unit, [email protected]
a National Research University Higher School of Economics, 20, Myasnitskaya str., Moscow 101000, Russian Federation b Russian Science Foundation, 14, bld. 3, Moscow 109992, GSP-2, Russian Federation
Abstract
This paper studies the Russian Science Foundation's first grant competition, which was held in 2014 to select exploratory or basic research projects, in order to shed light on the following two questions: (1) who wins the grants, and (2) what factors are attributed to winning? The subsample of winners (when compared with the whole sample of applicants) seem to have higher proportions of projects submitted to the life sciences section, projects affiliated with the Russian Academy of Sciences (RAS) and projects from Moscow or the Moscow region. Besides, the heads of the winning projects had
better publication indicators. We find that the main factor attributed to winning in the grant competition is the evaluation score given by external experts, while controlling for other factors. Although experts' score is the most influential factor, the probability of receiving grant is strongly associated with others as well. Thus, projects affiliated with the RAS and with the head of the project holding a doctor's degree have some advantages, all other factors being equal. Furthermore, projects from the regions and, most importantly, with young project heads, are more likely to win.
Keywords:
grant;
basic science; science policy; research funding; peer-review
Citation: Simachev Yu., Zasimova L., Kurbanov T. (2017) Grant Research Support in Russia: What Can We Learn from the Russian Science Foundation's First Grant Competition? Foresight and STI Governance, vol. 11, no 4, pp. 74-83. DOI: 10.17323/2500-2597.2017.4.74.83
One of the commonly applied basic research funding mechanisms is a grant awarded by science foundations (or, occasionally, by agencies operating along the same principles), primarily public ones. Such structures tend to have access to significant resources. For example, in 2015, the US National Science Foundation's (NSF) budget amounted to about $7.3 billion1, while the Japan Society for the Promotion of Science (JSPS) allocated more than $2.6 billion.2 Dozens of such foundations and agencies operate around the world, commanding quite impressive shares of the gross domestic expenditures on research and development (GERD) (Figure 1).
In Russia, the state Russian Foundation for Basic Research (RFBR) and Russian Foundation for Humanities (RFH) until recently remained the main grant awarding foundations. At the end of 2013, the Russian Science Foundation (RSF)3, established by a special law,4 was added to this list; it became the largest institution promoting fundamental and exploratory research in the country.
Our analysis focuses on the results of the first RSF grant competition for research projects5 held in 2014. In terms of the lessons to be learned from the RSF's activities and the promotion of best practices, not only are the project selection results of particular importance, but also the possible (and unavoidable for any expert evaluation) biases towards (or against) certain groups of applicants, and the reasons for such biases. At the time of writing this paper, more than three years have passed since the first grant competition: the RSF has held more than 20 other competitions since. The requirements for applications, conditions for taking part in the competitions, and rules for selecting winners have been adjusted, sometimes quite radically, while the network of experts recruited by the foundation has been considerably expanded and altered. Chances are that certain partiality still remains in project selection, but the applied criteria were changed following the significant transformations at the organization of the overall academic environment and the structure of the projects.
Figure 1. Leading International Science Foundations' Shares in GDP and GERD (USD)
18
16
14
Q
Pi
W
U
fy 12
10
ts -d C d
HELMHOLTZ
i
0
0.05
0.10
0.15
0.20
0.25
0.30
Foundation's share in the country's GDP (%)
Legend:
NIH — National Institutes of Health, US;
NRF — National Research Foundation of Korea;
RCN — The Research Council of Norway;
JSPS — Japan Society for the Promotion of Science;
CNRS — Centre national de la recherche scientifique, France;
NSF — National Science Foundation, US;
NSFC — National Natural Science Foundation of China;
DFG — Deutsche Forschungsgemeinschaft, Germany;
Helmholtz — Helmholtz Association of German Research Centres, Germany;
Humboldt — Alexander von Humboldt Foundation, Germany;
RFBR — Russian Foundation for Basic Research;
RFH — Russian Foundation for Humanities;
RSF — Russian Science Foundation
Source: calculated by the authors based on data published by the World Bank (GDP), OECD (GERD), various foundations, and research organizations..
8
RFH
0
1 Available at: http://www.nsf.gov/about/congress/114/highlights/cu15_0109.jsp, last accessed on 07.01.2016.
2 Available at: http://www.jsps.go.jp/english/aboutus/index4.html, last accessed on 07.01.2016.
3 For more about the RSF and its activities, see its official website at: http://rnf.rf/ru/about, last accessed on 07.01.2016.
4 Federal Law of 02.11.2013 № 291-FZ "On the Russian Science Foundation, and amendments to certain RF legislation".
5 The official name is the "Competition for grants awarded in the framework of the Russian Science Foundation's priority activity area "Basic and exploratory research conducted by individual research teams"".
Table 1. Characteristics of the First RSF Grant Competiton by Field of Knowedge
Academic field Number of accepted applications Number of supported projects Share of supported projects (%) Number of expert evaluations Number of expert evaluations per project(average) Number of evaluations per expert (median)
Mathematics, informatics, systems science 784 69 8.8 2 302 2.94 4
Physics, space sciences 1 305 115 8.8 3 850 2.95 5
Chemistry, materials sciences 1 328 122 9.2 3 910 2.94 7
Biology, life sciences 1 085 150 13.8 3 196 2.95 7
Basic research in medicine 972 123 12.7 2 842 2.92 7
Agricultural sciences 120 17 14.2 343 2.86 2
Earth sciences 703 70 10 2 077 2.95 5
Social sciences, humanities 3 390 94 2.7 9 772 2.88 14
Engineering sciences 1 528 115 7.5 4 350 2.85 3
Total 11 215 875 7.8 32 642 2.91 6
Source: calculated by the authors based on RFS data.
The establishment of the RSF has put an end to the competition between various approaches to funding fundamental research in Russia, including such traditional tools as government orders, thematic research plans, government programs and federal targeted programs based on contracts or budget subsidies, and even grants awarded by science foundations. The RSF has become the symbol of a resolute turn from targeted programs towards foundations, from public contracts to grants, thus changing the decision-making chains, the rules of the game, and the circle of major beneficiaries. Such a shift in the research policy prompted intense debates (not at all limited to academic circles) about the merits of the extended application of grant-based mechanisms, and whether it would create new problems to replace the old ones.
The first RSF competition was noteworthy because of its size: more than 12,000 applications were submitted by research teams specializing in all fields of knowledge, comprising more than 90,000 researchers altogether. When assessing the results of subsequent RSF competitions and discussing general approaches to supporting basic research in Russia, the participants of the relevant debates still tend to cite their personal impressions of the initial period of the foundation's operations.
A typical feature of Russian government policies (and science policy is no exception) are pendulum changes, when at each round of reforms, the faults of the previous one are absolutized while the changes not infrequently amount to swinging from one extreme to another, reproducing the problems in the process. The trade-off nature of certain decisions made during the early stages suggests the need to analyze the costs associated with the various ways of organizing science foundations and their operations. The value of retrospective analysis is that it provides the chance to avoid potential risks by learning from previous negative experiences. However, such an approach is not without its faults given that it does not imply a formalized or meaningful evaluation of the results and a comparison of them and the initial expert assessments of the submitted research project applications.
Specific Features of the First RSF Competition
The first competition for grants to fund research projects implemented by individual research teams in all academic fields was announced in February 2014 to provide grants for research in 2014-2016, with a possible extension by one or two years. Out of the 12,774 applications submitted by research teams from more than 1,200 organizations, 11,215 were selected for peer review (Table 1). Projects in the field of social sciences and humanities were in the obvious majority. The experts selected 875 research projects, though according to the organizers, the number of worthy applications was a good deal greater than the foundation could afford to fund [Khlunov, 2014].
The first of the two peer review stages involved assessing the applications and preparing conclusions about them. The coordinators of the evaluation council allocated project applications between the experts. The peer review procedure was unilaterally anonymous: researchers did not know who their reviewers were, while the latter could access information about the research teams and their members. Each application was assessed by three experts independently from one another other using several criteria. A project could receive a score between 0 and 120 points, which were awarded automatically when experts chose the relevant points of the expert evaluation. During the second stage, applications were evaluated by sections of the expert council on the basis of the preliminary expert opinions. To avoid any conflicts of interest, those projects for which members of the expert council had recommended funding had to be additionally approved by secret ballot following the second evaluation stage. The foundation's management stressed the need to take into account the researchers' opinions when final decisions were made [Khlunov, 2014].
Certain specific features of the first RSF competition should be noted, distinguishing it from tenders held by other Russian science foundations. By the beginning of 2014, two major research programs were completed: "Research and development in priority areas of the Russian S&T complex in 2007-2013" and "Research and academic personnel for an innovative Russia in 2009-2013". After that, the funding of numerous research teams was discontinued, which prompted the management of the newly founded science foundation to organize a competition as quickly as possible. The law establishing the RSF was signed on November 2, 2013, the first competition to award grants to research teams was announced already on February 6, 2014, and its results were published on May 20 of the same year [RSF, 2015]. Researchers' great interest in the competition was due not only to a lack of funding, but also to the unique opportunity to obtain significant resources (up to 5 million rubles a year per team) with flexible spending arrangements and a moderate bureaucratic load. The same factors also explained the unusually stringent, in most of the researchers' opinion, principles of project selection: the share of awarded projects (success rate) in the total number of applications did not exceed 8%. In the international context, such rate does not seem to be very unusual. For example, the relevant figure for France's Agence Nationale de la Recherche (ANR) is 8-12% [ANR, 2015]. However, the debates that began after the RSF competition results were announced revealed that many researchers who had previously applied for the RFBR and RFH grants expected about a third of the projects to be awarded grants while they would be able to easily repeat their successful experience of participating in other competitions. Therefore, defeat was an unpleasant surprise for them, and the logic of selecting the best projects out of so many good ones declared by the RSF was not warmly received.
A major innovation introduced by the organizers was the "entry ticket" principle, the requirement for a minimum number of publications in Web of Science- (WoS) or Scopus-indexed journals by the team lead. At the first competition, this rule did not apply to all knowledge areas, an exception was made for the social sciences and humanities (their representatives have successfully argued these fields have distinct disciplinary and national features). As a result, scientists specializing in these areas found themselves in an irrationally favorable competitive position and submitted the largest number of applications, though less than 3% of them were approved (see Table 1). The burden on the experts in this group was also the highest: on average, each expert had to evaluate 14 applications in two months' time. The peer review exercise in the scope of the first RSF competition was conducted on an exceptionally large scale and in a very short period of time. More than 2,500 experts participated in assessing the applications. The total number of prepared expert opinions exceeded 32,000, on average, there were three opinions per application. Given that at the time of its establishment the RFS lacked its own pool of experts, it recruited them from the RFBR and RFH as well as additional external experts specializing in certain subject areas [Klimenko, 2014].
Research Hypotheses and Initial Data
Grants are supposed to solve the problem of insufficient basic research funding by the private sector. The private sector is generally unable to provide such funding due to numerous reasons such as high risk, low or zero profit from the application of research results, and difficulties with the capitalization of fundamental discoveries [Nelson, 1959; Pavitt, 1991]. Without questioning the need to provide public support for research, our further deliberations are based on studies devoted to optimizing arrangements for the provision of basic research funding and, among other things, dealing with the issue of public investments pushing out private ones [Mowery, 1990] and finding the best ways to fund research through grants.
Studies on grant-based research funding revealed serious flaws in the relevant mechanisms employed by various countries. For example, applying for a grant not infrequently involves presenting already obtained research results, which means that only teams possessing a relevant portfolio have a chance to receive support [Lazeur, 1997]. Another common problem involves taking into account applicants' previous achievements, the so-called "Matthew effect" [Merton, 1968; Antonelli, Crespi, 2011]. In the case of awarding grants, it amounts to favorably treating research teams that at the time of application had a better reputation, more impressive results, better work conditions, etc. On the one hand, such policies adopted by science foundations encourage young researchers to more efficiently implement their projects due to the fact that their results would play a role in the future [Lazeur, 1997]. On the other, it means that the foundations' resources will be allocated in favor of more experienced applicants. The relevance of the aforementioned barriers is confirmed by recent empirical research. For example, Arora and Gambardella [Arora, Gambardella, 2005] analyze, using NSF selection practices, how peer review and final grant allocation decisions are affected by such variables as the team leads' or leading researchers' characteristics (gender, year they received their PhD); the name and type of applicants' organizations; the number of project participants; the number of publications during the previous five years, weighted by quality; and the reviewers' opinions, etc. The authors discovered a positive correlation between expert-assigned scores and the number of publications by applicants. Even if the projects proposed by researchers with impressive academic reputations turn out to have inferior formal parameters compared with competing applications, they still receive more favorable treatment when funding decisions are made. Meanwhile, young researchers only have a chance of receiving support if they are exceptionally successful (the authors call them "stars"). There is also a certain "positive discrimination," applicants from the regions tend to have a slight advantage over teams based in metropolitan areas. An obvious answer to the challenges associated with the public funding of research projects is designing an optimal project selection mechanism. Along with the direct correlation between the number of applicants' publications and the awarding of grants [Arora, Gambardella, 2005; Jacob, Lefgren, 2011; Gush
et al., 2015], there are also signs of an inverse relationship: researchers who were awarded grants tend to have much better chances to publish. For example, the experience of the New Zealand Marsden Fund [Gush et al., 2015] shows that publication activity increases even if the fund supports weak projects: the support is seen as evidence of the projects' quality and encourages scientific journals to publish papers by their participants.
An adequate project selection mechanism should provide a good balance between anonymous ("blind") peer review and space for "manual fine-tuning". The example of the NIH confirms that the higher the project's score assigned by experts at the first stage of selection was, the better results the supported team gets in terms of citation, patenting, publication, and contributing to the advancement of the relevant field of knowledge (keeping all other factors equal) [Li, Agha, 2015]. At the same time, the management of the program decided to make an exception for about 1% of the projects that did not score the minimum necessary number of points. Interestingly, the performance of these "manually selected" projects' teams measured in terms of their publication activity turned out to be comparable with those that received the highest scores at the project selection stage. Another study [Park et al., 2015] compared two mechanisms for the allocation of the NIH resources: regular and additional competitions. The latter's arrangements were the same as the main competition's, it was held for projects that had never been awarded grants but received high scores. The comparison did not reveal any significant differences between the performance of the teams awarded grants under these two tenders.
As we see, projects' prospects, and the validity of their peer review results, cannot be adequately estimated at the competitive selection stage. Therefore, studies of grant-based research funding concentrate on factors affecting project selection results. The main issues here are associated with the lack of transparency and predetermined results. Though peer review remains crucially important for the successful operation of science foundations, experts' decisions tend to be somewhat arbitrary and biased. Numerous studies reveal that such procedures are less than perfect, highlighting partiality regarding women, less prestigious organizations and subject areas, intellectual piracy, and barriers hindering publication [Smith, 2006; Benos et al., 2007]. Thus, designing an optimal project selection mechanism that would minimize the faults of the first and second kinds of selection has become a critical issue on the grant-based research funding agenda.
No large-scale empirical studies of research project selection have been conducted in Russia so far. International studies and domestic debates triggered by the results of the first RSF competition permit one to propose the following hypotheses on the factors affecting the selection of research projects to award grant funding:
Hypothesis 1: Research teams who, at the time of submitting an application, have a more impressive reputation, first of all due to the team leader's merits (such as his or her academic degree, number of publications, etc.) tend to have an advantage.
Hypothesis 2: The chances of receiving a grant depend not only on the scores awarded by individual experts, but also those given by the expert council as a whole, which in turn depends upon the characteristics of the research teams (their leaders and members) and applications (the requested amount of funding, the number of expected publications).
Hypothesis 3: If expert-assigned scores are equal, research teams from Moscow- and Moscow Region-based academic institutes tend to have better chances of receiving support.
The empirical basis of the study is comprised of over 32,000 expert opinions on more than 11,000 applications. The analysis was based on data on awarded applications, applicant research teams (the number of publications by their leaders and participants, age, gender), and the projects (research area, expected results, etc.). A regression analysis (binary choice model) was used to assess the factors affecting the probability of receiving grants.
Since it is not possible to construct a variable to accurately assess the quality of an application at the project selection stage, indirect project potential indicators were used. The model comprised the following groups of variables (see Table 2 for their characteristics):
(1) Project leader: age, gender, number of WoS-indexed publications, academic degree;
(2) Project participants: total number, number of PhD holders aged under 35, characteristics of their organization (location, status (national research university (NRU), academic institute, etc.));
(3) Project application: amount of funding requested, expected number of publications in WoS-indexed journals based on the project results, multidisciplinary project or not.
To assess the interaction of factors, we have used a logistic regression model with the variable "Grant awarded or not". In the general form it looks as follows:
ln(P / (1 - P)) = B + B\ Mark + ... + B* NRU (1)
v v J J 0 1 mean 26 v J
where:
P — probability of receiving project support,
B1, ...B26 — coefficients with independent variables,
Mark_mean, ... NRU — independent variables.
Different variations of the model comprising different sets of independent variables were tested for robustness. In the first model, mean scores were used as an independent variable; in the second, academic fields were added; in the third, a number of important other independent variables; and the fourth and final model included the full set of factors.
Table 2. Variables Used in the Analysis
Variable Value Min Max Standard deviation Mean
Awarded 1 — grant awarded, 0 — grant not awarded 0 1 0.268 0.078
Mark mean Average score of the project 0.5 120 19 68
PI WoS (Hirsh) Team leader's Hirsh index for WoS-indexed journals 0 84 8.3 9
PI WoS (publ-s) Number of team leader's publications in WoS-indexed journals during the previous 5 years 0 533 21.5 11.8
Msk_plus_obl 1 — team's organization is located in Moscow or the Moscow region 0 1 0.48 0.35
Age PI Team leader's age by April 1, 2014 22 92 13.7 53.2
Gender PI Male — 1, female — 0 0 1 0.46 0.7
WoS_promise Expected number of publications in WoS-indexed journals based on project results 0 65 4.8 5.9
Number participants Number of research team members 1 55 3.48 8.36
Number_young_candid Number of PhD holders aged under 35 at the time of the competition 1 21 1 1.74
Multidisciplinary 0 — multidisciplinary project, 1 — non-multidisciplinary project 0 0.27 0.92
NRU 0 — organization is not an NRU, 1 — organization is an NRU 0 0.34 0.13
Academy 0 — organization is not an academic institute, 1 — organization is an academic institute 0 0.48 0.355
PI_doct 1 — team leader has a Doctor of Sciences degree, 0 — team leader does not have a Doctor of Sciences degree 0 0.475 0.655
Requested finance 50 2000 1 — requested amount is between 50,000 and 2 million rubles, 0 — requested amount is 2 million rubles or more 0 0.34 0.13
Requested finance_2000_4000 1 — requested amount is between 2 million and 4 million rubles, 0 — requested amount is under 2 million rubles, or 4 million rubles or more 0 0.44 0.26
Requested finance 4000 5000 1 — requested amount is between 4 milllion and 5 million rubles, 0 — requested amount is under 4 million rubles 0 0.49 0.6
WoS_promise_0 1 — no publications are expected; 0 — number of expected publications is greater than 0 0 0.16 0.03
WoS_promise_1_10 1 — number of expected publications is between 1 and 10; 0 — other number of expected publications 0 0.34 0.87
WoS_promise_8_plus 1 — number of expected publications is 8 or more; 0 — other number of expected publications 0 0.43 0.25
WoS_promise_11_20 1 — number of expected publications is between 11 and 20; 0 — other number of expected publications 0 0.29 0.09
WoS_promise_21_65 1 — number of expected publications is between 21 and 65; 0 — other number of expected publications 0 0.12 0.014
Age_P I_b efore_35 1 — team leader's age is under 35; 0 — team leader's age is 35 or older 0 0.33 0.12
Age_PI_35_45 1 — team leader's age is between 35 and 45; 0 — team leader's age is under 35 or over 45 0 0.38 0.17
Age_PI_45_55 1 — team leader's age is between 45 and 55; 0 — team leader's age is under 45 or over 55 0 0.4 0.2
Age_PI_55_plus 1 — team leader's age is 55 or older; 0 — team leader's age is under 55 0 0.5 0.51
Source: calculated by the authors based on the RSF data.
Main Results
Let us take a brief look at the most significant differences between the projects that have and those that have not been awarded grants (Table 3). It can be noted straightaway that the funding decisions were mostly based on the results of the peer review exercise.
The results of the RSF competition for grants to support basic research projects in various knowledge areas show that most frequently grants were awarded to projects in the life sciences. This was probably due to the foundation's default objective to compensate for the inadequate attention paid by the conventional research funding system to academic fields directly connected with meeting people's needs, such as biology, medicine, and agriculture. A descriptive analysis revealed that the projects proposed by larger teams, headed by men, holders of PhDs, and of a noticeably more advanced age than their unlucky colleagues, were treated more favorably. However, any high-quality research project tends to have some of these characteristics. Compared with the structure of the total body of submitted applications, the winners included a higher share of academic institutes and organizations based in Moscow and the Moscow region. This, however, may simply be due to the high concentration of leading scientists at the Russian Academy of Sciences, and at research institutes based in Moscow.
The publication activity of project teams' members, and especially their leaders, turned out to be a very important factor affecting selection results (Table 4). Significant differences were also found between various academic fields, reflecting not so much the latter's specific features as the particulars features of their development in Russia, i.e., integration in, or, on the contrary, isolation from, the international context.
Table 3. ^mpa^t^e Characteristics of Projects that Were and Those that Were not Awarded Grants in the First RSF Competition
Indicators Awarded projects Rejected projects All applications
Average project score 98.7 65.4 68
Average team leader age 56.7 52.9 53.2
Share of male team leaders (%) 85 69 70
Average number of team leaders' publications in WoS-indexed journals over the previous 5 years 26.7 10.3 11.6
Team leader's Hirsh index for WoS-indexed journals 14.8 7.8 9
Share of team leaders with a PhD (%) 85.3 63.8 65
Average number of team members 11 8.13 8.4
Average number of young PhD holders aged under 35 on the team 2.14 1.7 1.7
Average requested amount of grant funding (thousand rubles) 4624 3764.4 3831.4
Average number of expected publications in WoS-indexed journals based on project results 8.4 5.6 5.9
Share of applicants from NRUs (%) 9.3 13.3 13
Share of applicants from academic institutes (%) 59.2 33.5 36
Share of teams based in Moscow or the Moscow region (%) 52 34 35
Share of multidisciplinary projects (%) 9 7.7 8
Share of projects in social sciences (%) 10.7 31.9 30.2
Share of projects in biology and medicine (%) 31.2 17.3 18.3
Share of projects in mathematics, physics, and chemistry (%) 35 30.1 30.5
Note: 875 awarded projects, 10,340 rejected ones. Source: calculated by the authors based on RSF data.
In order to reveal possible partiality in project selection, a regression analysis was conducted using four specifications with the different sets of independent variables (see Table 5), supplemented by a knowledge area-specific regression analysis (Table 6). It can be noted straight away that expert-assigned scores turned out to be significant in all specifications, with the highest marginal effect for this explanatory variable. Individual expert assessments, which in a number of cases provided certain guidance to the RSF council when it selected applications to receive grants, especially when expert opinions clashed, could differ from the relevant council section's final position. The level of such a divergence generally tended to be knowledge area-specific, depending upon the degree of the area's integration into the global context and its practical focus. For example, expert opinions on "hard science" projects usually were less contradictory than those regarding "soft" ones. Specific council sections' general predisposition towards the active or, on the contrary, the evolutionary development of relevant subject areas seems to be no less important. The greatest unanimity was demonstrated by such expert council sections as the engineering sciences, earth sciences, physics, and chemistry. Projects in mathematics, biology, medicine, social sciences, and the humanities required the most significant adjustments (see Table 6).
As to partiality regarding project leaders, the regression model revealed no gender inequality in project selection: this factor was insignificant in the third and fourth specifications alike. At the same time,
Table 4. Comparison of the Publication Activity of the Leaders of Awarded and Rejected Projects in WoS- and RSCI-indexed Journals, by Knowledge Area
Knowledge area Entry requirement for team leader Median number of WoS publications in 5 years' time Median number of RSCI publications in 5 years' time
awarded rejected awarded rejected
Mathematics > 3 WoS/Scopus publications 14 7 24 16
Physics > 3 WoS/Scopus publications 27 13 32.5 19
Chemistry > 3 WoS/Scopus publications 36 11 44 21
Biology > 3 WoS/Scopus publications 19 8 21 14
Medicine > 3 WoS/Scopus publications 18 7 34 22
Agriculture > 3 WoS/Scopus publications 8.5 5 23 21
Earth sciences > 3 WoS/Scopus publications 12 7 27 15
Social sciences, humanities > 3 WoS/Scopus or RSCI publications (or a peer-reviewed monograph) 2 0 23 13
Engineering sciences > 3 WoS/Scopus publications or > 2 such publications and 2 legally protected results of intellectual activities 16 4 31 18
Sources: calculated by the authors based on RSF data.
Table 5. Results of the Regression Analysis of the Awarding of Grants to Support Research Projects in the First RSF Competition
Indicators Specification (1) Specification (2) Specification (3) Specification (4)
Variable Coefficient value Coefficient value Coefficient value Coefficient value
Average project score 0,292*** (0,010) 0,367*** (0,013) 0.365*** (0.013) 0.367*** (0.014)
Team leader's age under 35 Not included in the model Not included in the model 0.912*** (0.31) 1.019*** (0.321)
Team leader's age between 35-45 Not included in the model Not included in the model 0.548** (0.229) 0.5778** (0.238)
Team leader has PhD Not included in the model Not included in the model 0.424** (0.199) 0.409** (0.207)
Based in Moscow or the Moscow region Not included in the model Not included in the model n.s. -0.314** (0.136)
Academic institute Not included in the model Not included in the model 0.515*** (0.129) 0.484*** (0.145)
Number of team members Not included in the model Not included in the model 0.092*** (0.018) 0.105*** (0.021)
Multidisciplinary project Not included in the model Not included in the model -0.521** (0.225) -0.631*** (0.235)
Knowledge area taken into account No Yes Yes Yes
Number of observations 11 211 11 211 11 041 8 761
Pseudo R2 0.64 0.71 0.72 0.714
Notes: *** — p < 0.01, ** — p < 0.05, * — p < 0.1, n.s. — not significant, standard errors shown in brackets; only significant variables are presented. Source: calculated by the authors based on RSF data.
an opposite age bias was discovered compared with the trend revealed through the descriptive analysis, i.e., in favor of younger project leaders. On the whole, this matches the mainline parameters of efficient grant-based funding presented in [Lazeur, 1997]. However, the members of the expert council were hardly guided by such theories. Rather, the feelings of personal moral responsibility for the future careers of young scientists played a role. At the same time, a somewhat increased attention to applicants' status can still be detected: all other conditions being equal, research teams headed by PhD holders tended to be awarded grants more often.
Research teams' expectations regarding future publications based on project results6 generally did not significantly affect grant decisions, including for specific fields of research (with the exception of biology).
Table 6. Regression Analysis Results for Specific Knowledge Areas (Significant Variables Only)
Variable Mathematics Physics Chemistry Biology Medicine Earth sciences Social sciences Engineering sciences
Average project score 0.252*** (0.032) 0.798*** (0.108) 0.547*** (0.072) 0.280*** (0.026) 0.272*** (0.028) 0.793*** (0.147) 0.362*** (0.046) 1.697*** (0.356)
Team leader's number of publications in WoS-indexed journals n.s. n.s. 0.021* (0.013) 0.044*** (0.015) 0.017* (0.010) n.s. n.s. n.s.
Team leader's gender (1 — male) 2.953** (1.26) n.s. n.s. n.s. n.s. n.s. n.s. n.s.
Team leader's age — 55 or older n.s. n.s. n.s. n.s. n.s. n.s. n.s. (borderline) n.s.
Moscow and the Moscow region n.s. n.s. n.s. -0.955*** (0.341) n.s. n.s. -0.903** (0.450) n.s.
Academic institute 1.544*** (0.439) 1.05* (0.61) n.s. n.s. n.s. n.s. n.s. n.s.
Number of young PhD holders (under 35) n.s. n.s. n.s. -0.267* (0.15) n.s. n.s. n.s. n.s.
8 or more publications expected n.s. n.s. n.s. 0.659* (0.357) n.s. n.s. n.s. (borderline) n.s.
Multidisciplinary project n.s. n.s. n.s. n.s. -1.54*** (0.589) n.s. -2.860** (1.434) n.s.
Number of team members n.s. n.s. 0.276*** (0.093) n.s. n.s. 0.355*** (0.108) n.s. 0.297** (0.149)
Number of observations 733 1237 1258 1047 877 665 1552 1283
Pseudo R2 0.55 0.84 0.84 0.67 0.64 0.84 0.71 0.94
Notes: *** — p < 0.01, ** — p < 0.05, * — p < 0.1. Source: calculated by the authors based on RSF data.
6 The descriptive analysis revealed that leaders of grant recipient teams not only had more publications than their less successful colleagues, but also expected to publish more papers based on the project results.
We believe this can be explained by researchers' inclination to recognize less than realistic publication plans. For example, some of the applicants promised to prepare more than 40 papers for publication in WoS-indexed journals even though the leaders of these teams had not published a single such paper over the course of the last five years.
Larger teams tended to have a certain advantage during the selection of projects, which is rather hard to explain. According to some studies, for example, [Park et al, 2015], team size can make a positive impact on project quality. However, in our opinion, for many teams, the RSF funding was a way to increase their total earnings but they were not actually prepared to compete for grants in earnest. Regarding multidisciplinary projects, the regression analysis yielded an effect directly opposite to the one observed during the selection process: all other conditions being equal, this factor negatively affected grant awarding. We believe this is due to the burdensome requirement to receive the approval of at least two sections of the expert council.
Reservations about the impartiality of rewarding grants to projects proposed by academic institutes on the whole were confirmed: the experts did tend to treat them preferentially. A possible explanation for this is the high share of representatives of academic organizations on the foundation's expert council: about six RAS staff members per university employee. However, another theory seems to be more convincing. At the beginning of 2014, when the competition took place, academic institutes were undergoing the painful process of organizational and financial transformation, so colleagues felt obliged to help researchers employed by these organizations. As to regional distribution, the hypothesis about projects proposed by Moscow and Moscow region-based organizations receiving favorable treatment was not confirmed. The foundation's experts probably believed that it was the provincial organizations that needed research funding the most. Supporting research and promoting the emergence of a favorable innovation environment in the regions is believed to be an important component of the relevant foundations' activities the world over. In particular, the analysis of the NSF project selection practices [Arora, Gambardella, 2005] also revealed a "positive discrimination" regarding this feature: regional projects had the best chances of receiving grants.
Discussion of the Results
The following questions seem to be relevant regarding the provision of support for basic research, in particular, and the advancement of Russian science policy in general:
How closely did the project selection results match the opinion of the academic community, i.e., the expert-assigned scores?
Peer review remains the main project selection mechanism. Occasionally, an evaluation by external experts is used, but peer review generally implies an assessment by fellow researchers. Contrary to certain statements, the qualifications of RSF experts (somewhat arbitrarily measured by the number of their publications in WoS- or Scopus-indexed journals) was comparable with that of the research team leaders, though lower than the winners'. The involvement of the practicing researchers in project selection has both positive and negative aspects. The first positive aspect includes their professionalism and the relevance of their opinions, the dissemination of various approaches and ideas across the academic environment, and the emergence of new collaborations. The costs of this approach, especially when only Russian experts are recruited to assess applications (which was quite common during the early period of the RSF), involve clique-like seclusion, conflicts of interest, and occasionally even the risk of inadequate selection due to the insufficient level of Russian science in certain academic fields.
The actual results of the RSF project selection provide evidence of the high priority given to expert opinions: the empirical analysis revealed that the average expert-assigned score played a decisive role in the award of grants.
What contribution did the awarded grants actually make?
Giving a valid answer to this question would require reviewing the results of supported projects and comparing them with those of rejected ones. However, it can already be noted that in all knowledge areas, the leaders of successful research teams had already published several times more papers in peer-reviewed journals and had several times higher Hirsh index scores for WoS-indexed journals than their less successful colleagues. We are not inclined to overestimate the importance of these indicators, but as statistical tools they are quite useful. It would not be a great exaggeration to state that the first RSF competition managed to successfully identify unique members of the Russian research community. And this, that is, supporting world-class research and researchers, was the overall objective of founding the RSF in the first place.
Keeping in mind the inevitable errors the first and second types of evaluation made during project selection, and the financial limitations that the RSF had to face, we must note that not all worthy projects were supported. On the whole, it can be said that the grants were awarded in line with meritocratic principles.
Did the experts share any ideologically similar inclinations, partiality, or biases?
In our opinion, the favorable treatment of larger projects teams and projects proposed by regional academic institutes fits the academic community's traditional paternalistic model, which combines competition with social equality. Two sets of logic clashed when grant decisions were made: rewarding achievements vs. encouraging development, a project-based approach vs. regular, ongoing funding. We believe that the pronounced social motives arose over the course of the selection process due to the unbalanced structure of the Russian basic research funding system, primarily the lack of basic funding.
The distorted motivation of the RSF applicants was largely due to a lack of other sources of basic research funding. For many research teams, the foundation's money was a matter of survival, which prompted them to submit applications and apply various forms of indirect pressure. Such an overload on science foundation-operated grant allocation systems caused by the limited availability of other basic funding sources was also noted in countries where the research sector received much more generous support. Active fundraising appears to play a positive role, expanding the opportunities for selecting the best projects. On the other hand, it reflects the decline of previously available opportunities combined with direct administrative encouragement. In particular, certain institutes and universities began to adopt targets for securing RSF grants, and sometimes for the submission of applications as well. How did the first RSF competition affect researchers' behavior and motives?
Firstly, it prompted researches to review and self-select their proposals before actually submitting an application to the science foundation, and treat their obligations more responsibly. Secondly, their personal motives for publishing in internationally peer-reviewed journals increased, while the editorial boards of Russian academic journals discovered additional incentives to get indexed by Scopus and WoS databases. Thirdly, the research community now recognizes the need to regulate ethical issues. Fourthly, there is stronger demand for able team leaders capable of setting ambitious research objectives and accomplishing them by implementing relevant projects and recruiting professionals from various organizations.
The important thing here is the emergence of a strong trend towards improving general RSF procedures, as opposed to making sensible but narrow-minded decisions. Practice shows that the research community is the main source of the best rules and practices, provided that science foundations steadily and strictly adhere to such rules afterwards.
The preliminary results of this study were presented at the HSE April International Academic Conference on Economic and Social Development (Moscow, April 19, 2016). Some of the additional results and interpretations were presented at the GSOM Emerging Markets Conference hosted by the Graduate School of Management of St. Petersburg State University (October 6, 2016.).
References
ANR (2015) Annual Report 2015, Paris: Agence Nationale de la Recherche.
Antonelli C., Crespi F. (2011) Matthew Effects and R&D Subsidies: Knowledge Cumulability in High-tech and Low-tech Industries (Working paper 11/2011), Rome: University 'Roma Tre'.
Arora A., Gambardella A. (2005) The impact of NSF support for basic research in economics. Annales d'Economie et de Statistique, no 79-80, pp. 91-117.
Benos D.J., Bashari E., Chaves J.M., Gaggar A., Kapoor N., LaFrance M., Mans R., Mayhew D., McGowan S., Polter A., Qadri Y., Sarfare S., Schultz K., Splittgerber R., Stephenson J., Tower C., Grace W.R., Zotov A. (2007) The ups and downs of peer review. Advances in Physiology Education, vol. 31, no 2, pp. 145-152.
Gush J., Jaffe A.B., Larsen V., Laws A. (2015) The Effect of Public Funding on Research Output: The New Zealand Marsden Fund (NBER Working Paper no w21652), Cambridge, MA: National Bureau of Economic Research.
Jacob B.A., Lefgren L. (2011) The impact of research grant funding on scientific productivity. Journal of Public Economics, vol. 95, no 9, pp. 1168-1177.
Khlunov A.V. (2014) Khoroshikh zayavok v dva s polovinoi raza bol'she, chem grantov [There is two and a half more good applications than available volume of grants (interview)]. Gazeta.Ru, 23.05.2014. Available at: http://www. gazeta.ru/science/interview/nm/s6040141.shtml, accessed 12.07.2016 (in Russian).
Klimenko A.V. (2014) Vsyo po chestnomu. Uchenye otsenyat ob'jectivnost' ekspertizy RNF (intervyu) [All is fair. Scientists will evaluate objectiveness of the RSF's expertise (interview)]. Nauchnoe obozrenie, 28.05.2014. Available at: http://scientific.ics.org.ru/news/uchenye-ocenyat-obektivnost-ekspertizy-rnf, accessed 12.07.2016 (in Russian).
Lazeur E.P. (1997) Incentives in Basic Research. Journal of Labor Economics, vol. 15, no 1, part 2: Essays in Honor of Yoram Ben-Porath, pp. S167-S197.
Li D., Agha L. (2015) Big names or big ideas: Do peer-review panels select the best science proposals? Science, vol. 348 (6233), pp. 434-438.
Merton R.K. (1968) The Matthew Effect in Science. Science, vol. 159 (3810), pp. 56-63.
Mowery D. (1990) The Growth of U.S. Industrial Research, Berkeley, CA: University of California (mimeo).
Nelson R. (1959) The simple economics of basic scientific research. The Journal of Political Economy, vol. 67, no 3, pp. 297-306.
Park H., Lee J.J., Kim B.C. (2015) Project selection in NIH: A natural experiment from ARRA. Research Policy, vol. 44, no 6, pp. 1145-1159.
Pavitt K. (1991) What makes basic research economically useful? Research Policy, vol. 20, no 2, pp. 109-119.
RSF (2015) Otchet Rossiiskogo nauchnogo fonda za 2014 god [RSF Annual Report 2014], Moscow: Russian Science Foundation (in Russian).
Smith R. (2006) Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, vol. 99, no 4, pp. 178-182.
