CC BY
62
COMPARATIVE ANALYSIS OF MODERN APPROACHES TO THE PERFORMANCE ASSESSMENT OF SCIENTIFIC MEDICAL ORGANIZATIONS IN RUSSIA AND ABROAD
Aniskevich AS1H, Halfin RA2, Tatarinova LV3
1 Ministry of Healthcare of the Russian Federation, Moscow, Russia
2 Higher School of Healthcare Management,
The First Sechenov Moscow State Medical University, Moscow, Russia
3 Russian Society of Regenerative Medicine, Medical Rehabilitation, Physiotherapists and Balneologists, Moscow, Russia
The article reviews basic methods of performance assessment of scientific medical organizations in Russia. Qualitative and quantitative effectiveness criteria are provided. International practices are described; a comparative analysis of assessment methods used in Russia and abroad is carried out. Global trends in the development of approaches to analyzing the effectiveness of scientific organizations are reviewed. Based on our analysis, a compelling rationale for developing more up-to-date criteria for evaluating the performance of scientific medical organizations is given.
Keywords: performance assessment of scientific medical organizations, effectiveness and relevance of scientific research, human resources, integration into world science, dissemination of scientific knowledge, enhancing the prestige of science, resource maintenance of the scientific organization
[X] Correspondence should be addressed: Anna Aniskevich
Rakhmanovskii per., d. 3, Moscow, Russia, 127994, GSP-4; anna.aniskevich@yandex.ru
Received: 20.02.2016 Accepted: 24.02.2016
СРАВНИТЕЛЬНЫЙ АНАЛИЗ СОВРЕМЕННОГО РОССИЙСКОГО И ЗАРУБЕЖНОГО ПОДХОДОВ К ОЦЕНКЕ ДЕЯТЕЛЬНОСТИ НАУЧНЫХ ОРГАНИЗАЦИЙ В СФЕРЕ ЗДРАВООХРАНЕНИЯ
А. С. Анискевич1 Р. А. Хальфин2, Л. В. Татаринова3
1 Министерство здравоохранения Российской Федерации, Москва
2 Высшая школа управления здравоохранением,
Первый Московский государственный медицинский университет имени И. М. Сеченова, Москва
3 Российское общество врачей восстановительной медицины, медицинской реабилитации, курортологов и физиотерапевтов, Москва
В статье приведены основные методы оценки деятельности российских медицинских научных организаций. Описаны количественные и качественные показатели эффективности. Представлен мировой опыт, проведен сравнительный анализ методов оценки, используемых в России и за рубежом. Рассмотрены мировые тенденции развития подходов к изучению эффективности научных учреждений. На основании проведенного анализа продемонстрированы убедительные данные о необходимости разработки более совершенных критериев оценки деятельности научных медицинских организаций.
Ключевые слова: оценка деятельности медицинских научных организаций, результативность и востребованность научных исследований, кадровый потенциал, интеграция в мировое научное пространство, распространение научных знаний, повышение престижа науки, ресурсное обеспечение деятельности научной организации
123 Для корреспонденции: Анна Сергеевна Анискевич
127994 ГСП-4, г Москва, Рахмановский пер., д. 3; anna.aniskevich@yandex.ru
Статья поступила: 20.02.2016 Принята к печати: 24.02.2016
A modern approach to assessing the performance of medical research institutions in Russia's health sector is mostly quantitative in nature and features a large number of indicators. On one hand, such comprehensive analysis facilitates collection of miscellaneous data about organizations and certain areas of their activities. On the other hand, however, it complicates regular monitoring and receipt of objective results. Use of numerical scales involves calculation of the
total average indicator based on which further comparative analysis is to be conducted. Combination of diverse factors, inability to identify the most important ones and complexity in clear interpretation of values increase the likelihood of errors in assessing research institutions while applying a quantitative approach. The assessment results may be incomplete or not sufficiently reliable to address the issue of further prospects of the research institution [1].
QB3QP I 3flPABQQXPAHEHME
According to information on the activities of research institutions carrying out research, developmental and technological activities for the purpose of monitoring and evaluation, approved by Order No 162 of the Russian Ministry of Education and Science on March 5, 2014, the following are the key performance indicators:
- effectiveness and relevance of scientific research;
- human resource development;
- integration into the global scientific space, dissemination of scientific knowledge and enhancement of the prestige of science;
- resourcing the activities of research institution.
Effectiveness and relevance of research institutions
The effectiveness and relevance of the activities of medical research institutions are enough for evaluation of the quantitative and qualitative characteristics of research activities of the institutions. For a comprehensive evaluation of this sector, resource indicators — funding, personnel and logistics — are also factored in.
Application of mainly quantitative methods for performance analysis of research institutions has led to active development of scientometric systems [2-4]. Presently, Russia is ranked 1518th in the world in terms of number of scientific publications. However, in terms of number of cited published works, the country does not make it in the list of 20 leading countries [5].
Scientometrics are actively used in Russia and abroad as a reliable tool for evaluating scientific organizations and communities. Domestic scientometrics has been used by Russian Science Citation Index since 2005 [2, 6]. The main indicators used to measure the impact of research include publication activity of scientists in Russian and foreign scientific journals [6-8]. The number of publications and number of cited works included in global databases Web of Science and Scopus, as well as national information-analytical system Russian Science Citation Index are mainly analyzed [9]. Quantitative analysis of publication activity is a simple, yet reliable and intuitive method for determining the effectiveness of a research organization. Qualitative evaluation methods are characterized by accuracy, timeliness, representativeness and accessibility [3, 4].
Most industrialized nations, including Russia, use quantitative assessment of indicators [10]. For example, the UK uses three main criteria for analysis. The first of these involves the study of the newness, importance and degree of development of scientific subjects. The second examines the magnitude of the research results [11]. The third investigates the competitiveness of a research institution [12]. Examination procedure uses double-blind method [13]. The effectiveness of the activities of national universities in the UK is assessed via the Research Assessment Exercise program once every 4 years. The final rating data serve as the basis for further financing. A similar analysis in the preparation of economic decisions is also applied in the United States [14, 15].
In the US, the main method for evaluation of the effectiveness and safety of the technologies created are systematic literature reviews, whose analysis is carried out by organizations specially created for these purposes. The second evaluation method is cost-effectiveness analysis in which the costs of achieving additional year of life adjusted for quality is calculated. Based on the values obtained, decisions on further financing are taken. Clinical and economic analysis method is less widely used [16].
Medical science is the most cited in the global scientific community. The most famous scientometric database — Web of Science and Scopus, high-demand library fund — the U.S. National Library of Medicine [16, 17].
National databases are widely used by non-English speaking scientific communities. For example, the Chinese Social Sciences Citation Index covers most of the country's journals. At the same time, they try to increase the share of publications in European and American journals. Similar projects have been implemented in Taiwan (Taiwan Humanities Citation Index) and in Japan (Citation Database for Japanese Papers) [18].
Among scientometric indicators in foreign countries, the impact factor of publications, which is annually calculated at the Institute for Scientific Information (ISI) and published in the Journal Citation Reports is used. Since this criterion has been used in the West since the 70s of the 20th century, this impact factor is quite high for many European and American journals, unlike Russian journals [19]. For example, the 2015 impact factor of one of the most prestigious medical periodicals The New England Journal of Medicine is 55.87 [20]. European and American researchers try to publish in journals with very high impact factors to increase prestige and promote career development [16].
The resource estimate of the effectiveness and relevance of scientific research is important, but today it is a vulnerable area for many Russian medical and scientific institutions. Often, research institutions operate a poor assessment system for evaluating the prospects of patents and intellectual right management [21, 22]. Only in recent years that effective measures aimed at creating small innovative enterprises that can make productive use of patented innovative products and services were applied. To upgrade this direction, a mechanism for state guarantee of procurement of innovative products and formation of technology transfer centers was created [5].
In most economically developed countries in the world, intellectual property is commercialized [23]. The US is characterized by transfer of property rights to intellectual products or services created through state support to the private sector due to the fact that the state by law could not be the owners of such products or services. In the EU, UK and Japan, the government owns certain rights to intellectual activity and is actively involved in commercialization of research products [24].
The financial impact of scientific research in Russia is evaluated using two indicators: sources of income of the research institution and type of activity. From the side of the government, financial support is provided from federal budget as part of Russian government's programs "Healthcare Development" for 2013-2020 and "Development of Science and Technology" for 2013-2020, and in accordance with government directives and support grants [16]. At the same time, government financial support for research institutions decreases and the share of private capital increases as the final result comes nearer. Russia is ranked 9th by level of research funding in the world [5, 25]. Improving public funding with increase in the cost of medical science is necessary, but it is not enough for modernization of this sector.
Most foreign countries actively search for the most effective mechanisms for funding of medical research institutions. The most common are grants and state support, as well as sponsorship from research funds, councils and business organizations [26]. The importance of innovation for the public sector is growing. The US is characterized by funding of both fundamental and applied research. In most EU countries,
the government provides effective resourcing; private sector support is weak and irregular [27]. At the same time, there is joint performance assessment of both research institutions and agencies responsible for research development and funding [28]. Quantitative analysis approaches should be complemented with qualitative analysis approaches, particularly the criteria of efficiency and safety of innovative medical technologies [16].
Human resource development
Among all the performance criteria of institutions, evaluation of personnel potential in Russian medical research institutions receives the least number of indicators (only four). Data obtained in the end is insufficient for system analysis. However, the majority of performance indicators of any research institution and its competitiveness depend precisely on the human resource capacity of that institution.
By number of scientists, Russia is ranked 4th behind China, USA and Japan [5]. In recent years, a new theory accounts for a shift in approach relating staff to costs. This theory considers the staff as the most important resource of the effectiveness of any institution.
Qualitative and quantitative assessment indicators of human resource capacity reflect the degree of implementation of research programs, the effectiveness of the institution's structure and use of human resources, increase in productivity and quality of research. Using unrelated criteria with different analysis significance makes it difficult to obtain an objective final assessment of the entire institution [29]. Lack of strictly formal assessment of personnel potential of research medical institutions and databases, as well as qualitative factors affecting the result of activities pose challenges for the objective assessment of the sector [30]. Human potential assessment in foreign countries depends primarily on the status and quality of research activities of the scientist and research institution in the international community [31].
The US National Science Foundation applies scientific management to modernization of the staffing sector. They use lifelong learning, professional development motivation, manifestation of leadership skills and creative potential of employees [5, 32].
Enhancing the prestige of Russian science
Evaluation of indicators of integration into the global scientific space, dissemination of scientific knowledge and enhancement of the prestige of science for Russia is extremely urgent thanks to the obvious need to modernize this sector. Russians have recently been coming up with the idea of prestige of scientific work and academic status [33]. Socio-economic instability has led to sharp decline in the reputation of scientists. Owing to low wages, only about 9 % of Russians regard research profession as prestigious [5]. The ongoing loss of staff by many research institutions due to economic reasons reduces the efficiency of research and the overall level of institutions. This, of course, is reflected in the assessment of this sector. Translational barriers [34], unattractiveness of investment in medical science and lack of competitiveness of intellectual production when compared with economically developed countries are only a small list of problems to be addressed.
In most economically developed countries worldwide, public opinion polls clearly demonstrate the opposite results.
According to statistics, about 51 % of respondents in the US believe that a career as a scientist is highly prestigious, 25 % said it is very prestigious, and 20 % consider it as prestigious [5]. International experience shows that high social status of the research elite reflects the socio-economic level of the country and the pursued information policy [33].
In the European Union, issues of protection of rights of not only those involved in clinical trials, but also the animals that take part in the experiments, as well as the problem of changes in the legislation are traditionally important for the medical research community. Some European and American scientific communities are in favor of public access to information on patients who participated in clinical trials with the aim of further analysis and receipt of reliable data for further diagnosis and treatment of a wide range of diseases, especially cancer. The European scientific elite, with support from one of the leading research organizations in Germany — Max Planck Society — is in favor of open access to scientific publications for any person in accordance with the Berlin Declaration on Open Access to Knowledge in the Sciences and Humanities. To draw attention to these issues, research societies and institutions, which are widely represented on the Internet and receive enough response from a non-indifferent population, are being created [35-37].
Resourcing the activities of a research institution
To ensure effective monitoring of Russian research and medical institutions, indicators responsible for resourcing of the activities of research institutions are among the most important [33]. In recent years, Russia has taken a number of measures to modernize her research institutions, which, in contrast to Western countries, has led to a reduction in the number of researchers in the research sector [38].
Mainly quantitative indicators, which are apparently formal in nature, are used to analyze the sector. The number of young scientists under the age of 39 is one of the consistently low indicators in many research institutions. Sociological studies showed that there is little interest among young people in research careers due to low pay, poor prestige of the profession, lack of research funding, poor social conditions and increasing bureaucracy [39]. The average number of young researchers does not exceed 25 % of the entire number of researchers [5].
Intangible assets in the modern Russian scientific institution are becoming an important criterion of the effectiveness of the organization. They are accounted for in the balance sheet as non-current assets. Exclusive intellectual rights of an institution contribute to monopolization of the right to use such rights, including to receive income from transfer of the rights to the industrial sector. Presence of innovative intangible assets allows to pay royalties, while adding the cost into the cost of the assets [40]. However, the actual use of these assets is small compared with other Russian science sectors. Patented technologies are often not used in production.
The salary of researchers is also an important factor in the effectiveness of the activities of scientific institutions and its final evaluation [41]. In most economically developed countries of the world, there is increased research funding, thereby attracting more researchers [33].
According to the US National Science Foundation, 6065 % of funds are spent on salaries of researchers, graduate and undergraduate students, 12 % goes to the purchase of new equipment, 11 % is spent on education, about 6 % on technology transfer and only 1-3 % on administration [16, 42].
FINDINGS
In order to effectively implement the Strategy Of Development Of Medical Science In The Russian Federation For The Period Till 2025, approved by Order No 2580-r of the Government of the Russian Federation on 28 December 2012 [5], there is need to develop better performance criteria for evaluation of research medical institutions.
According to some researchers [43], mainly qualitative indicators resulting from examination should be used to evaluate the effectiveness of research institutions. However, most
experts recommend building a model that factors in constant monitoring, assessment algorithms, creation and maintenance of a personalized database of employees' register [20]. Such an analytical model will form the final assessment of research institutions and will contribute to decision-making with regards to prospects. Therefore, it should include a systematic approach and have data accumulation, processing and storage functions [44]. Based on the above, at this stage of developing methods for research evaluation, the task of developing an objective approach that would obtain final performance indicators for Russian research health institutions is extremely important.
References
1. Pronichkin SV, Tikhonov IP. Razrabotka sistemy kriteriev i metodicheskikh podkhodov k ekspertnoy otsenke effektivnosti deyatel'nosti nauchnykh organizatsii. Natsional'nye interesy: prioritety i bezopasnost'. 2013; 37: 13-8. Russian.
2. Bobkov AV, Katalazhnova IN, Pavlov IV. Monitoring nauchnoi deyatel'nosti nauchno-obrazovatel'noi sistemy. Sovremennye problemy nauki i obrazovaniya. 2009; 6 (1): 49-50. Russian.
3. Vyalkov AI, Glukhova EA, Martynchik SA. Model' monitoring formirovaniya innovatsionnykh kompetentsii kadrovogo sostava nauchnoi meditsinskoi organizatsii, orientirovannogo na sozdanie kriticheski vazhnykh tekhnologii. Sotsial'nye aspekty zdorov'ya naseleniya. 2015; 42 (2): 17. Russian.
4. Vyalkov AI, Martynchik SA, Polesskii VA. Metodologicheskie osnovy standartizatsii otsenochnoi deyatel'nosti meditsinskoi nauchnoi organizatsii. Zdravookhranenie Rossiiskoi Federatsii. 2010; 6: 3-7. Russian.
5. Rasporyazhenie Pravitel'stva RF ot 28.12.2012 N 2580-r "Ob utverzhdenii Strategii razvitiya meditsinskoi nauki v Rossiyskoi Federatsii na period do 2025 goda" [Internet]. [cited 2016 Jan 14]. Available from: http:.www.consultant.ru/document/cons_doc_ LAW_140249/. Russian.
6. Sharabchiev YuT. Produktivnost' uchenykh: instrumenty otsenki. Nauka i innovatsii. 2013; 1 (119): 4-8. Russian.
7. Vyalkov AI, Kucherenko VZ, Martynchik SA, Glukhova EA. Metodologicheskie osnovy upravleniya kachestvom deyatel'nosti meditsinskoi nauchnoi organizatsii v konkurentnoi srede. Problemy upravleniya zdravookhraneniem. 2010; 4: 14-9. Russian.
8. Kucherenko VZ, Martynchik SA, Bashkina EM. Tsitirovanie v nauke i rol' elektronnykh resursov i otsenochnykh instrumentov. Problemy upravleniya zdravookhraneniem. 2009; 46 (3): 14-21. Russian.
9. Bykov VL. Bibliometriya vchera, segodnya i zavtra: kolichestvennye pokazateli i nauchnaya etika. Morfologiya. 2013; 144 (4): 7-13. Russian.
10. Mushlin AI, Ghomrawi HM. Comparative effectiveness research: a cornerstone of healthcare reform? Trans Am Clin Climatol Assoc. 2010; 121: 141-54; discussion 154-5.
11. VanLare JM, Conway PH, Sox HC. Five next steps for a new national program for comparative-effectiveness research. N Engl J Med. 2010 Mar 18; 362 (11): 970-3.
12. Platt R, Wilson M, Chan KA, Benner JS, Marchibroda J, McClellan M. The new Sentinel Network — improving the evidence of medical-product safety. N Engl J Med. 2009 Aug 13; 361 (7): 645-7.
13. Higher Education Funding Council for England [Internet]. 2014 Research Excellence Framework [updated 2014 Dec 18]. Assessment criteria and level definitions [updated 2014 Dec 12; cited 2016 Jan 15]. Available from: http://www.ref.ac.uk/panels/as sessmentcriteriaandleveldefinitions/.
14. Larichev OI. Verbal'nyj analiz reshenii. Petrovskii AB, editor. Moscow: Nauka, 2006. Chapter 9.5.2. Organizacionnye kriterii. Russian.
15. Smyth RL. A risk adapted approach to the governance of clinical trials. BMJ. 2011 Oct 25; 343: d6756.
16. Perkhov VI, Stebunova RV, Yankevich DS, Yurkin YuYu. Analiz zarubezhnogo opyta finansirovaniya i organizatsii nauchnykh issledovanii v oblasti zdravookhraneniya. Menedzher zdravo-
okhraneniya. 2013; 7: 49-56. Russian.
17. Califf RM. The patient-centered outcomes research network: a national infrastructure for comparative effectiveness research. N C Med J. 2014 May-Jun; 75 (3): 204-10.
18. Lee D, Kim S, Cha SH. Evaluating the effectiveness of research centers and institutes in universities: Disciplines and life cycle stages. KEDI J Educ Policy. 2014; 11 (1): 119.
19. Ranking of Research Institutions SIR World Report 2010 Health Sciences [Internet]. SCImago Journal & Country Rank. [cited 2016 Jan 15]. Available from: http:www.scimagojr.com.
20. Selby JV, Lipstein SH. PCORI at 3 years — progress, lessons, and plans. N Engl J Med. 2014 Feb 13; 370 (7): 592-5.
21. Kucherenko VZ, Martynchik SA, Osokina OV. Informatsion-nye sredstva kontrollinga proizvodstvenno-khozyaystvennoi deyatel'nosti meditsinskoi organizatsii. Metodicheskoe posobie. Ekaterinburg: UGMA, 2008. Russian.
22. Martynchik SA, Osokin RS, Osokina OV. Organizatsion-naya osnova monitoringa proizvodstvenno-khozyaystvennoi deyatel'nosti meditsinskoi organizatsii. In: Polesskii VA, editor. Monitorirovanie sostoyaniya zdorov'ya, kachestva i obraza zhizni naseleniya Rossii. Vliyanie povedencheskikh faktorov riska na zdorov'e naseleniya: Proceedings of the Vserossiyskaya nauchno-prakticheskaya konferentsiya; 2011 Jun 06 — 2011 Jul 06; Moscow, Russia. Moscow; 2011. p 233-5. Russian.
23. Zerhouni EA, Alving B. Clinical and translational science awards: a framework for a national research agenda. Transl Res. 2006 Jul; 148 (1): 4-5.
24. Academy of Medical Sciences [Internet]. A new pathway for the regulation and governance of health research, 2011. [cited 2016 Jan 13]. Available from: http://www.acmedsci.ac.uk/policy/ policy-projects/.
25. Martynchik SA. Planirovanie gosudarstvennogo (municipal'nogo zadaniya) v zdravookhranenii. Zdravookhranenie. 2012; 4: 18-23. Russian.
26. Sully BG, Julious SA, Nicholl J. A reinvestigation of recruitment to randomised, controlled, multicenter trials: a review of trials funded by two UK funding agencies. Trials. 2013 Jun 9; 14: 166.
27. Arnold E, Brown N, Eriksson A, Jansson T, Muscio A, Nählinder J, et al. The Role of Industrial Research Institutes in the National Innovation System. Stockholm: VINNOVA; 2007.
28. Warlow C. A new NHS research strategy. Lancet. 2006 Jan 7; 367 (9504): 12-3.
29. Chernova AA, Kurshakova NB. Podkhody k otsenke rezul'tativnosti sistemy menedzhmenta kachestva v nauchno-issledovatel'skikh organizatsiyakh. Innovatsionnaya ekonomika i obshchestvo. 2015; 3 (9): 90-5. Russian.
30. Martynchik SA, Glukhova EA, Galustova LR. Trebovaniya k postroeniyu sistemy dlya otsenki rezul'tativnosti i potentsiala nauchnoi deyatel'nosti na urovne organizatsii. Sotsial'nye aspekty zdorov'ya naseleniya. 2013; 32 (4): 10. Russian.
31. Sveiby K-E. What is Knowledge Management? [Internet]. 1996 Mar [updated 2001 Apr; cited 2016 Jan 15]. Available from: http://www.sveiby.com/articles/KnowledgeManagement.html.
32. Kahn C, McGourty S. Performance Management at R&D Organizations. Practices and Metrics from Case Examples. Bedford, MA: The MITRE Corporation; 2009.
33. Fadeeva IM, Osipova OYu, Fadeeva ES. Kompetentsii molodykh uchenykh dlya nauchno-issledovatel'skoi deyatel'nosti i akademicheskoi kar'ery. Integratsiya obrazovaniya. 2012; 1: 7-13. Russian.
34. Ipatova OM, Medvedeva NV, Archakov AI, Grigor'ev AI. Translyatsionnaya meditsina — put' ot fundamental'noi biomeditsinskoi nauki v zdravookhranenie. Vestnik Rossiyskoi akademii meditsinskikh nauk. 2012; 6: 57-65. Russian.
35. Anderson ML, Califf RM, Sugarman J; participants in the NIH Health Care Systems Research Collaboratory Cluster Randomized Trial Workshop. Ethical and regulatory issues of pragmatic cluster randomized trials in contemporary health systems. Clin Trials. 2015 Jun; 12 (3): 276-86. doi: 10.1177/1740774515571140.
36. Frewer LJ, Coles D, van der Lans IA, Schroeder D, Champion K, Apperley JF. Impact of the European clinical trials directive on prospective academic clinical trials associated with BMT. Bone Marrow Transplant. 2011 Mar; 46 (3): 443-7.
37. Neaton JD, Babiker A, Bohnhorst M, Darbyshire J, Denning E, Frishman A, et al. Regulatory impediments jeopardizing the conduct of clinical trials in Europe funded by the National Institutes of Health. Clin Trials. 2010 Dec; 7 (6): 705-18.
38. Gokhberg LM, Zaichenko SA, Kitova GA, Kuznetsova TE. Nauchnaya politika: global'nyi kontekst i rossiyskaya praktika. M.: NIU VShE; 2011. Russian.
Литература
1. Проничкин С. В., Тихонов И. П. Разработка системы критериев и методических подходов к экспертной оценке эффективности деятельности научных организаций. Нац. интересы: приоритеты и безопасность. 2013; 37: 13-8.
2. Бобков А. В., Каталажнова И. Н., Павлов И. В. Мониторинг научной деятельности научно-образовательной системы. Совр. пробл. науки и обр. 2009; 6 (1): 49-50.
3. Вялков А. И., Глухова Е. А., Мартынчик С. А. Модель мониторинга формирования инновационных компетенций кадрового состава научной медицинской организации, ориентированного на создание критически важных технологий. Соц. аспекты здоровья насел. 2015; 42 (2): 17.
4. Вялков А. И., Мартынчик С. А., Полесский В. А. Методологические основы стандартизации оценочной деятельности медицинской научной организации. Здравоохр. РФ. 2010; 6: 3-7.
5. Распоряжение Правительства РФ от 28.12.2012 № 2580-р «Об утверждении Стратегии развития медицинской науки в Российской Федерации на период до 2025 года» [Интернет]. [Дата обращения 14.01.2016]. Доступно по: http://www.consultant.ru/document/cons_doc_LAW_140249/.
6. Шарабчиев Ю. Т. Продуктивность ученых: инструменты оценки. Наука и инновации. 2013; 1 (119): 4-8.
7. Вялков А. И., Кучеренко В. З., Мартынчик С. А., Глухова Е. А. Методологические основы управления качеством деятельности медицинской научной организации в конкурентной среде. Пробл. упр. здравоохр. 2010; 4: 14-9.
8. Кучеренко В. З., Мартынчик С. А., Башкина Е. М. Цитирование в науке и роль электронных ресурсов и оценочных инструментов. Пробл. упр. здравоохр. 2009; 46 (3): 14-21.
9. Быков В. Л. Библиометрия вчера, сегодня и завтра: количественные показатели и научная этика. Морфология. 2013; 144 (4): 7-13.
10. Mushlin AI, Ghomrawi HM. Comparative effectiveness research: a cornerstone of healthcare reform? Trans Am Clin Climatol Assoc. 2010; 121: 141-54; discussion 154-5.
11. VanLare JM, Conway PH, Sox HC. Five next steps for a new national program for comparative-effectiveness research. N Engl J Med. 2010 Mar 18; 362 (11): 970-3.
12. Platt R, Wilson M, Chan KA, Benner JS, Marchibroda J, McClellan M. The new Sentinel Network — improving the evidence of medical-product safety. N Engl J Med. 2009 Aug 13; 361 (7): 645-7.
13. Higher Education Funding Council for England [Internet]. 2014 Research Excellence Framework [updated 2014 Dec 18]. Assessment criteria and level definitions [updated 2014 Dec 12;
39. Fadeeva IM, Shamanaev PA, Sokolova MYu. Upravlenie kadrovym potentsialom issledovatel'skogo universiteta na osnove informatsionnykh sistem. Universitetskoe upravlenie: praktika i analiz. 2011; 6: 23-31. Russian.
40. Zubova LG, Andreeva ON, Antropova OA. K voprosu o rezul'tativnosti deyatel'nosti gosudarstvennykh nauchnykh organizatsii (po otsenkam sotsiologicheskogo monitoringa 20052011 gg.). Innovatsii. 2012; 12 (170): 51-60. Russian.
41. Starodubov VI, Kurakova NG, Tsvetkova LA, Markusova VA. O novykh kriteriyakh otsenki rossiiskoi akademicheskoi i vuzovskoi meditsinskoi nauki. Meditsinskoe obrazovanie i professional'noe razvitie. 2011; 1: 16. Russian.
42. Altbach PG, Reisberg L, Yudkevich M, Androushchak G, Pacheco IF, editors. Paying the Professoriate: A Global Comparison of Compensation and Contracts. New York: Routledge, 2012.
43. Bondarenko TV. «Rabota nad oshibkami», ili kak razrabotat' effektivnuyu sistemu KPI. Menedzhment segodnja. 2010; 4: 23641. Russian.
44. Martynchik SA, Glukhova EA, Galustova LR. Trebovaniya k postroeniyu sistemy dlya otsenki rezul'tativnosti i potentsiala nauchnoi deyatel'nosti na urovne organizatsii. Sotsial'nye aspekty zdorov'ya naseleniya. 2013; 32 (4): 10. Russian.
cited 2016 Jan 15]. Available from: http://www.ref.ac.uk/panels/as sessmentcriteriaandleveldefinitins/.
14. Ларичев О. И. Вербальный анализ решений. Петровский А. Б., редактор. М.: Наука, 2006. Раздел 9.5.2 Организационные критерии.
15. Smyth RL. A risk adapted approach to the governance of clinical trials. BMJ. 2011 Oct 25; 343: d6756.
16. Перхов В. И., Стебунова Р. В., Янкевич Д. С., Юркин Ю. Ю. Анализ зарубежного опыта финансирования и организации научных исследований в области здравоохранения. Менеджер здравоохр. 2013; 7: 49-56.
17. Califf RM. The patient-centered outcomes research network: a national infrastructure for comparative effectiveness research. N C Med J. 2014 May-Jun; 75 (3): 204-10.
18. Lee D, Kim S, Cha SH. Evaluating the effectiveness of research centers and institutes in universities: Disciplines and life cycle stages. KEDI J Educ Policy. 2014; 11 (1): 119.
19. Ranking of Research Institutions SIR World Report 2010 Health Sciences [Internet]. SCImago Journal & Country Rank. [cited 2016 Jan 15]. Available from: http:www.scimagojr.com.
20. Selby JV, Lipstein SH. PCORI at 3 years — progress, lessons, and plans. N Engl J Med. 2014 Feb 13; 370 (7): 592-5.
21. Кучеренко В. З., Мартынчик С. А., Осокина О. В. Информационные средства контроллинга производственно-хозяйственной деятельности медицинской организации. Методическое пособие. Екатеринбург: УГМА, 2008.
22. Мартынчик С. А., Осокин Р. С., Осокина О. В. Организационная основа мониторинга производственно-хозяйственной деятельности медицинской организации. В: Полесский В. А., редактор. Мониторирование состояния здоровья, качества и образа жизни населения России. Влияние поведенческих факторов риска на здоровье населения: Тезисы Всероссийской научно-практической конференции; 2011 июнь 06 — 2011 июль 06; Москва, Россия. М., 2011. с. 233-5.
23. Zerhouni EA, Alving B. Clinical and translational science awards: a framework for a national research agenda. Transl Res. 2006 Jul; 148 (1): 4-5.
24. Academy of Medical Sciences [Internet]. A new pathway for the regulation and governance of health research, 2011. [cited 2016 Jan 13]. Available from: http://www.acmedsci.ac.uk/policy/ policy-projects/.
25. Мартынчик С. А. Планирование государственного (муниципального задания) в здравоохранении. Здравоохранение. 2012; 4: 18-23.
26. Sully BG, Julious SA, Nicholl J. A reinvestigation of recruitment to
randomised, controlled, multicenter trials: a review of trials funded by two UK funding agencies. Trials. 2013 Jun 9; 14: 166.
27. Arnold E, Brown N, Eriksson A, Jansson T, Muscio A, Nählinder J, et al. The Role of Industrial Research Institutes in the National Innovation System. Stockholm: VINNOVA; 2007.
28. Warlow C. A new NHS research strategy. Lancet. 2006 Jan 7; 367 (9504): 12-3.
29. Чернова А. А., Куршакова Н. Б. Подходы к оценке результативности системы менеджмента качества в научно-исследовательских организациях. Инновац. эконом. и о-во. 2015; 3 (9): 90-5.
30. Мартынчик С. А., Глухова Е. А., Галустова Л. Р. Требования к построению системы для оценки результативности и потенциала научной деятельности на уровне организации. Соц. аспекты здоровья насел. 2013; 32 (4): 10.
31. Sveiby K-E. What is Knowledge Management? [Internet]. 1996 Mar [updated 2001 Apr; cited 2016 Jan 15]. Available from: http:// www.sveiby.com/articles/KnowledgeManagement.html.
32. Kahn C, McGourty S. Performance Management at R&D Organizations. Practices and Metrics from Case Examples. Bedford, MA: The MITRE Corporation; 2009.
33. Фадеева И. М., Осипова О. Ю., Фадеева Е. С. Компетенции молодых ученых для научно-исследовательской деятельности и академической карьеры. Интеграция обр. 2012; 1: 7-13.
34. Ипатова О. М., Медведева Н. В., Арчаков А. И., Григорьев А. И. Трансляционная медицина — путь от фундаментальной биомедицинской науки в здравоохранение. Вестн. РАМН. 2012; 6: 57-65.
35. Anderson ML, Califf RM, Sugarman J; participants in the NIH Health Care Systems Research Collaboratory Cluster Randomized Trial Workshop. Ethical and regulatory issues of pragmatic cluster randomized trials in contemporary health systems. Clin Trials. 2015 Jun; 12 (3): 276-86. doi: 10.1177/1740774515571140.
36. Frewer LJ, Coles D, van der Lans IA, Schroeder D, Champion K, Apperley JF. Impact of the European clinical trials directive on prospective academic clinical trials associated with BMT. Bone Marrow Transplant. 2011 Mar; 46 (3): 443-7.
37. Neaton JD, Babiker A, Bohnhorst M, Darbyshire J, Denning E, Frishman A, et al. Regulatory impediments jeopardizing the conduct of clinical trials in Europe funded by the National Institutes of Health. Clin Trials. 2010 Dec; 7 (6): 705-18.
38. Гохберг Л. М., Заиченко С. А., Китова Г. А., Кузнецова Т. Е. Научная политика: глобальный контекст и российская практика. М.: НИУ ВШЭ; 2011.
39. Фадеева И. М., Шаманаев П. А., Соколова М. Ю. Управление кадровым потенциалом исследовательского университета на основе информационных систем. Университетское управление: практика и анализ. 2011; 6: 23-31.
40. Зубова Л. Г., Андреева О. Н., Антропова О. А. К вопросу о результативности деятельности государственных научных организаций (по оценкам социологического мониторинга 20052011 гг.). Инновации. 2012; 12 (170): 51-60.
41. Стародубов В. И., Куракова Н. Г., Цветкова Л. А., Маркусо-ва В. А. О новых критериях оценки российской академической и вузовской медицинской науки. Мед. обр. и проф. развитие. 2011; 1: 16.
42. Altbach PG, Reisberg L, Yudkevich M, Androushchak G, Pacheco IF, editors. Paying the Professoriate: A Global Comparison of Compensation and Contracts. New York: Routledge, 2012.
43. Бондаренко Т. В. «Работа над ошибками», или как разработать эффективную систему KPI. Менеджмент сегодня. 2010; 4: 236-41.
44. Мартынчик С. А., Глухова Е. А., Галустова Л. Р Требования к построению системы для оценки результативности и потенциала научной деятельности на уровне организации. Соц. аспекты здоровья насел. 2013; 32 (4): 10.
