CC BY
34
KEY PERFORMANCE INDICATORS FOR HEALTHCARE RESEARCH ORGANIZATIONS BETWEEN 2011 AND 2015
Aniskevich AS1 ^ Halfin RA2
1 Ministry of Healthcare of the Russian Federation, Moscow, Russia
2 Higher School of Healthcare Administration,
I. M. Sechenov First Moscow State Medical University, Moscow, Russia
In this work we identify 16 key indicators to evaluate the performance of healthcare research organizations. These indicators comprehensively characterize such aspects of performance as research output and relevance, human resource development, integration into the international scientific community, distribution of scientific knowledge, promotion of the prestige of science, and resource provision. Below, we review the existing classification of medical research institutions and their key features. We present the results of the comprehensive performance evaluation of healthcare research organizations. We demonstrate the significance of the proposed indicators that accurately reflect the output and relevance of scientific research and stress that indicators currently used for performance evaluation are insufficient. We also emphasize the need for a systemic approach to personnel capacity assessment and confirm the importance of additional evaluation criteria that amount to 37.5 % of all key indicators.
Keywords: research organization performance, key performance indicators, healthcare research organization, state-funded research organization
gg Correspondence should be addressed: Anna Aniskevich
Rakhmanovsky per., d. 3, Moscow, Russia, 127994; anna.aniskevich@yandex.ru
Recieved: 10.02.2017 Accepted: 17.02.2017
КЛЮЧЕВЫЕ ПОКАЗАТЕЛИ РЕЗУЛЬТАТИВНОСТИ ДЕЯТЕЛЬНОСТИ НАУЧНЫХ ОРГАНИЗАЦИЙ В СФЕРЕ ЗДРАВООХРАНЕНИЯ ЗА 2011-2015 ГГ.
А. С. Анискевич1 Р. А. Хальфин2
1 Министерство здравоохранения Российской Федерации, Москва
2 Высшая школа управления здравоохранением,
Первый Московский государственный медицинский университет имени И. М. Сеченова, Москва
Определены 16 ключевых критериев результативности деятельности научных организаций в сфере здравоохранения, характеризующих результативность и востребованность научных исследований, развитие кадрового потенциала, интеграцию в мировое научное пространство, распространение научных знаний и повышение престижа науки, ресурсное обеспечение деятельности организаций. В рамках классификационного подхода рассмотрены медицинские научные учреждения, показаны их особенности. Представлена системная оценка результативности деятельности научных организаций в сфере здравоохранения. Продемонстрирована высокая значимость показателей, отражающих результативность и востребованность научных исследований. Отмечены недостаточность применяемых параметров и отсутствие системного подхода к анализу кадрового потенциала. Доказана важная роль дополнительных критериев оценки, составивших 37,5 % общей доли ключевых показателей.
Ключевые слова: результативность деятельности научной организации, ключевые показатели результативности, научная организация в сфере здравоохранения, государственная научная организация
123 Для корреспонденции: Анискевич Анна Сергеевна
Рахмановский пер., д. 3, г. Москва, 127994; anna.aniskevich@yandex.ru
Статья получена: 10.02.2017 Статья принята в печать: 17.02.2017
Evaluating the performance of healthcare research institutions is exceptionally significant and of vital importance for the further development of Russia's research potential. The importance of this analysis was dictated by the need to increase efficiency amidst modernization of the public sector of science [1, 2]. On the other hand, modern theoretical and practical developments and tools used to assess the output of medical research institutions often do not facilitate comprehensive monitoring of their activities.
The peculiarity of the Russian methodological approach to analysis of the performance of healthcare research
organizations is that a large number of criteria are used, unlike what is obtainable in a number of Western countries [3, 4]. A typical evaluation method applied in Russia [2] is based predominantly on quantitative rather than qualitative indicators. Additional criteria are used for systemic monitoring, including for institutions with bed space facilities. Since the outcome of an analysis directly determines financing decisions, identification of key criteria is particularly important.
The activities of medical research institutions have characteristic specificity and important differences from the activities of other research institutions [5, 6]. Performance
indicators approved for healthcare research institutions include monitoring of the following areas:
• output and relevance of scientific research;
• human resource development;
• integration into the international scientific community, dissemination of scientific knowledge and promotion of the prestige of science;
• resource support for the activities of the research institution.
METHODS
Performance indicators for 55 research institutions subordinated to the Russian Ministry of Health for the period 2011-2015 were investigated.
In accordance with the accepted classification, the above research institutions subordinated to the Russian Health Ministry (n = 55) were divided into three groups: those with bed space facilities (n = 41), those without bed space facilities (n = 4), expert and socially important organizations (n = 10).
Statistical analysis was performed using software package Statistica 10.0. The Kolmogorov-Smirnov test by Lilliefors was used to test statistical hypotheses about type of distribution. It was found that in all cases, the data were not subject to the normal law of distribution; so nonparametric tests were used for statistical analysis.
Kruskal-Wallis test and the median test were used to determine the key performance indicators for healthcare research organizations. Additionally, the years 2011 and 2015 were compared in terms of Mann-Whitney U test and it was found that these figures change sufficiently enough.
Correlation analysis was performed using the Spearman's rank correlation coefficient. Indicators that change over time were included in the correlation analysis with background indicators for 2011, since it is the starting point of reference. P-value < 0.05 was taken as the level of statistical significance.
RESULTS
Most research institutions (74.5 %) under the Russian Ministry of Health have bed space facilities. However, this does not simplify the assessment procedure — you cannot apply indicators characterizing healthcare delivery to the other research organizations. In this regard, additional criteria are applied in analyzing the performance of research institutions with bed space facilities.
Statistical analysis of the 5-year dynamics (2011-2015) of performance criteria uncovered 16 key performance indicators (p < 0.05). The structure of the criteria earmarked and their proportion in the total number of parameters are shown in Fig. 1 and 2.
For the period 2011-2015, the following key indicators were identified.
Key performance indicators and relevance of scientific research:
1. Cumulative impact factor of journals where the articles of the organization are published. It should be noted that this indicator was found to have a moderate correlation with the number of Doctors of Sciences (the highest academic degree in Russia and many other post-Soviet states obtained after obtaining a PhD degree) (rs = 0.455, p < 0.01), with intramural (rs = 0.449, p < 0.01) and extramural (rs = 0.411, p < 0.01) current expenditure on scientific research and development, and with the number of articles of the organization published in journals indexed in Web of Science (rs = 0.406, p < 0.01).
2. Number of used intellectual property transferred under a license agreement. It was found that this indicator has moderate correlation with intramural current expenditure on research and development, including expenditure on exploratory research (rs = 0.475, p < 0.01).
3. Number of used intellectual property contributed into the authorized capital.
7 -
Z 6 -
E <
100
©
©
®
50
Indicators:
key
other
Areas of expertise:
J Output and relevance of scientific research;
^ Integration into the global scientific community, dissemination of scientific knowledge and promotion of the prestige of science;
^ Resource support for the activities of the research institution;
[J;] Additional indicators, including for evaluation of the performance of research institutions that have bed space facilities
Fig. 1. Structure of key performance Indicators for healthcare research Institutions
(1 Output and relevance of scientific research @ Human resource development
@ Integration into the global scientific community, dissemination of scientific knowledge and promotion of the prestige of science
@ Resource support for the activities of the research institution
(5 Additional indicators, including for evaluation of the performance of research institutions that have bed space facilities
Fig. 2. Proportion of key performance indicators for healthcare research institutions in the total number of indicators in each field
8
5
4
3
2
0
0
4. Number of small innovative enterprises created with the participation of that institution.
5. Cumulative average staff number at small innovative enterprises.
6. Cumulative income at small innovative enterprises.
7. Financial impact of the research institution by income sources. This indicator was found to have a weak relationship with intramural current expenditure on research and development (rs = 0.302, p < 0.05), including a positive correlation with basic research (rs = 0.351, p < 0.01) and negative correlation with exploratory research (rs = -0.324, p < 0.05).
Key indicator: integration into the global research community, dissemination of scientific knowledge and enhancement of prestige of science
The number of traffic (visits) to official sites and/or web pages of the institution on the Internet is defined by a single key indicator. The criterion was found to have a moderate correlation with the number of positive and neutral mentions of the institution in the federal media (rs = 0.357, p < 0.01), including in online publications (rs = 0.325, p < 0.05), and with the number of publications of the organization in journals indexed in the Russian Science Citation Index (rs = 0.339, p < 0.05).
Key indicators: resource support for the research institution:
1. Expenditure on fixed assets and intangible assets, including buildings and structures, machinery and equipment. The indicator was found to have a moderate correlation with financial performance of research institution by type of work performed and services rendered, including production goods, works and services (rs = 0.491, p < 0.01).
2. Intramural current expenditure on basic research. This indicator correlated with the financial performance of research institution by type of work performed and services rendered: by number of research and developments (rs = 0.548, p < 0.01); with number of employees engaged in research and development (rs = 0.516, p < 0.01), including researchers (rs = 0.418, p < 0.01), among whom are PhD holders (rs = 0.405, p < 0.01) and Doctors of Sciences (rs = 0.368, p < 0.01) aged not above 39 years (rs = 0.505, p < 0.01).
All the key indicators of resource support for a research institution were interconnected with the financial performance of the research institution by type of work performed, by services rendered and by income sources.
of scientific, design and technological works (rs = -0.273, p < 0.05), with the number of positive and neutral mentions of the organization in the media (rs = -0.311, p < 0.05), with financial performance of the research institution by type of work performed and services rendered (by research and development) (rs = -0.308, p < 0.05), and with the number of prepared draft healthcare delivery standards and procedures, clinical practice guidelines and other regulations (rs = -0.276, p < 0.05).
3. Number of innovative medical technologies used at the institution and approved by the Scientific Council of the Russian Ministry of Health. This indicator correlated with the number of publications of the organization in journals indexed in Scopus (rs = 0.367, p < 0.01), with the number of intellectual property created (rs = 0.406, p < 0.01), including those with state registration and/or legal protection in Russia (rs = 0.392, p < 0.01), and with the percentage of residents of other regions that received high-tech health care (HHC) at that institution (rs = 0.360, p < 0.01).
4. Federal nature of the institution: the percentage of people from other regions who have received specialized medical care in that institution. This indicator moderately correlated with the number of people from other regions who received HHC (rs = 0.474, p < 0.01), and with the number of Federal subjects of Russia, whose residents were treated in the reporting year (rs = 0.437, p < 0.01).
5. Percentage of medical care expenses under mandatory health insurance (percentage of the total cost). This indicator correlated with the number of intellectual property created (rs = 0.350, p < 0.01), including those with state registration and/ or legal protection in Russia (rs = 0.459, p < 0.01), with financial performance of the research institution by income sources (rs = 0.369, p < 0.01), with intramural current expenditure on basic research (rs = 0.417, p < 0.01).
6. Number of prepared draft healthcare delivery standards, draft healthcare delivery procedures, clinical practice guidelines (treatment protocols) and other regulations. This indicator was found to be correlated with the number of publications of the organization in journals indexed in Scopus (rs = 0.384, p < 0.01), with the financial performance of the research organization by type of work performed and services rendered, including educational services (rs = 0.394, p < 0.01), specialist training under continuing professional education programs on unique technologies along the profile of that research institution (rs = 0.449, p < 0.01), and with the percentage of residents of other regions who received HHC (r = 0.358, p < 0.01).
Key additional performance indicators for a research institution:
1. Number of research critical technologies from the list approved by the Scientific Council of the Russian Ministry of Health. This indicator correlated with the number of publications in journals indexed in Scopus (rs = 0.367, p < 0.01), and with the number of innovative medical technologies used at the institution and approved by the Scientific Council of the Russian Ministry of Health (rs = 0.356, p < 0.01). The indicator was found to be weakly correlated with the number of positive and neutral mentions of the organization in the federal media (rs = 0.295, p < 0.05), including in the federal print media, television and radio (rs = 0.291, p < 0. 05), and with the number of researchers sent to work in leading Russian and international research and educational organizations (rs = 0.268, p < 0.05).
2. The proportion of highly skilled medical doctors out of the total number of medical doctors. Curiously, this indicator was found to have a negative correlation with the total number
DISCUSSION
Analysis of performance indicators for research institutions subordinated to the Russian Health Ministry identified 16 key criteria, most (43.75 %) of which characterize the output and relevance of scientific research.
Scientometric analysis, which is of particular importance for systemic evaluation [7-9], revealed that among the criteria analyzed (number of publications in journals indexed in the Russian Science Citation Index, Web of Science and Scopus, total number of citations of publications of the institution, etc.), the cumulative impact factor of journals where the articles of the organization are published is a key indicator. This is probably due to a more careful selection of journals — since publications in top-ranking journals, including foreign journals, increase not only the citation of authors, but also the interest of foreign colleagues in Russian research and future collaboration.
The innovative development model of Russian science involves closer relationship with the economic sector [10]. Against this background, a special role belongs to small innovative enterprises and key indicators reflecting their activities. However, over the studied period — 2011-2015 — medical research institutions showed virtually no interest in commercialization of their intellectual products. It was found that small innovative enterprises created only 14.5 % of organizations with bed space facilities.
The standard human resource evaluation for healthcare research organizations includes four quantitative indicators, none of which is a key indicator. The absence of qualitative factors and the different importance of investigated parameters create obstacles to a full-fledged analysis. According to a number of researchers [5, 11], an effective human resource evaluation requires a systemic approach, which includes monitoring of professional competence, learning ability, conditions and wages, and many other indicators.
Since the output of institutions depends on the research performed and other works related to the main activities of the institutions, as well as on conformity of the state of national science to modern world standards [12-15], there is need to develop quality criteria for a particular area.
During the period under consideration — 2011-2015 — despite a decline in the number of mentions of research organizations in the media, a sharp increase in visits to the official websites of medical research institutions was revealed. Presumably, this is due to increased online activity by interested persons.
The use of additional performance criteria for evaluation of healthcare research organizations has proved its worth — they
accounted for 37.5 % of the total number of key indicators. It is curious to note that the criterion reflecting the number of used scientific critical technologies from the list approved by the Scientific Council of the Russian Health Ministry correlates with the number of publications of the organization in journals indexed in Scopus and with the number of researchers sent to work in leading Russian and international research and educational organizations.
CONCLUSIONS
In assessing the performance of healthcare research organizations in the period 2011-2015, 16 key performance indicators were identified, among which are especially significant criteria, reflecting the output and relevance of scientific research.
During the evaluation, the additional criteria were found to be relevant. Data obtained clearly show that the used research critical and innovative medical technologies from the list approved by the Scientific Council of the Russian Health Ministry, prepared draft healthcare delivery standards, procedures and clinical practice guidelines are crucially important for further development of the scientific potential of institutions.
For institutions with bed space facilities, the proportion of highly qualified medical doctors, the percentage of people from other regions who have received specialized medical care, and the percentage of medical care expenses under mandatory health insurance served as additional key indicators.
It was shown that the existing standard indicators for human resource analysis are insufficient and that there is need to come up not only with quantitative but also with qualitative criteria.
References
1. Resolution of the Russian Government No. 312 of April 8, 2009 (revised 2016 December 29) "Ob otsenke i o monitoringe rezul'tativnosti deyatel'nosti nauchnykh organizatsii, vypolnya-yushchikh nauchno-issledovatel'skie, opytno-konstruktorskie i tekhnologicheskie raboty grazhdanskogo naznacheniya". Russian.
2. Order of the Russian Ministry of Education and Science No. 161 of March 5, 2014 (revised 2015 July 17) "Ob utverzhdenii tipovogo polozheniya o komissii po otsenke rezul'tativnosti deyatel'nosti nauchnykh organizatsii, vypolnyayushchikh nauchno-issledova-tel'skie, opytno-konstruktorskie i tekhnologicheskie raboty grazhdanskogo naznacheniya, i tipovoi metodiki otsenki rezul'-tativnosti deyatel'nosti nauchnykh organizatsii, vypolnyayushchikh nauchno-issledovatel'skie, opytno-konstruktorskie i tekhno-logicheskie raboty grazhdanskogo naznacheniya". Russian.
3. Pronichkin SV, Tikhonov IP. Razrabotka sistemy kriteriev i metodicheskikh podkhodov k ekspertnoy otsenke effektivnosti deyatel'nosti nauchnykh organizatsii. Natsional'nye interesy: prioritety i bezopasnost'. 2013; (37): 13-8. Russian.
4. Filonyuk VA, Sachek MM, Malakhova IV, Dudina TV, Elkina AI. Kompleksnaya otsenka rezul'tativnosti meditsinskoi nauki i opredelenie reitinga nauchnykh organizatsii za rubezhom i v Belarusi. Voprosy organizatsii i informatizatsii zdravookhraneniya. 2013; (2): 20-30. Russian.
5. Vyalkov AI, Glukhova EA, Martynchik SA. [Monitoring model for developing personnel innovative competences of the scientific medical organization, focused on producing critically important technologies]. Sotsial'nye aspekty zdorov'ya naseleniya. 2015; 42 (2): 17. Russian.
6. Martynchik SA, Glukhova EA, Galustova LR. [Requirements for developing a system to assess performance and potential of scientific activities at the organizational level]. Sotsial'nye aspekty zdorov'ya naseleniya. 2013; 32 (4): 10. Russian.
7. Oganov RG, Trushchelev SA. [Scientometric approaches to
the analysis of research results]. Kardiovaskulyarnaya terapiya i profilaktika. 2012; 11 (2): 90-5. Russian.
8. Starodubov VI, Kuznetsov SL, Kurakova NG, Tsvetkova LA, Aref'ev P.G., Ivanov A.V., et al. [Publication activity of the Russian medicine in focus of national science policy: estimating the feasibility of policy targets]. Vestn Ross Akad Med Nauk. 2013; (3): 8-14. Russian.
9. Sharabchiev YuT. Produktivnost' uchenykh: instrumenty otsenki. Nauka i innovatsii. 2013; 119 (1): 4-8. Russian.
10. Zubova LG, Andreeva ON, Antropova OA. [To the question of the efficient activities of the public R&D institutions (basing on sociological monitoring of 2005-2011 years)]. Innovatsii. 2012; 170 (12): 51-60. Russian.
11. Kucherenko VZ, Martynchik SA, Glukhova EA. Ekspertnoe otsenivanie i indikatory kachestva deyatel'nosti spetsialistov nauchnoi (meditsinskoi) organizatsii po profilyu obshchestvennoe zdorov'e i upravlenie zdravookhraneniem. Problemy upravleniya zdravookhraneniem. 2010; (5): 24-9. Russian.
12. Kuznetsova TE. Pravovoi faktor nauchno-tekhnicheskogo razvitiya Rossii: oblast' neopredelennosti. In: Alakhverdyan AG, Semenova NN, Yurevich AV, editors. Naukovedenie i novye tendentsii v razvitii rossiiskoi nauki. Moscow: Logos; 2005. p. 122-38. Russian.
13. Rozin VM. Nauka: proiskhozhdenie, razvitie, tipologiya, novaya kontseptualizatsiya. Moscow: Moscow psychologic-social institute, Voronezh: MODEK; 2008. 600 p. Russian.
14. Kurakova NG, Kolin SK, Oliver EA. [Independent validation system of scientific medical organizations in the paradigm of twenty years remoteness]. Menedzher zdravookhraneniya. 2014; (8): 29-36. Russian.
15. Larin SN, Lazareva LYu, Ostrovsky SM. [Formalized model and procedure of multicriterion evaluation of efficiency of scientific organizations' innovation activity]. Innovatsii v nauke. 2013; (181): 6-19. Russian.
Литература
1. Постановление Правительства РФ от 08.04.2009 № 312 (ред. от 29.12.2016) «Об оценке и о мониторинге результативности деятельности научных организаций, выполняющих научно-исследовательские, опытно-конструкторские и технологические работы гражданского назначения».
2. Приказ Минобрнауки России от 05.03.2014 № 161 (ред. от 17.07.2015) «Об утверждении типового положения о комиссии по оценке результативности деятельности научных организаций, выполняющих научно-исследовательские, опытно-конструкторские и технологические работы гражданского назначения, и типовой методики оценки результативности деятельности научных организаций, выполняющих научно-исследовательские, опытно-конструкторские и технологические работы гражданского назначения».
3. Проничкин С. В., Тихонов И. П. Разработка системы критериев и методических подходов к экспертной оценке эффективности деятельности научных организаций. Нац. интересы: приоритеты и безопасность. 2013; (37): 13-8.
4. Филонюк В. А., Сачек М. М., Малахова И. В., Дудина Т. В., Ёлкина А. И. Комплексная оценка результативности медицинской науки и определение рейтинга научных организаций за рубежом и в Беларуси. Вопр. орг. и информ. здравоохр. 2013; (2): 20-30.
5. Вялков А. И., Глухова Е. А., Мартынчик С. А. Модель мониторинга формирования инновационных компетенций кадрового состава научной медицинской организации, ориентированного на создание критически важных технологий. Соц. аспекты здоровья насел. 2015; 42 (2): 17.
6. Мартынчик С. А., Глухова Е. А., Галустова Л. Р. Требования к построению системы для оценки результативности и потенциала научной деятельности на уровне организации. Соц. аспекты здоровья насел. 2013; 32 (4): 10.
7. Оганов Р. Г., Трущелев С. А. Наукометрические подходы к анализу результатов научно-исследовательской деятельно-
сти. Кардиоваскул. тер. и профил. 2012; 11 (2): 90-5.
8. Стародубов В. И., Кузнецов С. Л., Куракова Н. Г., Цветкова Л. А., Арефьев П. Г., Иванов А. В. и др. Публикационная активность российской медицинской науки в фокусе актуальной научной политики: оценка достижимости целевых показателей. Вестн. РАМН. 2013; (3): 8-14.
9. Шарабчиев Ю. Т. Продуктивность ученых: инструменты оценки. Наука и инновации. 2013; 119 (1): 4-8.
10. Зубова Л. Г., Андреева О. Н., Антропова О. А. К вопросу о результативности деятельности государственных научных организаций (по оценкам социологического мониторинга 2005-2011 гг.). Инновации. 2012; 170 (12): 51-60.
11. Кучеренко В. З., Мартынчик С. А., Глухова Е. А. Экспертное оценивание и индикаторы качества деятельности специалистов научной (медицинской) организации по профилю общественное здоровье и управление здравоохранением. Пробл. управ. здравоохр. 2010; (5): 24-9.
12. Кузнецова Т. Е. Правовой фактор научно-технического развития России: область неопределенности. В кн.: Алахвердян А. Г., Семенова Н. Н., Юревич А. В., редакторы. Науковедение и новые тенденции в развитии российской науки. М.: Логос; 2005. с. 122-38.
13. Розин В. М. Наука: происхождение, развитие, типология, новая концептуализация. М.: МПСИ, Воронеж: МОДЭК; 2008. 600 с.
14. Куракова Н. Г., Колин С. К., Оливер Е. А. Вневедомственная система оценки научных медицинских организаций в парадигме двадцатилетней давности. Менеджер здравоохр. 2014; (8): 29-36.
15. Ларин С. Н., Лазарева Л. Ю., Островский С. М. Формализованная модель и методика многокритериальной оценки эффективности инновационной деятельности научных организаций. Инновации в науке. 2013; (18-1): 6-19.
