CC BY
30ОБРАЗОВАТЕЛЬНАЯ ПОЛИТИКА
DOI 10.15826/umpa.2019.03.015
THE MONITORING OF MONITORING: WHAT'S WRONG WITH THE MINISTRY'S NEW APPROACH TO SUPERVISION OF EFFECTIVENESS OF HIGHER EDUCATION INSTITUTIONS' PERFORMANCE?
V. Bolotova, G. Motovab, V. NavodnoV
aNational Research University Higher School of Economics 20 Myasnitskaya str., Moscow, 101000, Russian Federation bNational Centre for Public Accreditation 206a Volkov str., Yoshkar-Ola, Mari El, 424000, Russian Federation; g.motova@ncpa.ru
Abstract. The article discusses the Monitoring of higher education institutions' (HEIs') effectiveness, which has been conducted annually by the Ministry of Education and Science in the last five years. The method of the Monitoring is based on the technology of collecting and systematizing information used by the state accreditation in 1997-2010. The reasons for the introduction of the Monitoring by the Ministry are described; the features of the performance indicators are justified by the state educational policy. A model of presentation of the Monitoring results in the league table format is suggested, which makes it possible to use its results more flexibly for the management of the higher education system. It also allows higher education institutions to analyze their positions and to form their own ways of the development. Keywords: Monitoring of effectiveness, performance indicators of higher education institutions, criteria, ratings, multidimensional ranking, league tables
For citation: Bolotov V., Motova G., Vladimir Navodnov V. The Monitoring of Monitoring:
What's Wrong with the Ministry's New Approach to Supervision of Effectiveness of Higher Education Institutions' Performance? University Management: Practice and Analysis. 2019; 23(3): 5-13. (In Russ.). DOI: 10.15826/umpa.2019.03.015
Introduction
The year of 2017 can be viewed as a milestone for the system of education in the modern history of Russia. In 1992 the first law «On Education» was adopted. It radically changed the concept, content and structure of education, and launched the process of its transformation (Gounko & Smale, 2007). In the first 20 years, the national higher education system experienced an educational boom in the new social and economic conditions. Higher education became more accessible and the demand and supply of educational services increased dramatically (Heyneman, 2010). The last five years were marked by a demographic dive, which is a consequence of the falling birth rates in the 1990s.
The year of 2012 saw a new version of the law «On Education in the Russian Federation», which was groundbreaking for the system of education. Never before did the Russian education system go through such drastic changes due to social and economic and
political reforms. A five year period is too short a term to make far reaching conclusions, yet it is long
enough to identify trends in the development of education and see the first impact of the new law.
The most obvious and controversial feature of the law was strengthening control and oversight procedures: licensing and licensing control (scheduled and random), state accreditation, state control of education quality (scheduled and random), oversight of law implementation (scheduled and random). The Monitoring of HEIs' effectiveness has become yet another supervision procedure whose main objective is declared to be education quality enhancement.
However, new approaches to overseeing education were developed without public participation and open discussions, in fact, behind the scenes (Pukharenko, Norina & Norin, 2017). No wonder that for treating the ailing system of education radical surgery was chosen as the most effective method of achieving quality enhancement. As a result,
a number of HEIs and their branches were cut down significantly.
The purpose of the article is to suggest a new method of presenting the results of the Monitoring of HEIs based on the analysis of the current practices. The study involves the analysis of social, economic, demographic and political prerequisites for implementing the procedure of the Monitoring. The changing social, economic and demographic conditions in the country impacted the scope of higher education, whereas the political developments brought about changes in the educational policy of the Ministry.
The paper also looks into the analysis of the Monitoring indicators and results of the last
5 years. The state educational policy determines the Monitoring indicators, whereas the results of the Monitoring serve as a mechanism for governing the system of higher education
On the basis of the analysis, the study will suggest a new method for presenting the Monitoring results in the league table format. It is argued that this method will work towards the development of higher education instead of its downsizing.
While preserving the current indicators as vectors of educational policy, more flexible methods for calculating the Monitoring indicators and presenting results are suggested. This method of evaluating HEIs' performance may be of interest to administrative bodies and HEIs not only in Russia, but also in other countries which use monitoring for controlling the quality of education.
Methodology
The analysis of the Monitoring of HEIs' effectiveness was based on the study of legislative and regulatory documents of the Ministry; the Monitoring indicators and methodology for their calculation; statistical data and decisions made on the results, and public opinion about the effectiveness and appropriateness of the procedure. A more flexible method for presenting the Monitoring results in the league table format is based on McKinsey-Abel's method and can be used as one of possible methods to enhance the performance of HEIs. The article presents the authors' expert opinion on the subject based on the personal experience in the sphere of quality assurance in education.
Monitoring as a tool of the state educational policy
In the Federal Law «On Education in the Russian Federation», the monitoring is defined as «systematic standardized supervision over the conditions of educa-
tion and the dynamics of its outcomes, the conditions of implementing educational activities, student population, academic and extracurricular achievements of students and career development of graduates...». «The order., and the mandatory information to be presented is established by the Government of the Russian Federation» (RF Federal Law, 2012).
Based on the law monitoring can be defined as a specially organized procedure for information collecting and analysis with the aim of continuous standardized oversight of a process, its diagnostics and predicting its development. However, since 2012 monitoring has become something different from a tool for supervision, diagnostics and predicting. It has become an instrument for decision-making on ineffective higher educational institutions and their branches, which are to be reorganized.
Enhancing the quality of higher education provision, which in many cases does not meet the required standards, remains an urgent issue. However, tackling it by reorganisation of ineffective educational institutions is not necessarily the most appropriate way. Firstly, the concept of an effective educational institution can imply different features, and therefore needs in-depth discussion. Secondly, the focus should be on the quality of programs, but not the effectiveness of a HEI. In one and the same HEI there may be both popular up-to-date and adequately equipped programs, as well as programs which fail to meet the required standards.
Obviously, the introduction of the Law and outlined procedures was a response to the President's Decree of May 2012 and his direct order to carry out «the monitoring of higher education institutions with the purpose of assessment of their effectiveness and reorganization of ineffective state HEIs» until the end of December 2012 (Decree, 2012). Following this order, in the autumn of 2012 the Ministry of Education conducted the first «effectiveness» monitoring of the state HEIs, which caused controversy in the academic community. There were several reasons for such controversy.
First of all, the outcomes of the State Monitoring. There's nothing unusual in the monitoring procedure in itself. The governing body, in the case of the Russian Federation it's the Ministry of Education,, should have at its disposal full and credible information on the activity of subordinate organizations (Brennan, 1997), especially if there was «an order from above». The Ministry certainly collected various data before, however, in light of the new Law the monitoring procedures became highly burdensome for HEIs and, more over, turned into controlling and restructuring the HEI network.
The second reason is the shift in the direction of the state educational policy: from granting autonomy
and rights to expand the higher education sector to tightening the state regulation and decreasing education expenditures (Forrat, 2012). Alongside with the Federal Law, a roadmap for education development has been worked out. According to this document, one of the ways for effectiveness enhancement of higher education is reduction of the number of HEIs: in the period from 2012 to 2017 twenty-eight HEIs have to be reduced annually (the total of 168 HEIs) (Order of the Gov., 2012).
Despite criticism these measures could be justified by the state of higher education sector by 2010. From the beginning of 1990-s until 2009, the
higher education system greatly increased in scope: the number of students tripled - from 2.5 million to 7.5 million (Fig. 1); there was an increase in the number of educational programs (from 10.2 thousand to 32.5 thousand) and of the number of HEIs (without branches) - from 528 to 1,134 (Fig. 2). But already in 2010 there was a prominent tendency towards a sharp decrease in the number of students due to the demographic dive caused by the falling birth rates (10 % annually) during the perestroika period and dramatic deterioration of social economic conditions in the country (Fig. 3).
c-j со к. _
№ Ol Jl Ol Ol Ol Ol ffi № O) Í1 ® ® ®
OJ Ol CM CM
счсчсчсчсчсчсчсчсчсчсчсчсчгчсчсчсчсчсчсч
-Number of students
"Number of students enrolled in state HEIs ■Number of students enrolled in private HEIs
Fig. 1. Dynamics оf the number of students enrolled in state and private HEIs (in thousands) (Isras.org, 2018)
Fig. 2. Dynamics of the number of HEIs (excluding branches) (Gks.ru, 2018)
Fig. 3. Dynamics of the number of births in Russia against the data of 1975,% (Gks.ru, 2018)
Moreover, the education boom at the turn of the century had a negative impact on the quality of graduates in HEIs: drastic increase in the number of students could hardly be supported by adequate material, human and financial resources. The Ministry set the task of enhancing education quality, first and foremost through the reduction of ineffective HEIs.
The concept of effectiveness and the criteria for its assessment had not been defined by any regulatory document. However, the monitoring procedure was launched in the autumn of 2012 (502 state educational institutions and 930 branches); and since 2013 it has been conducted annually as on April 1. All state HEIs (since 2013 all private HEIs as well) have to fill in specially developed forms with information on their activity.
The obtained information is systematized and compared against the threshold values. The results are reviewed by a special committee comprised of members from ministries and bodies representing higher education. The committee makes recommendations for the founders of HEIs and their branches which are found to be ineffective. The founder makes a decision either to close the HEI or the brunch (with the students being transferred to other HEIs) or to merge it with another HEI which is found effective.
Thus, the goal of enhancing the quality of education is considered to be achieved. However, until now there have not been any verified data that the administrative measures reducing the number of institutions have been really effective in achieving this goal.
The Monitoring indicators as a vector of the state educational policy
There are 154 criteria for collecting information, which are divided into seven groups (Indicators.
miccedu.ru, 2018). The monitoring indicators, in fact, characterize the direction of the state educational policy and determine the developmental trends of Russia's higher school.
These indicators include the following:
Educational performance: the quality of admitted applicants. This indicator is calculated by the grade point average of the Unified State Exam, which is a high school final and university entrance exam taken upon completion of the 11th grade of the secondary comprehensive school. Only full-time students enrolled on the basis of general competition are taken into account. This aspect is indicative of the HEI prestige, the demand for the programs it delivers. But these are not the direct results of the institution's performance. Obviously, better prepared applicants make a prerequisite for a higher level of graduates; however, the HEI does not contribute to the training of applicants.
Research activity: the estimated indicator for allocating funds for research from all possible sources per one HEI teacher involved in teaching and research. This indicator can be regarded as a general performance indicator, but not as an indicator of educational activity as such. It is supposed to characterize the level of teachers, as a modern teacher should carry out research. However, it is impossible to estimate how much this or that researcher is involved in teaching.
International cooperation of a HEI is characterized by the number of admitted international students. It is calculated against the total number of students enrolled in a HEI or a branch. It is not clear why the share of international students is for the most part (99 %) comprised of compatriots from the former Soviet Union and developing countries from Asia and Africa. And these countries admittedly have a lower
level of education in comparison with Russia. How can this indicator possibly determine the effectiveness or performance of a HEI activity? This might be a contribution to the solution of certain geopolitical issues, but they are beyond the scope of the statutory activities of HEIs. This indicator could be interpreted in different ways, and there is no reason to believe that the quality of education is directly related to the number of international students enrolled in a HEI.
Financial and economic activity is an indicator of the income received by a HEI from all possible sources: educational activity (fee-paying students), research, grants, sponsorship, publication activity etc. This indicator characterizes the ability of a HEI to draw budget and out of budget means. It can characterize the effectiveness of the institution, but not its educational activity. This indicator is more indicative of entrepreneurship activity.
Infrastructure of a HEI: the estimated floor space of academic buildings per one student. This indicator can be interpreted in different ways: on the one hand, the less the better, as the floor space should be used effectively. But how it is going to impact the educational and research activity is hard to tell. On the other hand, adequate laboratory facilities testify to a HEI's ability to offer quality educational services. But the issue whether there is a direct correlation between the size of laboratories and the quality of education needs to be further explored. Besides, many HEIs borrow material resources for their educational process. These are, first of all, medical and arts and culture HEIs. Since 2015, this indicator has been excluded from the monitoring list.
The average pay of the teaching staff as related to the average pay in the real economy of the region, where a HEI is located (this indicator was introduced in 2015). The average pay in Russia's regions is calculated by the Federal State Statistics Service. The correlation of the teachers' salary and the average pay in the region does not have any relevance either to educational or research activity. This is a vector of the state social policy, but unfortunately, it is not underscored by the budgetary funding. The indicator is determined by the administration of a HEI.
Employability of graduates: this indicator is calculated on the basis of the tax received by the Pension Fund of the Russian Federation from the graduates employed in the previous year. The information is provided by the Pension Fund of Russia. The employabil-ity of graduates should be one of the most important. But in fact, it is not because besides representing the average percentage of graduates' employability in a given HEI, it does not draw a complete and clear picture of the quality of graduates and their achievements.
Additional indicatorsis applicable to specific HEIs and their branches offering programs in arts, sports, military training, transportation, medicine, agriculture. For every group of HEIs, there are used specific indicators, which, as a rule, characterize the quality of the teaching staff.
Practically each of the indicators can be subjected to criticism by HEIs, and for a good reason. The competition based on the results of the Unified State Exams characterizes the quality of admission rather than a HEI's performance. It is also influenced by the location of a HEI and the labor market situation. Neither is it correct to evaluate research based on the allocated funds because technology and sciences research is much more substantially funded than research into humanities or economics. Especially problematic is this indicator for teacher training and culture and arts institutions.
The requirement to increase the admission of foreign students has drawn the strongest criticism: for regional HEIs and for the HEIs training specialists for the local regional labor market this requirement is perplexing, to say the least. Besides, the solution of this task makes HEIs minimize the admission requirements for foreign applicants; which means trying to meet the indicator regardless of the level of applicants and their readiness for education and training. This will decrease the quality of higher education still further.
The indicators of economic activity and teachers' pay rate reveals obvious imbalance when calculating the funding of HEIs, and the calculation of tax revenue characterizing employability of graduates iden-tifieds gaps in the calculation of tax returns.
Thus, the values of the majority of indicators do not depend on a HEI's performance, but are contingent on external factors; whereby these external factors operate in the conditions of different HEIs in different ways, which, under the current methodology, puts HEIs in unequal conditions. The selection of data should not be based on their accessibility, but on the indicators that reflect the quality and strong points of an institution. It should be made clear why this or that indicator is selected and what it shows. The suggested indicators characterize, in the first place, a HEI's potential, but not quality, because it is much more difficult to assess quality than resources (Marshall, 2016). This is also true regarding the accessibility of data.
To sum up, the weaknesses of the present Monitoring are: firstly, the indicators do not directly refer to the quality of education provision or the effectiveness of educational activities (only indirectly). Secondly, statistical information submitted by HEIs is not verified as is done during the accreditation procedure where there is always a site visit by experts.
There have been instances of mistakes and misrepresentation of information when filling in statistical report forms. And finally, practically every year the «rules of the game» change: criteria, indicators and calculation methodology are reviewed and altered. These rules are not supported by any regulatory documents and are not communicated to HEIs, or are communicated very late (less than a month before the deadline of submitting information).
The dynamics of the Monitoring outcomes
It should be noted that the methodology of indicator calculation was based on the model of state accreditation, which was effectively used in 1999-2009 when determining the status of a HEI (Motova & Pykko, 2012). The criteria of state accreditation were determined by the value of the lower quartile in the sampling of HEI of one kind (institute, academy, university) and, as a rule, were set at a threshold value for the period of 5 years.
As previously mentioned, the decision was made on the basis of all indicators taken together with the consideration of the compensation mechanism.
But unlike the state accreditation, in the methodology of calculation of the Monitoring the threshold value is set as the median value of the previous year. The decision is made based on the threshold value, which divides all HEIs into effective and ineffective regardless of their kind (university, academy, institute). The profile of each HEI is taken into consideration (arts and culture, sports, transportation, medicine, agriculture etc.).
The threshold value is calculated for each indicator separately. A HEI's effectiveness is determined by the number of its indicators which are equal to the threshold values or are above them. A HEI is considered effective if out of seven indicators four or more are above the threshold values.
Simple calculation shows that if we apply this model, the probability of a HEI's being considered effective is 0.5.
It means that every other HEI will have values below threshold in every indicator. Besides, the thresh-
old values will be different every year, as the indicator values of the institutions do not remain the same.
The statistics of HEIs' participation in the Monitoring procedures of 2012-2018 (Table 1) shows that since 2014 there has been a decrease in the number of HEIs submitting statistical information.
It is a direct consequence of introducing the procedure of monitoring, as based on its results the Ministry initiated the procedures of optimizing the higher school system. On the basis of the obtained data and the calculation of effectiveness, the founder is recommended that a decision concerning the future of its HEI or its branch should be made: either to close or reorganize it, i. e., in fact, to merge it with a more successful HEI.
The required number of indicators is distributed unevenly by year. 2014 was the year when the largest number of HEIs was recognized ineffective: 1,010 (43.46 %), including 238 HEIs (79 state and 159 private) and 772 branches (489 state and 283 private).
1,289 institutions of higher education (712 HEIs and 577 branches) took part in the monitoring procedures of 2017 (Table 2). 121 institutions (9.4 %) did not meet the necessary requirements.
In the last five years, as a result of the demographic crisis and the state educational policy, the national higher education experienced a dramatic decrease in the scope, and, as predicted in the officially published sources, this tendency is likely to persist. The number of students has decreased from 7.5 million to 5 million. The number of HEIs and their branches has decreased by 1,000.
This hard-hitting system of decision-making will undoubtedly turn the Monitoring procedure into a «high-stakes game», which is bound to raise a wave of indignation and criticism. On the other hand, these methods of education quality enhancement do not significantly reflect on the quality.
The reorganisation of HEIs by merger adds to the numerous problems connected with the redistribution of governance functions, merger of departments and faculties, reduction of the number of the teaching staff and increase of their workload. Besides, there is no guarantee that after the acquisition of a weaker institution the stronger HEI will retain its leading posi-
Table l
The number of HEIs submitting statistical information for the Monitoring (2012-2018)
2012 2013 2014 2015 2016 2017 2018
HEIs 502 934 968 887 830 769 731
Branches 930 1478 1356 1229 932 692 583
Total 1432 2412 2324 2116 1762 1461 1314
Table 2
The monitoring results of HEIs effectiveness: the data of the Ministry of Education and Science of the Russian Federation for 2017
Indicators fulfilled HEIs Branches Total
0 0 5 5
1 2 1б 18
2 9 25 34
3 21 43 б4
4 109 1б0 2б9
5 221 157 378
б 228 125 353
7 122 4б 1б8
Total 712 577 1,289
tion. There are also doubts whether in this situation research schools will be preserved. Among HEIs, which lost their autonomy there are HEIs with long-standing traditions and achievements in research and education of highly-qualified graduates. The disappearance of these HEIs from the map of national higher education will cause confusion not only in Russian, but also in the international community.
League tables of Russian HEIs based on the monitoring results in 2017
The main objective of the Monitoring procedure by the educational authority, is resource optimisation. However, in the nearest future this process has to be changed - the activities should be shifted from reducing the number of HEIs and their branches to increasing the effectiveness of the remaining ones. The method of dividing HEIs into two groups is a blunt instrument, because among successful HEIs there are excellent institutions as well as those, which may lose their prominent positions, if neglected. Moreover even 'ineffective' HEIs are not really «hopeless». Most of them need help in identifying and rectifying problems, and guiding further development. . The above mentioned indicators could be used for this purpose, provided the decision making-policy is changed and a more flexible approach is introduced. For example, instead of dividing the HEIs into two groups - above and below the threshold value, it stands to reason to divide the sampling into several groups. Thus, in each of the seven indicators, a HEI may occupy different positions in relation to the threshold value, and can be represented in a multidimensional format and be ranked in the league table (van Vught &Westerheijden, 2010).
The application of the league methodology, on the one hand, makes it possible to give an overall assessment of a HEI's performance with the consideration of various needs of different categories of users. On the other hand, it will not combine the assessment of different areas of activity (i. e. educational, research, international) in one aggregated indicator. Several ranking indicators are integrated into one. Based on this indicator, HEIs are distributed on a single scale (Bergseth, Petocz & Abrandt Dahlgren, 2014). Compared to this, the league table represents a more flexible governing instrument for the support of HEI development.
The McKinsey 7S model in the league table format (as one of possible models) can serve as a basis for the ranking methodology.
In 1970s, the McKinsey group and General Electric developed an analytical model for the assessment of strategic positions of businesses, based on integrated indicators and assigning a place in the league table.
The sampling of educational institutions by an individual indicator can be divided into five areas and marked by letters. Area A includes the values of indicators comprising the first quartile of the sampling; area B - the second quartile; area C - the values between the median and the threshold criterion; area D - the values between the threshold and the third quartile; area E - the values of the fourth quartile.
Applying the McKinsey method to the monitoring results it is possible to distinguish a certain number of leagues, for example, 10, where the first league will be represented by HEIs whose indicators belong to the upper quartile of the sampling and represent an AAAAAAA model. The HEIs with D-s and E-s in the majority of indicators will fall into league 10. Different letter combinations will show the distribution of HEIs in the leagues. The first seven leagues will be represented by effective HEIs, while HEIs
Area A Area B Area C Area D Area E
i I
А
Xl/4
В
M
п С
D
Хзч
Е
25% 50% 75% 100%
Educational institutions
Fig. 4. Distribution of monitoring results in the league table
with over four D-s or E-s will fall into the remaining three leagues.
This system of grouping reveals that out of 121 ineffective HEIs (the Monitoring results of 2017, 8-10 leagues) only 27 HEIs are truly ineffective, while the other 94 HEIs can be improved through an adequate correction and development program (Table 3). Unfortunately, in the last five years hardly more than three HEIs entered the first league. And in 2017 there was only one. This was not one of the flagships of the Russian HE system (Lomonosov State University or SPbSU), but a national research university with a long history - Gubkin Russian State University of Oil and Gas.
In general, the league distribution which is close to the norm will show that most HEIs are placed in the fourth, fifth and sixth leagues. The first three leagues will comprise the leading national HEIs. However, the ranking order of HEIs based on the monitoring results differs significantly from their places in the world university rankings (QS, ARWU, THE).
Table 3
League table based on the Monitoring results of the Ministry of Education and Science of the Russian Federation in 2013-2017
League 2013 2014 2015 2016 2017
1 1 1 3 3 1
2 7 18 27 35 26
3 91 145 180 208 160
4 239 143 244 290 222
5 509 310 348 406 417
6 444 176 267 306 234
7 320 23 77 131 108
8 84 609 180 155 63
9 100 139 71 86 31
10 79 150 72 126 27
Total: 1,874 1,714 1,469 1,746 1,289
Undoubtedly, the indicators of the world rankings differ significantly from the indicators of the national Monitoring, mainly because they take into consideration stakeholders' opinion (up to 40 %); availability of outstanding academics among teachers and graduates; international recognition of research achievements etc. The Monitoring indicators approved by the state do not take into consideration such achievements. The state initiates and supports both the Monitoring and participation in the world rankings. The former is a result of the responsible state policy, the latter is
aimed at education quality enhancement on the national and international levels. But the methods used to achieve this goal are fundamentally different.
Conclusion
The Monitoring of HEIs' effectiveness in Russia has been carried out in the last seven years. Though it is viewed as a necessary procedure, it has a number of significant drawbacks, among which the hard-hitting decision making and follow-up measures on reduction and merger of HEIs. This «high-stakes game» generates much controversy on all levels of legislative and executive power and the academic community.
Based on the indicators imposed by the Ministry, it becomes obvious that the effectiveness of HEIs' performance relies on their potential and resourcefulness. The tools and calculation methodology of the Monitoring of HEIs' effectiveness does not meet the set goals and objectives, because the indicators do not characterize HEIs' efficiency and effectiveness. The assessment method, which uses neither grouping nor correction coefficients, a priori puts HEIs in unequal conditions.
At the same time, it is noticeable that the list of the Monitoring indicators reflects the key directions of the state educational, social and international policy: education quality enhancement achieved through the increased requirements to admission; extension of the export of Russian education by means of increased foreign students admission; strengthening the teachers' motivation by raising their salary; employability of graduates, etc.
In spite of the fact that the indicator calculation is controversial, the Monitoring can really reflect the condition of the education system and the dynamics of its results.
Recognizing the importance of challenges faced by the national system of education, it is necessary at the same time to provide HEIs with a strategic vision of their further development (Billing, 2004). For this purpose, there should be a more subtle, clear and facilitating governing instrument, unlike the inflexible binary scale of decision-making (effective/non-effective). The suggested methodology of presenting the Monitoring results of higher education institutions in the league table format may help them work out the strategy of development and performance enhancement.
References
1. Bergseth B., Petocz, P. and Abrandt Dahlgren, M. (2014). Ranking quality in higher education: guiding or misleading? Quality in Higher Education, 20(3), pp. 330-347.
2. Billing D. (2004). International comparisons and trends in external quality assurance of higher education: Commonality or diversity? Higher Education, 47(1), pp. 113-137.
3. Brennan, J. (1997). Authority, legitimacy and change: The rise of quality assessment in higher education. Higher Education Management, 9 (1), pp. 7-29.
4. Decree of The President Of The Russian Federation of May 7, 2012 No 599 «About the measures of implementation of the state policy in the sphere of education and science» (2012)
5. Forrat N. (2012). Comparative-Historical Social Science (CHSS) Working Paper Series. Global Trends or Regime Survival: the Reforms in Russian Higher Education. Evanston, IL: The Buffett Institute.
6. Gks.ru. (2018). FSSS. Main. Retrieved from: http:// www.gks.ru/wps/wcm/connect/rosstat_main/rosstat/en/ main/ [Accessed 7 Mar. 2018].
7. Gounko T., & Smale W. (2007). Modernization of Russian higher education: exploring paths of influence. Compare: A Journal of Comparative and International Education, 37(4), pp. 533-548. http://dx.doi. org/10.1080/03057920701366358
8. Heyneman S. (2010). A Comment on the Changes in Higher Education in the Former Soviet Union. European Education, 42(1), pp. 76-87. Retrieved from: http://dx.doi. org/10.2753/eue1056-4934420104
9. Indicators.miccedu.ru. (2018). Main information and computation center. Retrieved from: http://indicators.miccedu. ru/monitoring/[Accessed 7 Mar. 2018].
10. Kartavuzov.ru. (2018). Monitoring of the system of education. Retrieved from: http://kartavuzov.ru [accessed 7 Mar. 2018].
11. Marshall S. (2016). Quality as sense-making. Quality in Higher Education, 22(3), pp. 213-227.
12. Motova G. and Pykko R. (2012). Russian Higher Education and European Standards of Quality Assurance. European Journal of Education, 47(1), pp. 25-36.
13. Order of the Government of the Russian Federation from 30.12.2012 No 2620-p and the plan of activities («Roadmap») of the changes in social sphere aimed at the enhancement of education and science effectiveness (2012).
Submitted on 17.04.2019
14. Pukharenko Y., Norina N., & Norin V. (2017). Russian engineering education in the era of change. European Journal of Engineering Education, 42(2), pp. 171-187. Retrieved from: http://dx.doi.org/10.1080/03043797.2017.1311307
15. RF Federal Law «On Education in Russian Federation» (2012), No 273-FZ.
16. Rossiyskaya Gazeta (2012). Federalnyy zakon ot 29 dekabrya 2012 g. No 273-FZ «Ob obrazovanii v Rossiyskoy Federatsii» [Federal law from 29 December 2012, No 273-FZ «On Education in the Russian Federation]. [online] Retrieved from: https://rg.ru/2012/12/30/obrazovanie-dok.html [accessed 7 Mar. 2018]. (In Russ.).
17. Rossiyskaya Gazeta (2012). Ukaz Prezidenta Rossiyskoy Federatsii ot 7 maya 2012 g. N 599 «O merakh po realizatsii gosudarstvennoy politiki v oblasti obrazovani-ya i nauki» [The Decree of the President of the Russian Federation of May 7, 2012 № 599 «About measures to implement the state policy in the sphere of education and science»]. Retrieved from: https://rg.ru/2012/05/09/nauka-dok.html [accessed 7 Mar. 2018]. (In Russ.).
18. Rossiyskaya Gazeta (2013). Rasporyazhenie Pravitel'stva Rossiyskoy Federatsii ot 30 dekabrya 2012 g. N 2620-r g. Moskva.[Order of the Government of the Russian Federation from 30.12.2012 N 2620-p and the plan of activities ('Roadmap») of the changes in social sphere aimed at enhancement of education and science effectiveness.]. Retrieved from: https://rg.ru/2013/01/14/obraz-eff-site-dok.html [accessed 7 Mar. 2018]. (In Russ.).
19. Sheregi F., Arefev A., Kliucharev G., & Tiurina I. (2015). Chislennost' obuchayushchikhsya, peda-gogicheskogo i professorsko-prepodavatel'skogo per-sonala, chislo obrazovatel'nykh organizatsiy Rossiyskoy Federatsii. (Prognoz do 2020 goda i otsenka tendentsiy do 2030 goda). [Number of students, number of the teaching staff, and number of educational institutions in the Russian Federation]. Moscow: Institute of Sociology of the Russian Academy of Sciences. 270 p. (In Russ.).
20. Van Vught F. and Westerheijden D. (2010). Multidimensional ranking. Higher Education Management and Policy, 22(3), pp. 1-26.
Information about the authors:
Victor Bolotov - Dr. hab. (Pedagogy), Research Advisor of the Centre for Education Quality Monitoring, Higher School of Economics; vbolotov@hse.ru.
Galina Motova - Dr. hab. (Pedagogy), Deputy Director of the National Centre for Public Accreditation; g.motova@ncpa.ru. Vladimir Navodnov - Dr. hab. (Engineering), Director of the National Centre for Public Accreditation; director@ncpa.ru.
