CC BY-NC-ND
60
Evaluation of Universities in Western Europe: From Quality Assessment to Accreditation
Erik De Corte
Erik de Corte
Professor Emeritus, Center for Instructional Psychology and Technology (CIP&T), University of Leuven, Belgium. Address: Katholieke Universiteit Leuven, Oude Markt, 13, Bus 5005 3000 Leuven, Belgium. E-mail: [email protected]
Abstract. The emergence of national evaluation schemes in higher education in Western Europe occurred in the mid-1980s and originated in the Netherlands, Flanders, France, and the United Kingdom. In subsequent years, many other Western European countries established similar evaluation systems. Over the past thirty years, these systems have undergone important changes. This article discusses the following topics relating to evaluation in Western European higher education: What brought about the establishment of schemes for the evaluation of higher education institutions in Western Europe in the mid-1980s? The main concepts relating to the evaluation of higher education programs—quality, quality assurance, quality control, quality management, quality audit, quality assessment, evaluation, and accreditation—will be defined. Quality assessment introduced in the mid-1980s lasted until the beginning of the 21st century and focused on contributing to the
improvement of higher education. The Dutch quality assessment system, which was representative for Western Europe, will be presented. Notwithstanding the largely positive influence of the quality assessment system, a shift occurred in Western Europe during the early years of the 21st century from quality assessment to accreditation (a formal judgment that the quality of a degree course or an institution meets certain standards). Although based on quality assessment, this approach shifted the focus from improvement to accountability. The Dutch accreditation scheme will be discussed as an example of this shift. Because of the shift from improvement to accountability, but also because of the bureaucratic burden and the high costs, the accreditation scheme began to be criticized more and more heavily. This led to very intensive discussions and consultations, which resulted in a revised accreditation system that has been operational in the Netherlands since 2012. The major changes in the accreditation approach will be reviewed. The article concludes with some final comments and a future perspective.
Keywords: national evaluation schemes, higher education, quality assessment, quality assurance, accreditation, improvement of higher education.
At the start of my career in the early 1970s as an associate professor in the Faculty of Psychology and Educational Sciences at the University of Leuven, assessment of teaching was more or less nonexistent. In those days, assessment and evaluation were well-established
Received in April 2014
1. Introduction
http://vo.hse.ru
1
THEORETICAL AND APPLIED RESEARCH
practices with respect to students; their achievement was and still is regularly evaluated as a basis for making decisions about their academic status and the awarding of a degree at the end of their study. Assessment of academics was focused on decisions about appointment and promotion and about the allocation of grants; such assessment addressed mainly the research activities of individual scholars. My university’s Department of Education was the first to take an initiative toward the evaluation of teaching behavior by the development in the late 1970s of the EVADOC Questionnaire, a tool for assessing individual professors’ lecturing quality by means of student evaluation. Administration of the questionnaire was on a voluntary basis and enabled professors to get insight in strengths and weaknesses relating to seven observable aspects of their teaching behavior [De Neve, Janssen, 1984].
The emergence of national evaluation schemes in higher education in Western Europe occurred in the mid-1980s and originated in the Netherlands, Flanders, France, and the United Kingdom. In subsequent years, many other Western European countries established similar evaluation systems. In Central and Eastern Europe, such schemes were introduced during the 1990s and onwards. Over the past thirty years, these systems have undergone important changes. In this article, I will present and discuss the following topics relating to evaluation in higher education:
• Causes for the establishment of evaluation schemes in Europe.
• Concepts and definitions relating to evaluation in higher education.
• Quality assessment focused on the improvement of higher education.
• Accreditation focused on accountability of higher education.
• Revising the accreditation approach to meet criticisms.
• Final comments and a future perspective.
For an extensive overview of how quality assessment schemes and accreditation have spread across Europe during the past decades, I refer to the comparative study of 20 countries published in the volume Accreditation and Evaluation in the European Higher Education Area [Schwarz, Westerheijden, 2004a] .
2. Causes of the emergence of evaluation schemes in the 1980s and initial developments
Pursuing quality in order to achieve excellence has always been a major goal in higher education. Until the 1970s, quality in higher education was controlled by bureaucratic measures such as legal conditions for starting institutions, faculties and study programs, formal rules for the appointment of academic personnel, etc. In addition, there was a high degree of trust in society that universities themselves could guarantee quality. However, in the late 1970s, there was
2
Educational Studies. 2014. No 4
Erik De Corte Г ■
Evaluation of Universities in Western Europe: From Quality Assessment to Accreditation
a loss of confidence in the role of academics as guardians of quality and efficiency. Moreover, quality assurance used as a management tool in the industrial sector was seen more and more as an instrument that could also be appropriately applied to the management of higher education. The application of this tool was greatly stimulated by several other developments as well [Schwarz, Westerheiden, 2004b; Teichler, 2007] .
First of all, the enormous expansion and “massification” of higher education in the 1970s called for new forms of management. This necessity was strengthened by the growing concern that this “massification” could slacken the quality and performance of higher education institutions.
Second, as another consequence of the expansion of higher education and its increasing complexity and differentiation, central control reached its limits. Whereas in the previous era governments were heavily involved in the planning and organization of higher education, a loss of trust in the capacity of governments to guarantee and enhance its quality, relevance, and efficiency occurred in the 1980s. Nevertheless, governments, being the major financial source of universities, were eager to maintain and even improve the level of quality and performance of higher education through the application of external tools of control.
Third, in the 1980s and 1990s hopes grew that the use of quality assurance as a management tool would stimulate institutionalized reflection on the activities and outcomes of universities and that growing managerial capacity in higher education institutions would foster improved levels of quality and performance.
It must be noted that all of these reasons did not play an equally important role across different countries. For instance, as argued by Schwarz and Westerheiden [2004b], in countries that were slow at establishing an evaluation system, the public confidence in the self-regulation of quality by academics was for the most part not higher than in countries that were pioneers in the introduction of evaluation systems.
The described causes have certainly had an impact—albeit it to various degrees—on the establishment of evaluation schemes in the pioneer countries: the Netherlands, France, and the UK. Inspired by the new management tools and approaches referred to above, the governments of these countries aimed at making universities more accountable for the funding they received, but also at restraining detailed regulation in favor of market-like mechanisms. As argued by Westerheijden:
Quality assurance can be seen, on the one hand, as a policy instrument supporting transparent markets for students and graduates by making information about quality differences public, and, on the other hand, as a safeguard against too blunt minimizing of
The original text was provided by the author to the Editorial of the journal Educational Studies.
http://vo.hse.ru
3
THEORETICAL AND APPLIED RESEARCH
quality levels in the free supply behavior of higher education providers on (quasi-) markets [Westerheijden, 2007. P. 12] .
However, it is important to state explicitly that quality assurance systems intended also to stimulate and foster quality improvement, taking into account the changing demands on higher education graduates that were a result of the development from industry-based to service-based economies and toward a knowledge society.
From 1990 on, the quality assurance movement was disseminated quickly throughout Western Europe. This dissemination was stimulated by the European Union’s 1994 Pilot Project on external evaluation methodology [Management Group, European Committee Meeting, 1995]. The project was based on the observation that most quality assurance systems used a four-phase approach that consisted of a national coordinating office, a self-evaluation report, a peer review phase, and a public report about the outcomes of the evaluation. In the project, one or two teaching programs in different knowledge domains were evaluated in all (then) EU countries and some other Western Europe countries. In 1998, as a major consequence of the project, the EU decided to launch the European Network for Quality Assurance in Higher Education (ENQA). Soon thereafter this network played an important role in the well-known Bologna Process initiated by the Bologna Declaration of June 1999 and aimed at harmonizing higher education in Europe. By the end of the 20th century almost all Western European countries had installed a policy and system of quality assessment in higher education, Germany being a remarkable exception until 2003.
3. Some As argued by Teichler [2007], “quality assurance” has become the concepts and most popular umbrella term in Europe, referring to all kinds of assess-definitions ments in higher education relating to activities aimed at developing and improving the quality of higher education. A key question, then, is what is meant by “quality” in higher education.
Looking over the literature, it quickly becomes obvious that there is no simple answer to the question. According to Westerheijden, all definitions of quality in the literature refer “to the link between the good or service under consideration and desires of customers as the essence of quality” [Westerheijden, 2007. P. 9] .
But with respect to higher education, there are different categories of customers representing distinct stakeholders. For instance, from the standpoint of students the quality of higher education refers to what happens in the classroom. The same holds true for the authorities of institutions, although they may focus on other aspects as well. But quality has a different meaning from the perspective of employers, who look for graduates as sources of manpower, but also for governments that are the main funders of higher education. In addition, quality may have a different meaning at different moments in
4
Educational Studies. 2014. No 4
Erik De Corte Г ■
Evaluation of Universities in Western Europe: From Quality Assessment to Accreditation
time or in distinct geographical regions. Taking all these considerations into account, Harvey and Green [1993] have grouped the widely differing conceptualizations of quality into five discrete but interrelated categories—quality can be viewed as exception, perfection, fitness for purpose, value for money, or transformative. Consequently, determining criteria for assessing quality in higher education requires an understanding of different conceptions of quality that inform the preferences of stakeholders. In this respect, employers may focus on exception, governments on value for money, students and authorities of institutions on fitness for purpose, starting thereby from descriptors for undergraduate and graduate programs and learning outcomes.
Other terms that are commonly used in relation to quality assurance are quality control, quality management, quality audit, quality assessment, evaluation, and accreditation. Quality control and quality management refer to systems used internally in higher education institutions. In quality control, the focus is on the measurement and maintenance of educational standards, whereas quality management aims to link quality control to planning and control cycles in the event of of possible changes and improvements.
Quality assessment and evaluation refer to external quality assurance systems oriented toward both the qualitative and quantitative measurement of quality. Quality audit refers to an evaluation procedure that focuses on quality management arrangements within higher education institutions. The latter system is not widespread at present, but as an example I refer to the evaluation of leadership and management of education at the University of Helsinki in 2008 [Saari, Frimodig, 2009].
Accreditation differs from quality assessment in function as well as in the kind of judgment. Schwarz and Westerheijden define accreditation as follows:
Institutionalised and systematically implemented evaluation schemes of higher education institutions, degree types and programmes that end in a formal summary judgment that leads to formal approval processes regarding the respective institution, degree type or programme [Schwarz, Westerheijden, 2004b. P. 2] .
Accreditation thus involves two phases: an evaluation phase provides the data and information as a basis for the approval that grants (or, alternatively, rejects) the “right to exist” of an institution, degree type, or programme in a higher education system. The approval is granted by a legitimate, supra-institutional organization or power. An obvious implication is that the focus of accreditation is on accountability.
http://vo.hse.ru
5
THEORETICAL AND APPLIED RESEARCH
4. Quality assessment focused on improvement of higher education
Higher education evaluation originated in the USA in the 1970s and took the format of accreditation from the very beginning. As mentioned above, the emergence of national evaluation schemes started in Western Europe in the second part of the 1980s. Whereas the introduction of such schemes was inspired by practices in the USA, it did not then take the format of accreditation. According to Teichler [2007], Western Europeans felt no need at the time to install an accreditation system for several reasons. A major reason was that the focus in Europe was more on quality improvement, and assessment was seen as a tool to initiate reflection on the quality and performance of higher education institutions as a lever for improvement. In addition, no need was felt to link quality assessment to an approval of the “right to exist” or install a system designed to identify the “black sheep” that should not have this right. Consequently, quality assessment, not accreditation, was the focus of evaluation schemes in Western Europe until the late 1990s. A representative example of this kind of higher education evaluation is the system that was developed in the Netherlands and Flanders (the Flemish part of Belgium), which were among the pioneer countries in developing and implementing quality assessment systems. The Dutch system was operational in 1988 and served as model for the Flemish scheme, which was implemented in 1992. In that year, an agreement between the two countries led to the organization of joint evaluation activities. I will briefly review here the Dutch system, which differs only slightly from the Flemish scheme. For a more detailed discussion of both systems I refer to Jeliazkova and Westerheijden [2004], Van Damme [2004], and Wijnen [2007] .
The Dutch quality assessment system can be described as consisting of the following steps:
• internal quality assurance as a starting point;
• writing of a self-evaluation report by the institution;
• visitation by an external review or evaluation committee;
• writing of a report by the external review committee;
• meta-evaluation by the inspectorate.
An important point is that the quality assessment procedure builds on the internal quality assurance arrangements of the institutions, which demonstrates the greater autonomy that was granted to them. Similarly, the basis of the evaluation was a self-study report compiled by the institution describing the individual programs to be evaluated. This report had to follow a pre-determined format covering the following topics:
6
• objectives of the program;
• structure and content of the program;
• input-throughput-output of students;
Educational Studies. 2014. No 4
Erik De Corte Г ■
Evaluation of Universities in Western Europe: From Quality Assessment to Accreditation
• description of the learning environment;
• feasibility;
• quality of the graduates;
• effectiveness of the organization;
• quality of the personnel;
• facilities;
• internationalization;
• internal quality care.
Next, an external review committee visited the institution over the course of several days. This committee was always composed in consultation with the institution, which shows again that the procedure was not at all a purely externally designed activity. The committee involved peers (i. e., scholars from the same discipline), an educational professional, and often a student. Based on the self-evaluation report, the committee critically reviewed the programs by analyzing documents (e. g., exams, report of internships, theses, etc.) and interviewing all relevant actors and stakeholders (authority members, teaching staff, students, graduates, and members of the administrative staff). The aim of these interviews was to clarify, verifiy, and gather additional information. The same committee visited similar programs in different universities (e. g., programs in educational sciences) .
Based on the interviews and their observations, the external committee compiled a report focusing on the strengths and points of improvement, using the topics addressed in the self-evaluation report as a guideline. As the committee visited similar programs in different universities, it was possible to derive some general comments and conclusions about the programs in the evaluated discipline. It is important here to stress that in the committee’s report, a strong focus was on suggestions and advices for improvement.
The final step, a meta-evaluation by the inspectorate, took place regularly but not always. The inspectorate could make additional comments and suggestions complementing those of the external committee. For instance, the inspectorate could stress certain remarks of the committee that needed special attention during the next visit.
As stated by Wijnen [2007], important characteristics of this quality assessment scheme were:
• final responsibility by the institution;
• important role of peer assessment;
• formal sanctions as an exception;
• improvement as a major goal;
• trust as a leading principle.
As mentioned before, this quality assurance procedure, which was repeated every six years, was mainly based on internal quality assessment. Higher education institutions themselves were granted the
http://vo.hse.ru
7
THEORETICAL AND APPLIED RESEARCH
main responsibility for quality care. The autonomy of the institutions was highly valued, and no decisions were taken by external agencies such as the external committee or the inspectorate. Consequently, external quality assessment was not threatening for the institutions as there were no formal sanctions. As argued by Wijnen, this evaluation system had a positive impact:
The self-study reports were public, there was a growing openness about educational programmes, external review committees brought ideas from elsewhere and there was a positive drive for improvement in the HEI’s (higher education institutions) [Wijnen, 2007. P. 128] .
Based on my own involvement in different evaluation committees, I can confirm that the role of improvement was very important in this quality assessment system. Writing the self-evaluation report was a very instructive experience for the institutions and stimulated reflection on their teaching programs and practices. I have often observed that the institutions themselves became well aware of their own strengths and weaknesses as a result. The reports of the external committee often involved useful suggestions for improvement and induced exchange of ideas and cooperation in joint projects and action programs. An important side effect was the increased interest in and attention to education (as opposed to research) in higher education institutions.
5. Accreditation focused on accountability of higher education
Notwithstanding the largely positive influence of the quality assessment system, the Netherlands and Flanders switched over from this system to an accreditation approach in the early years of the 21st century. More specifically, an accreditation system was installed in the Netherlands in 2003, and in 2005 the Dutch-Flemish Accreditation Organization (NVAO) was founded. In this way, the cooperation of the previous era between the two countries was continued. An important argument for this was that jointly they could play a more important role at the international level. In line with the definition given above, accreditation was described as a formal judgment that the quality of a degree course or an institution meets certain standards. This judgment is based on quality assessment. This shows that accreditation and quality assurance are connected [Dittrich, 2004]. In other words, accreditation was placed on top of quality assessment because it leads to an independent decision on whether or not the quality of a program is satisfactory, i. e., whether it achieves a basic level of quality.
Considering the positive influence of the quality assessment procedure, a relevant question is why the Netherlands and Flanders moved to accreditation. A major reason for the change was certain-
8
Educational Studies. 2014. No 4
Erik De Corte Г ■
Evaluation of Universities in Western Europe: From Quality Assessment to Accreditation
ly the impact of the Bologna Process, which stimulated international benchmarking to enable comparisons between programs within Europe. Accreditation makes such comparisons possible, which is important for the facilitation of student mobility in Europe, a major objective of the Bologna Process. An additional reason was that the quality assessment system slackened—over the years, it lost some of its effectiveness due to the development of routines and the fact that the expected improvements did not materialize. Therefore, the need was felt more and more to establish a new stricter and more effective approach to evaluation of higher education. Accreditation was also regarded as the appropriate answer because it facilitates governmental decisions. Indeed, a positive accreditation outcome has clear consequences for an institution or a program: it qualifies the institution for government funding, students can get bursaries, and the institution can issue degree certificates that are recognized by the government.
As of 2003, all teaching programs of higher education institutions in the Netherlands are subjected to an accreditation every six years. The accreditation involves four steps:
• writing of a self-evaluation report by the institution;
• visitation by an external review or evaluation committee;
• writing of a report by the external review committee;
• decision by the Dutch-Flemish Accreditation Organization.
This makes it clear that accreditation was built upon the previous quality assessment procedure. Indeed, the first three steps are the same, although there are differences in the specifics of these steps. For instance, more specific requirements were introduced for the format of the self-evaluation report. The report needs to address six subjects, involving altogether 21 facets that are related to explicitly stated criteria:
• objectives of the program
- domain-specific requirements
- level of the program
- orientation of the program (professional or academic)
• quality of the program
- relationship between the objectives and the content of the program
- requirements of professional and academic orientation of the program
- coherence of the program
- study load
- linking-up to the entrance level of students
- size of the program
- alignment of design and content of the program
- assessment and testing
http://vo.hse.ru
9
THEORETICAL AND APPLIED RESEARCH
• quality and quantity of the personnel
- professional or academic orientation
- quality of personnel
- quantity of personnel
• available facilities
- material facilities
- guidance of students
• internal quality assurance
- evaluation of the results
- measures for improvement
- involvement of co-workers, students, alumni and professionals
• academic outcomes
- realized level of education
- output of the program
The visit of the external evaluation panel is organized by an intermediate agency which is in charge of and responsible for the committee’s report. A major change was that the committee now had to assess the 21 facets on a four-point scale: unsatisfactory, satisfactory, good, or excellent. Based on these judgments the six subjects were evaluated dichotomously: satisfactory or unsatisfactory. Unsatisfactory judgments had to be justified. A single unsatisfactory judgment for one of the six subjects was sufficient to refuse accreditation.
The committee’s report was made available to the institution that had the opportunity only to check it for possible inaccuracies, not to comment on or to criticize the assessment of the committee. The final version of the report was delivered to the Dutch-Flemish Accreditation Organization, which made a decision about accreditation.
This Dutch accreditation system was more or less copied in Flanders, and, as shown by Schwarz and Westerheijden [2004b], was again fairly representative for Western Europe. For instance, in all countries, study programs are the unit of analysis, all degree programs are addressed (except for doctoral programs in most countries), the modes of inquiry and assessment are similar for core elements, and a national accreditation agency was established with a close link to the government. Of course, there were also differences between the countries with regard to the specific elaboration and implementation of the accreditation schemes [Witte, 2009] .
It is obvious that this accreditation scheme differed substantially from the previous quality assessment system. Wijnen [2007] describes the major changes as follows:
• from improvement to accountability
• from trust to distrust
• from peers to bureaucrats
• from institutions to governments
• from decentralization to centralization.
10
Educational Studies. 2014. No 4
Erik De Corte Г ■
Evaluation of Universities in Western Europe: From Quality Assessment to Accreditation
The shift from a focus on improvement to accountability and the fact that the responsibility for quality assurance was removed from the higher education institutions were certainly two very important changes. As argued by Westerheijden [2007] the ownership of quality assurance shifted from the institutions to the state, and the improvement dimension almost vanished. Because of these fundamental changes, but also because of the bureaucratic burden and the fact that the system became very expensive, the accreditation scheme was heavily criticized, especially by the higher education institutions. This led to very intensive discussions and consultations, which resulted in a revised accreditation system that has been operational in the Netherlands since 2012. A similar revised system was introduced in Flanders during the 2013/2014 academic year.
A major step in the Bologna Process was the adoption of the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG) of the European Network for Quality Assurance in Higher Education (ENQA) at a ministerial meeting in Bergen, Norway in 2005. The ESG were developed to meet the need for a common understanding of quality assurance in European higher education. The new Dutch accreditation framework was developed on the basis of these standards and guidelines. In developing the new scheme, the NVAO has attempted to translate and rearrange the ESG in order to shift the focus from quality assurance to quality enhancement (see: www.nvao.net) .
In the new system of accreditation, the focus is still on the quality of individual programs. However, a major innovation is that higher education institutions can now ask the NVAO to conduct an institutional quality assurance assessment. Such an institutional assessment is meant to determine whether an institution as a whole has implemented an effective quality assurance system that enables it to guarantee the quality of its individual programs. Institutions that have received a positive evaluation as result of an institutional assessment can benefit from a so-called limited program assessment scheme: an assessment panel of independent experts evaluates an individual program on a limited number of standards that relate to the educational quality and the content of the program. It is assumed that this can reduce the accreditation burden by 25 percent; to meet the same purpose, limitations are imposed with regard to the number and size of the annexes.
It is important to mention that for institutions that do not request an institutional assessment, or received a negative judgment on such an assessment, the individual programs are subjected to an extensive accreditation procedure that is more or less similar to what it was before.
6. Revising the accreditation approach to meet criticisms
http://vo.hse.ru
11
THEORETICAL AND APPLIED RESEARCH
6.1. Institutional quality assurance assessment
6.2. Limited program assessment
The institutional quality assurance assessment addresses five interrelated issues that translate into five standards:
• the institution has a broadly supported vision of the quality of its education and the development of a quality culture;
• the institution pursues an adequate policy in order to realize its vision of the quality of its education. This comprises such aspects as policies in the field of education, staff, facilities, accessibility for students with disabilities, embedding of research in the education provided;
• the institution has insight into the extent to which its vision on the quаlity of its education is realized. It gauges and evaluates the quality of its programs regularly;
• the institution can demonstrate that it systematically improves the quality of its programs whenever required;
• the institution has an effective organization and decision-making structure with regard to the quality of its programs which clearly defines the tasks and responsibilities.
The NVAO appoints an audit panel of at least four members to conduct the assessment. The panel involves administrative, educational, and audit expertise and is counseled by a process coordinator and supported by a secretary. Based on the information provided by the institution and two visits to the institution, the panel evaluates each of the five standards using the following judgments: meets, does not meet, or partially meets the standard. Subsequently, the panel formulates a substantiated general judgment about the overall question of whether the institution has implemented an effective quality assurance system that enables it to guarantee the quality of its educational programs. The judgment can be positive, negative, or conditionally positive. In case of a negative or conditionally positive evaluation, the institution can, under certain circumstances, be given a chance for remediation within a period of, for instance, one year.
As mentioned above, the limited program evaluation is applied to programs of institutions that have obtained a positive result following an institutional quality assurance assessment. The assessment is based on a discussion with peers and is focused on the quality and the content of the program. Three main issues are addressed that translate into three standards:
• Intended learning outcomes: The intended learning outcomes of the program have been concretized with regard to content, level and orientation (bachelor or master; academic or professional); they meet international requirements;
•Teaching-learning environment: The curriculum, staff and program-specific services and facilities enable the incoming stu-
12
Educational Studies. 2014. No 4
Erik De Corte Г ■
Evaluation of Universities in Western Europe: From Quality Assessment to Accreditation
dents to achieve the intended learning outcomes. The curriculum, staff, services and facilities constitute a coherent teaching-learning environment for students;
• Assessment and achieved learning outcomes: The program has an adequate assessment system in place and demonstrates that the intended learning outcomes are achieved.
The review committee for a limited program assessment is convened by the institution or by an external quality assessment agency in consultation with the institution. The panel has to be independent of the institution and must be approved by the NVAO. The panel consists of at least four members, among whom must be at least two domain experts and a student. Besides domain expertise, the committee commands international expertise, educational expertise, assessment and audit expertise, and, if relevant, professional expertise relevant to the program. The members of the panel receive a limited number of specified documents to read before the visit to the institution. Based on the information in the documents and the discussions during the visit, the committee formulates a substantiated judgment regarding each of the three standards on a four-point scale: unsatisfactory, satisfactory, good, or excellent. Subsequently, the committee formulates a substantiated overall judgment about the quality of the program on the same four-point scale. The four judgment options are defined as follows:
• Unsatisfactory: the program does not meet the current generic quality standards and shows serious shortcomings in several areas;
• Satisfactory: the program meets the current generic quality standards and shows an acceptable level across its entire spectrum;
• Good: the program systematically surpasses the current generic quality standards across its entire spectrum;
• Excellent: the program systematically well surpasses the generic quality standards across its entire spectrum and is regarded as an (inter) national example.
The secretary prepares an assessment report of about 20 pages that is finalized after it has been checked by the members of the panel. The main part of the report consists of the panel’s judgments about the three standards and the underpinnings of these evaluations. Suggested measures for improvement are presented in a separate section.
Based on the assessment report the institution applies to the NVAO for accreditation. The NVAO can take one of three possible decisions: to accredit the program, not to accredit it, or to grant-under certain stipulated circumstances-an improvement period.
http://vo.hse.ru
13
THEORETICAL AND APPLIED RESEARCH
7. Final comments and future perspective
Besides the fact that the introduction of accreditation has led to a shift in the focus of higher education evaluation from improvement to accountability, Wijnen [2007] lists five additional unwarranted effects of the accreditation system—it:
• is expensive;
• stimulates bureaucracy,
• leads to uniformity,
• stimulates window dressing,
• hinders real innovation.
It is now certainly too early to fully judge the extent to which the revised accreditation system can remove or at least reduce these negative effects. It cannot be denied that an effort has been made to focus more on the content and the quality of the programs. Nevertheless, I tend to agree with the viewpoint of van Schijndel and van Kemenade [2011] that the criticisms of the previous system have only been partially removed. For instance, it must be admitted that eliminating the nightmare of evaluating the previous scheme’s 21 facets has brought with it a certain simplification. On the other hand, however, an institutional assessment has been introduced and is complemented with a limited program assessment. The danger that this could lead to duplication is still present. To what degree the bureaucratic burden will decrease remains thus to be seen, and the same holds true for the reduction of costs.
Notwithstanding the effort to focus more on quality, the system remains in the first place an output measure that aims to assure a basic level of quality, and accountability still tends to dominate the improvement function. Moreover, the scheme still involves the risk of receiving a negative accreditation, and this, together with the focus on basic quality, can indeed lead to uniformity and window dressing. It encourages the tendency to conform to traditional, accepted ideas and patterns and to over-accentuate a program’s positive aspects and hide its weaknesses in the self-evaluation report. Consequently, accreditation is also an obstacle for innovation because institutions do not want to take any risk: new approaches and innovative ideas could be rejected by the accreditation organization (see also [Dittrich, Klaassen, 2007]) .
According to Brennan [2007] the two factors that are most important for quality improvement are self-assessment and peer review—self-assessment provides the academic community an internal opportunity for discussion, dialogue, and interaction, whereas peer review induces the exchange of good practices and innovative ideas with external evaluators. However, these two factors do not at all prevail in the accreditation philosophy and approach and are rather at odds with the dominant accountability function.
14
Educational Studies. 2014. No 4
Erik De Corte Г ■
Evaluation of Universities in Western Europe: From Quality Assessment to Accreditation
Table 1. Two-way system of intelligent accountability: separating control from improvement in an accreditation process
External assessment Internal assessment
Function Accountability Improvement
Compulsory Voluntary
Object Institution Programme
Basic rules External minimal standards Fitness for purpose (the organisation sets its owm standards)
Stakeholders All, but mainly the government All, but mainly the student and wotld of work
Organisation Governmental The institution inself, using external stakeholders
Metodology Document analysis (including management review) Site visit Comprehensive self-evaluation report Site visit Improvement plan
Accreditation decision Once in six years Professional audition, off-campus audiences Control Yearly Peers Continuous improvement and commitment
As mentioned earlier, Wijnen [2007] has also pointed to a move from trust to distrust as a major difference between the original quality assessment and the accreditation approach. On this point, Wes-terheijden [2007] has characterized external quality assurance as policy instruments of “organized distrust.” Policymakers and higher education managers deplore this state of affairs and would like to return back to a high-trust situation, but at the same time, they want to keep in place a form of quality assurance that has proven to be a useful instrument for them. Westerheijden comments on this: “The fundamental question here is whether it is possible at all to shift back to a high-trust situation. Who coined the one-liner: Innocence once lost cannot be regained?” [Westerheijden, 2007. P. 15] .
It seems to me that this question is still relevant with respect to the revised accreditation system. Especially considering that accreditation seems to be a serious hindrance to the improvement function, the important issue is whether it is possible to combine the accountability and improvement functions in one quality assurance scheme.
I tend to support the viewpoint of several authors that both functions are relevant and needed, but that they require a different approach [van Kemenade, Hardjono, 2010; van Schijndel, van Kemenade, 2011; Wijnen, 2007]. In this regard, taking into account the results of an empirical study with a large group of university lecturers, van Kemenade and Hardjono have proposed what they call a two-way system
http://vo.hse.ru
15
THEORETICAL AND APPLIED RESEARCH
References
of intelligent accountability wherein accountability and improvement are separated as shown in Table 1 [van Kemenade, Hardjono, 2010. P. 266].
In this system, accountability is realized through an external audit based on an analysis of documents and a site visit by external professional auditors to objectify the performance outcomes of an institution or a program. Improvement is pursued and achieved through internal audits organized by the higher education institutions themselves based on self-evaluation and site visits by peers.
1. Brennan J. (2007) The Multiple Functions of Evaluation and Quality Assessment. Quality Assessment for Higher Education in Europe (ed. A. Cavalli), London: Portland Press, pp. 17-25.
2. De Neve H. M.F., Janssen P. T. (1984) Validity of Student Evaluation of Instruction. Higher Education, vol. 11, no 5, pp. 543-552.
3. Dittrich K. (2004) Methodological Common Instruments for Assessment and Accreditation in the European Framework. Paper presented at 2th Seminar on Methodological Common Instruments for Assessment and Accreditation in the European Framework, Santander, Spain, 28-30 July, 2004.
4. Dittrich K., Klaassen L. (2007) The Accreditation System in the Netherlands and Flanders. Quality Assessment for Higher Education in Europe (ed. A. Cavalli), London: Portland Press, pp. 115-125.
5. Harvey L., Green D. (1993) Defining Quality. Assessment and Evaluation in Higher Education, vol. 18, no 1, pp. 9-34.
6. Jeliazkova M., Westerheijden D. (2004) The Netherlands: A Leader in Quality Assurance Follows the Accreditation Trend. Accreditation and Evaluation in the European Higher Education Area (eds S. Schwarz, D. Westerheijden), Dordrecht: Kluwer Academic Publishers, pp. 323-345.
7. Management Group, European Committee Meeting (1995) European Pilot Project for the Evaluation of Quality in Higher Education. Brussels: European Commission DG XXII “Education, Training and Youth”.
8. Saari S., Frimodig M. (eds) (2009) Leadership and Management of Education. Evaluation of Education at the University of Helsinki 2007-2008. Helsinki: University of Helsinki.
9. Schwarz S., Westerheijden D. F. (eds) (2004a) Accreditation and Evaluation in the European Higher Education Area. Dordrecht: Kluwer Academic Publishers.
10. Schwarz S., Westerheijden D. (2004b) Accreditation in the Framework of Evaluation Activities: A Comparative Study in the European Higher Education Area. Accreditation and Evaluation in the European Higher Education Area (eds S. Schwarz, D. Westerheijden), Dordrecht: Kluwer Academic Publishers, pp. 1-41.
11. Teicher U. (2007) Accreditation: The Role of a New Assessment Approach in Europe and the Overall Map of Evaluation in European Higher Education. Quality Assessment for Higher Education in Europe (ed. A. Cavalli), London: Portland Press, pp. 55-64.
12. Van Damme D. (2004) Quality Assurance and Accreditation in the Flemish Community of Belgium. Accreditation and Evaluation in the European Higher Education Area (eds S. Schwarz, D. Westerheijden), Dordrecht: Kluwer Academic Publishers, pp. 127-157.
16
Educational Studies. 2014. No 4
Erik De Corte Г ■
Evaluation of Universities in Western Europe: From Quality Assessment to Accreditation
13. Van Kemenade E., Hardjono T. W. (2010) A Critique of the Use of Self-Evaluation in a Compulsary Accreditation System. Quality in Higher Education, vol. 16, no 3, pp. 257-268.
14. Van Schijndel B., van Kemenade E. (2011) Hoe Goed is Het Nieuwe Ac-creditatiestelsel? Tijdschrift voor Hoger Onderwijs & Management, vol. 18, no 4, pp. 6-10.
15. Westerheijden D. F. (2007) The Changing Concepts of Quality in the Assessment of Study Programmes, Teaching and Learning. Quality Assessment for Higher Education in Europe (ed. A. Cavalli), London: Portland Press, pp. 5-16.
16. Wijnen W. (2007) Accreditation in the Netherlands: An Improvement of External Quality Assessment? Quality Assessment for Higher Education in Europe (ed. A. Cavalli), London: Portland Press, pp. 127-133.
17. Witte J. (2009) Parallel Universes and Common Themes: Reforms of National Curricular Governance in Germany, the Netherlands, France and England in the Context of the Bologna Process. European Integration and the Governance of Higher Education and Research: The Challenges and Complexities of an Emerging Multi-Level Governance System (eds A. Amaral, P. Maassen, C. Musselin, G. Neave), Dordrecht: Kluwer Academic Publishers, pp. 227-255.
http://vo.hse.ru
17
