CC BY
42
Cristina d'Alessandro1, Vasileios Psarianos2
1Fondazione FORMIT 2National Technical University of Athens E-mail: с.dalessandro@formit.org; bpsarian@mail.ntua.qr
THE TEMPUS PROJECT HDMCURF: QUALITY ASSURANCE PROCEDURES
This paper presents the Quality Assurance process developed in the framework of the project «HDMCuRF - Highway Design and Management: Curricular Reform for Russian Federation Design and Implementation of Higher Education Master Courses in Russia» funded by the Education, Audiovisual and Culture Executive Agency (EACEA) of the European Commission within the framework of the TEMPUS IV program. The project supported the transfer of EU higher education best practices in Highway Design and Management to develop advanced university courses to train competent resources in Russian Universities. The project made available, to Russian Universities, knowledge, management approaches, modelling and assessment techniques which are adapted and taken over by the Russian higher education system. Those best practices are based on the principles of the European Credit Transfer System and the recognition of the university degree. The EU participants shared best practices, training management and quality assurance approaches according to the Bologna principles and in line with Level 7 of the European Qualification Framework to support the development and diffusion of an innovative experience in technical higher education in Russian institutions supporting the capacity and knowledge building in highway design and management.
Key words: Quality Assurance, European Qualification Framework, Master course, learning outcomes, TEMPUS program, Russian Federation.
The Quality Assurance process
Quality control and monitoring have been considered, since the beginning of the HDMCuRF project (www.formit.it/hdmcurf) and even in the proposal phase, an integral part of all project activities and results. As a process going on during all the project activities, it has been inserted as a work package itself in the project, with its deliverables and results to be monitored. The responsible for the Quality Assurance process, the Quality Manager, has been Fabio Bisogni from FORMIT.
The quality assurance has been achieved at internal and external level. The internal quality review of the project activities has been done by a quality task force composed by 1 person from each partner, in charge of evaluating the right fulfilment of quality standards and procedures. The quality monitoring has been done on the basis of a series of quality indicators, to monitor the project activities.
The Quality Assurance has been carried out according to the guidelines expressed in a Quality Manual. These guidelines follow the requirements set up in the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ENQA, 2009). The Quality Plan contained all the specific actions to be taken to correctly develop the project activities according to high quality standards.
Outputs coming from the quality assurance process, collected in Quality Assurance reports, have been used to refine the project's running, taking into consideration feedback provided by all the actors involved in the monitoring activities.
As a foundation for the quality of the teaching material and the achievement of the desired learning outcomes (European Parliament, 2008), the educational material has been subject to quality control and assessment criteria according to a strict quality review process. The review process has been managed by the Review Board which was coordinated by the Review Manager in the person of Basil Psarianos from NTUA. In the following picture it is reported a schema for the Quality Governance inside the project.
Outputs
Quality Manual
The Quality Manual is a methodology document for internal use of the consortium. The Quality Manual sets out the organization, management, standards and procedures used in the project to ensure it meets its objectives with the declared and desired quality. The following activities have been made in order to deploy the quality manual:
- Collection of quality guidelines both in Russian and EU Universities;
- Design and definition of quality indicators for carrying out the project activities.
Quality Plan
The Quality Plan is a report that contains all the specific actions to be taken to correctly develop project activities according to high quality standards, and it is for internal use of the consortium. The Quality Requisites Planning procedures have
been adopted by the project team in order to respect the plan activities and the requirements. The following activities have been done for deploying the Quality Plan:
- Definition of the Quality Plan features according to Quality Manual guidelines;
- Consultation with all partners to check the proposed quality procedures.
The executive planning defined the delivery requirements (what), period or date of review (when) and procedures for controlling and approving the results (how).
At the end of the each year of the project, the Quality Manager delivered a Quality Assurance document that reported on Quality assurance activities, and elaborates internal and external quality monitoring results.
Quality Assurance reports
Reports on monitoring the Quality Assurance are to be disseminated at international level. The following activities have been done in order to deploy the quality assurance reports:
- Internal feedback report;
- External feedback report.
The internal quality review was intended as the intrinsic quality of project activities and results with the support and participation of all partners, while external quality was intended to be as an external tutorship of the project implementation
and its result referring to the definition and building of the Master course.
Internal quality
The quality monitoring has been made on the basis of a series of quality indicators, shared with all partners, took into account when running the project activities. The scope of the internal quality process was assuring that the quality of the produced deliverables (Outputs or Outcomes) was implemented according to the required level. Control was also foreseen for progress reports during the project development, in case of in-time corrections. Quality control procedures have been related to each work packages' deliverables (outputs/outcomes) and to respect of deadlines. Monitoring process supervised the project in-between the deadlines.
Teaching material review process
A review process of the teaching material has been introduced and a Review Board has been appointed. The process started on December 2012. The teaching material has been translated in Russian. Then, it has been reviewed by three persons, one from each Russian University. All comments from the 3 Russian universities and for all modules have been received and sent to European teachers. Teachers have taken care of the comments into their lecture material and
Figure 1. Quality Governance
resubmitted to Russian universities their material with the comments incorporated. The individual steps of this process are presented in more details in the discussion that follows. The teaching material review process was part of the internal quality.
Internal survey for the training workshops
The internal survey for training workshops has been set in order to implement the internal quality monitoring of the project. This phase has involved Russian university staff, involved in European training workshops. The results collected after this survey will be used in order to verify the HDMCuRF project quality standards compliance with the Quality Manual. Moreover, the survey results will be discussed among all partners of the Consortium under the supervision of the Quality Manager. To be noted that this survey is not intended to evaluate each single course deployed to Russian universities' staff, but it is intended to ascertain the effectiveness of each organised workshop for the HDM Master implementation.
External Quality
The scope of External Quality is assuring that the quality of the Master organisation and implementation was at the required level. For doing that, during the first year, each partner involved other Universities colleagues, in order to receive an objective contribution to the quality aspects of the project. The external monitoring was realized through a web-based survey in which a feedback on TEMPUS programme and on the project was asked. Target persons were University colleagues, both from the own university and from EU/non EU universities. The survey was on a voluntary
base, no expenses are foreseen for this task. In the second and third year, the valuable input for evaluation of the quality of new courses and teaching methods was done by involved students. Students' feedback was collected through a questionnaire, for both the two years of the Master course.
External monitoring via web
The following survey is set in order to implement the external quality monitoring of the project. This phase has involved partner universities colleagues which have already (or not) worked and participated in past edition of TEMPUS Projects. The results collected after this survey have been used in order to verify the HDMCuRF project quality standards compliance with the Quality Manual. Moreover, the survey results will be discussed among all partners of the Consortium under the supervision of the Quality Manager. Questionnaire
Welcome to the HDMCuRF questionnaire. This questionnaire is for collecting outlooks on our project. Please read the executive summary that follows and express your opinion on it, answering the questions. We thank you for the time you dedicate to it!
Executive summary
To design, develop and implement a new curriculum programme in Highway Design and Management in a joint effort between EU and Russian Universities in line with Bologna requirements. The former will transfer EU best practices, experiences and methodologies according to the Bologna process to support the development and diffusion of an innovative experience in technical higher education in RU institutions
iL
□2D
□D
Are project objectives clear?
Are courses in line with project objectives?
Is training model transferable easily from EU to RU?
3) Are courses contents sufficient for the future Master implementation? 1 2 3 4 5
4) Is courses duration adequate for the future Master implementation? 1 2 3 4 5
5) Is training material, provided along the courses, sufficient for the future Master implementation? 1 2 3 4 5
5
2
3
4
5
2
3
4
5
2
3
4
Figure 2. Internal survey for training workshops
supporting the capacity and knowledge building in highway design and management engineering. The realization of a Master programme will follow a scenario assessment taking into account of RU sector needs and requirements, and EU best practices developed.
The questionnaire collects data in an anonymous way and the data are used only for project purpose. Please start the questionnaire answering this first question.
1. Have you ever participated to the TEMPUS Programme?
• Yes (please, go to section a.)
• No (please, go to section b.)
External monitoring addressed to students
The following survey is set in order to implement the external quality monitoring of the project. This phase will involve Partner Countries university students, involved in Master courses. The results collected after this survey will be used to verify the HDMCuRF project quality standards compliance with the Quality Manual and Quality Plan. Moreover, the survey results will be discussed
among all partners of the Consortium under the supervision of the Quality Manager. To be noted that this survey is intended to evaluate each single module/sub-module deployed by Partner Countries university staff, to ascertain also the effectiveness of each organised module/submodule for the HDM Master implementation.
Questionnaire
Dear student,
This questionnaire is intended to collect data and information that might be used to support teachers in improving her/his didactics as a part of the Master course in Highway Design and Management.
Quality Assurance results
Here follows a list of the main results for the Quality Assurance process:
- Delivery of a Quality Manual;
- Delivery of a Quality Plan;
- Delivery of three Quality Assurance reports, one per each year with the identification and valorisation of the indicators for all project activities;
- Delivery of one external monitoring survey on the website (http://formit.it/hdmcurf/
a. Section to be filled by persons who have b. Section to be filled by persons who have
already participated to the TEMPUS never participated to the TEMPUS
Programme. Programme.
1. What do you think about the TEMPUS 1. Do you know the TEMPUS Programme
programme objectives and principles? objectives and main principles? • Yes
2. With reference to the provided Executive • No
Summary, do you think that the HDMRCuRF 2. Have you ever participated to the
project respects and it is complaint to the implementation of EU projects in
TEMPUS Programme General Objectives? collaboration with international partners?
Yes, • Yes
Because... • No
No because 3. With reference to the provided Executive Summary, do
you think that objectives and
3. Considering you past experience, what are activities of the HDMCuRF project are
the most relevant "criticalities" you have expressed in a clear and understandable
perceived in implementing the project tasks way?
• Yes, because...
and deliverables?
• No, because...
4. Considering your past experience, would do 4. On the basis of the provided Executive
you like to give suggestions to your Summary, what are, in your opinion, the
colleagues in order to avoid common strengths and weaknesses of the HDMCuRF
difficulties in implementing the project project?
tasks? • Strenghts:
• Waeknesses:
5. Considering your past experience, what is 5. Would you give some general suggestions
the best way to sustain the TEMPUS to properly implement the tasks and to
project's results? reach the objectives of the project?
Figure 3. External monitoring via web
Part A -
A.1
A.2
A.3
A.4
A.5
Reference course/module
School/Faculty (reference code)
Master course (reference code)
Module (reference code)
Sub-module (reference code)
Number of teaching professors
*The evaluation scale is: 1 - Low, 5 - High
Part B - The student
B.1
Gender
M
B.2
Age
B.3
City
B.4
Employment status
unemployed temporary employed
part-time employed full-time employed
B.5
Academic year
B.6
Number of successful exams
B.7
Attended modules
B.8
Attendance
at this module/sub-module
assiduous over 80% regular 50%-80%
irregular 20%-49% occasional
B.9
Secondary school Diploma (type of)
tbd tbd
tbd tbd
F
*The evaluation scale is: 1 - Low, 5 - High
Part C - Infrastructures
C.1
Adequacy of classrooms
C.2
Equipments and integrative activities classrooms
if not foreseen, please indicate Not Applicable (N.A.)
1 2 3 4 5
1 2 3 4 5 N.A.
*The evaluation scale is: 1 - Low, 5 - High
Part D - Didactics organisation
D.1 Did the teacher provide clear explanations on the module/sub-module programme and its objectives? 1 2 3 4 5
D.2 Was the lesson methodology satisfactory in terms of teaching materials? 1 2 3 4 5
D.3 Were the integrative activities (exercises, laboratories, workshop, etc.) useful for the learning progress? 1 2 3 4 5
D.4 Was the examination procedure explained in a clear way? 1 2 3 4 5
D.5 Was the study effort balanced by the acquired credits? 1 2 3 4 5
*The evaluation scale is: 1 - Low, 5 - High
Part E. Overall satisfaction
E.1
Which is the overall level of satisfaction of the entire module/sub-module?
1 2 3 4 5
*The evaluation scale is: 1 - Low, 5 - High
Part F. The teacher For each teacher, please answer to the following questions:
F.1 Was the teacher clear in explaining the module/sub-module contents? 1 2 3 4 5
F.2 Did the teacher motivate interest toward the module/submodule contents? 1 2 3 4 5
F.3 Did the teacher respect the planned timetable for the deployment of her/his didactics (lessons, consulting hours, etc.)? 1 2 3 4 5
F.4 Was the teacher really available for explanation and clarifications? 1 2 3 4 5
F.5 Was the teaching material, given or suggested by the teacher, adequate for the study of the module/sub-module contents? 1 2 3 4 5
F.6 Did the teacher pay attention to arising problems? 1 2 3 4 5
F.7 Did the teacher explain carefully the didactics' evaluation process, showing adequately this survey? 1 2 3 4 5
*The evaluation scale is: 1 - Low, 5 - High
index.php/survev) targeted to colleagues of Universities participating to the project;
- Realisation of three internal surveys for the training workshops held in Naples, Stockholm, and Athens filled by the Russian trainees: 9 in Naples, 10 in Stockholm, 9 in Athens;
- Realisation of the external survey targeted to students attending first and second year of the master courses in MADI (12 students), ASTU (12 students), and OSU (9 students).
- Review of the teaching material by Russian professors, chaired by NTUA.
- Continuous monitoring of the quality of delivered outcomes/outputs.
Educational Material Review process
The review process was managed by the Review Board. Members of the Board are Basil Psarianos (NTUA, Review Coordinator), Costas Antoniou (NTUA), Haneen Farah (KTH), and Alfonso Montella (UniNA).
The review process consisted of the following steps:
1) The teaching material is translated in Russian.
2) The teaching material of each sub module is, first, reviewed by three persons1, one from each Russian University2:
a) Prof Victor Ushakov sends to the Review Coordinator the list and e-mails of the MADI's reviewers (Senior Professors);
b) Prof Galina Merentsova sends to the Review Coordinator the list and e-mails of the ASTU's reviewers (Senior Professors),
c) Prof Oleg Krikotov sends to the Review Coordinator the list and e-mails of the OSU's reviewers (Senior Professors).
3) The material is then revised by the responsible teacher.
4) The revised teaching material of each sub module is reviewed again by two persons, one from an EU University and one from Russian Federation:
a) Prof Costas Antoniou sends to the Review Coordinator the list and e-mails of the NTUA's reviewers;
b) Prof Haneen Farah sends to the Review Coordinator the list and e-mails of the KTH's reviewers;
c) Prof Alfonso Montella sends to the Review Coordinator the list and e-mails of the UniNA's reviewers,
d) The Review Board selects the Russian Federation reviewers, selecting reviewers already involved in the step 2.
5) The teaching material is finalized by the responsible teacher.
6) The final teaching material is approved by Project Management Board during a Skype meeting.
The final teaching material was checked by the Project Manager and proposed for approval by Project Management Board on June 28, 2013.
One of the basic qualitative criteria for developing the educational material was to provide the students ofthe Master's course the corresponding educational capacity as described for the Level 7 of the European Qualification Framework. This framework is presented shortly in the following.
European Qualification
Framework as a reference
European Union Council in an effort to provide a reference system for associating various educational systems in the country members and the corresponding level degrees of graduates of educational institutions throughout Europe has established an eight level educational framework the so called European Qualification Framework (EQF). This framework worked initially on a volunteer basis but now it is an established framework incorporated in the member countries of the EL) in their educational System (European Commission, 2008).
The EQF ranges from the Level 1 (basic education) to Level 8 which represents the upmost level of education. In this Framework Ph.D. Diploma is associated with the highest Level 8 while the M.Sc. Diploma is associated to Level 7.
The EQF defines the various educational levels explicitly through Descriptors. These are:
1Teaching material of modules 8 and 11 is prepared by Moscow State Automobile and Road University (MADI). Forthis material, only reviewers selected by Altai State Technical University named after I.I. Polzunov (ASTU) and Orenburg State University (OSU) are used.
2Russian reviewers can be selected also in Universities not involved in the HDMCuRF project in order to reach a greater involvement of the Russian Academia.
o Knowledge
o Skills
o Autonomy & Responsibility
For level 7 (M.Sc.) specifically these Descriptors have the following meaning:
1. Knowledge is defined as:
• highly specialised knowledge, some of which is at the forefront of knowledge in a field of work or study, as the basis for original thinking;
• critical awareness of knowledge issues in a field and at the interface between different fields.
2. Skills are defined as the ability:
• to apply specialist research and problemsolving skills, including analysis and synthesis,
• to develop new knowledge and procedures and
• to integrate knowledge from different fields.
3. Finally Autonomy and Responsibility are defined as the competence:
• in managing and transforming work or study contexts that are complex, unpredictable and require new strategic approaches;
• to take responsibility for contributing to professional knowledge and practice and for reviewing the strategic performance of teams.
As shown in Figure 4 all efforts have been made to bring the final content of the educational material in line with Level 7 of the EQF.
The actors of the Review process, i.e. EU teachers and the Russian teachers, and the process itself, coordinated by NTUA, was a fundamental part of this effort (Fig. 5).
By reviewing the material the focus was in particular was twofold:
- To set up a complete set of power point presentations and supporting materials (standards, guidelines, papers, videos, etc.).
- Not to rank the teaching material but more specifically...
Figure 4. Balance among Teaching Material and EQF
Figure 5. Review process
• It was aimed at improving the teaching material's quality in order to provide the intended learning outcomes of the master programme, and subsequently
To evaluate the appropriateness and the shortcomings of the teaching material as well as suggestions for improvement.
This process is a continuous and dynamic process that has to be followed constantly in the coming years by all three Russian Universities that provide the programme.
It is expected that these high qualitative standards embodied in the current Master Programmes will serve the educational system and the society of the Russian Federation in the best way.
References:
1. ENQA - European Association for Quality Assurance in Higher Education, (2009). Standards and Guidelines for Quality Assurance in the European Higher Education Area, http://www.enqa.eu/pubs.lasso
2. European Parliament and the Council of the European Union, (2008). Recommendations of the European Parliament and of the Council of 23rd April 2008 on the establishment of the European Qualification Framework for Lifelong Learning. Official Journal of the European Union, Clll, pp.1-7
3. European Commission, Education and Culture, (2008). The European Qualifications Framework for Lifelong Learning (EQF). DOI 10.2766/14352
Information about authors:
Cristina d'Alessandro, Fondazione FORMIT, c.dalessandro@formit.org Vasileios Psarianos, National Technical University of Athens, bpsarian@mail.ntua.qr
