CC BY
30
УДК378.147:005.6
DOI: 10.18413/2313-8971-2020-6-2-0-3
S.A. Kucheryavenko S.G. Stenyushkina,
A.N. Nazarova,
B.A. Bondarenko
Quality assessment of university students training
Belgorod State National Research University, 14, Studencheskaya Str., Belgorod, 308007, Russia Kucheryavenko_s@bsu.edu.ru*
Received on March 27, 2020; accepted on June 09, 2020; published on June 30, 2020
Abstract. Focus on the quality of education is a priority of the state program of the Russian Federation "Development of education" (approved by the RF government decree of December 26, 2017 № 1642). The main activities of the direction "Improving the management of the education system" of this program include measures to implement mechanisms for evaluating and ensuring the quality of education in accordance with state educational standards. Assessment of the quality of education in the universities is determined by the necessity of the competence orientation of the educational process and its content, technology implementation, re-evaluation procedures, technologies and tools to assess the quality of training students within the framework of the competence-oriented educational standards. The article deals with the problem of evaluating the quality of training of students of higher educational institutions. The definition of the concept of quality of education from the point of view of stakeholders is given. The article presents a method for evaluating the formation of learning outcomes based on a two-criteria approach (competence and disciplinary). The main criteria for evaluating the formation of learning outcomes are highlighted. The results of testing the methodology in one of the areas of training are considered. To study the level of competence formation, a scale of assessment of competence formation (the stage of competence formation) is proposed. The results of testing the methodology for evaluating the formed learning results presented in the article can be useful for interested persons who are directly involved in the process of designing, developing and implementing educational programs, as well as in making methodological and managerial decisions aimed at improving the quality of education.
Key words: quality of education; quality of training; level of competence; competence approach; evaluation of residual knowledge; the system of internal evaluation of the quality of education; quality management system; educational organization Information for citation: Kucheryavenko S.A., Stenyushkina S.G., Nazarova A.N., Bondarenko B.A. (2020), "Quality assessment of university students training", Research Result. Pedagogy and Psychology of Education, 6 (2), 26-33, DOI: 10.18413/23138971-2020-6-2-0-3.
Кучерявенко С.А, Стенюшкина С.Г, Назарова А.Н.,
* • ,
Оценка качества подготовки обучающихся вуза
Бондаренко Б.А.
Белгородский государственный национальный
исследовательский университет, ул. Студенческая, 14, Белгород, 308007, Россия Kucheryavenko_s@bsu.edu.ru*
Статья поступила 27 марта 2020; принята 09 июня 2020; опубликована 30 июня 2020
Аннотация. Ориентир на качество образования является приоритетной задачей государственной программы РФ «Развитие образования» (утвержденной постановлением правительства РФ от 26 декабря 2017 года №1642). Среди основных мероприятий направления «Совершенствование управления системой образования» этой программы имеют место мероприятия по реализации механизмов оценки и обеспечения качества образования в соответствии с государственными образовательными стандартами. Оценка качества образования в вузах определяется необходимостью компетентностной ориентации как самого образовательного процесса, так и его содержания, технологий реализации, переориентации оценочных процедур, технологий и средств оценки качества подготовки обучающихся в рамках компетентностно-ориентированных требований образовательных стандартов. В статье рассматривается проблема оценки качества подготовки обучающихся высших учебных заведений. Дается определение понятия качества образования с точки зрения заинтересованных сторон. Представлена методика оценки сформированно-сти результатов обучения, построенная на двухкритериальном подходе (компе-тентностном и дисциплинарном). Выделены основные критерии оценки сформиро-ванности результатов обучения. Рассмотрены результаты апробации методики по одному из направлений подготовки. Для исследования уровня сформированности компетенции предлагается шкала оценки сформированности компетенции (этапа сформированности компетенций). Результаты апробации методики оценки сформированных результатов обучения, представленные в статье, могут быть полезны для заинтересованных лиц, принимающих непосредственное участие в процессе проектирования, разработки и реализации образовательных программ, а также в ходе принятия методических и управленческих решений, направленных на повышение качества образования.
Ключевые слова: качество образования; оценка качества подготовки обучающихся; уровень сформированности компетенций; компетентностный подход; оценка остаточных знаний; система внутренней независимой оценки качества образования; система менеджмента качества; образовательная организация. Информация для цитирования: Кучерявенко С.А., Стенюшкина С.Г., Назарова А.Н., Бондаренко Б.А. Оценка качества подготовки обучающихся вуза // Научный результат. Педагогика и психология образования. 2020. Т.6. №2. С. 26-33. Б01: 10.18413/2313-8971 -2020-6-2-0-3.
Introduction. Currently, the Ministry of Science and Higher Education of the Russian Federation pays special attention to the independent assessment of the quality of education. Obvious on this basis is the expediency of forming a system of independent assessment of the quality of education in higher education institutions of the Russian Federation. The basis for the development is the need for a comprehensive independent assessment of educational activities and training of students, expressed in the degree of their compliance with The Federal State Educational Standards, the requirements of specialized organizations and enterprises, as well as the requirements and needs of an individual or legal entity in whose interests educational activities are carried out, including the degree of achievement of the planned results of the educational program.
One of the components of an internal independent assessment of the quality of training of students is the implementation of measures to control the availability of students ' formed learning results in previously studied disciplines (modules)1.
In this article, the quality of education is considered as a complex characteristic of educational activities and training of students, expressing the degree of their compliance with The Federal State Educational Standards, The Federal State Requirements and (or) the needs of an individual or legal entity in whose interests educational activities are carried out, including the degree of achievement of the planned results of the educational program.
The procedure for evaluating the formation of learning outcomes in higher education institutions should be based on the following legislative and legal documents:
- Federal law "On education in the Russian Federation" dated 29.12.2012 No. 273-FL (ed. from 06.04.2020) (article 95 "Independent assessment of the quality of education");
Guidelines for conducting an independent assessment of the quality of educational activities of organizations engaged in educational activities: letter of the Ministry of Education and Science of the Russian Federation dated 03.04.2015 no. AP-512/02. (In Russian)
- Federal State Educational Standards of higher education;
- Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG);
- Methodological recommendations for the organization and conduct of internal independent assessment of the quality of education in higher education institutions for higher education programs - bachelor's programs, specialty programs and master's programs (letter of the Ministry of Education and Science of the Russian Federation dated 15.02.2018 № 05-436 "On methodological recommendations");
- University regulations: Regulations on the system of independent assessment of the quality of education, Regulations on the main educational program of higher education, Regulations on the main professional educational program of higher education, Regulations on the Fund of evaluation funds, Regulations on the point-rating system for evaluating the quality of development of basic educational programs, Regulations on the electronic educational and methodological complex of the discipline for the e-learning system, etc.
Main Part. Research analysis. The problem of the quality of education has been the subject of research of many scientists for a long period of time, and yet, a common opinion on the criteria for evaluating the quality of education has not yet been formed among researchers (Tjurikov A.G., Borovinskih O.S., Golubeva K.A., Kunizheva D.A., 2018; Zhuravleva M.M., Nakonechnyh V.N., 2019; Ledovskaja T.V., Solynin N.Je., 2019; Razov P.V., Amiraslanova A.N., 2018; Abdallah K.M., 2019).
Analysis of research results of Russian scientists (V.P. Bespalko, V.P. Simonov, N.V. Pakharenko, I.N. Zolnikov, M.M. Zhuravleva, V.N. Nakonechnyh, T.V. Ledovskaja, N.Je. Solynin, etc.) indicate the presence of different approaches to the evaluation of completeness of learning outcomes, lack of common view on the assessment of competence, lack of scientific and methodological basis of the study of the level of formation of competences, differences in understanding of requirements of The Feder-
al State Educational Standards of Higher Education in terms of evaluation of competences and the insufficient development of technologies for the creation of assessment materials (Aguado, ML., 2018; Goos, M., Salomons, A., 2017;Karin J. Gerritsen-van Leeuwenkamp, Desiree Joosten-ten Brinke, Liesbeth Kester, 2017; Wang, Dongfang & Sun, Yuting & Jiang, Ting, 2018).
Analysis of foreign studies on the methodology of evaluating learning outcomes (B. Bloom, D. Bock, J. Guilford, S. Thorpe, J. Clifford) indicates the possibility of using multi-level taxonomies (Gerritsen-van Leeuwenkamp, Karin & Brinke, Desiree & Kester, Liesbeth, 2019; );Cechova, Ivana & Neubauer, Jin & Sedlacik, Marek, 2019;Deng, S, Que, X., 2019).
A common understanding of quality assurance in all countries and among all stakeholders is provided by The Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG - The European Standards and Guidelines) developed by the European Network for Quality Assurance on direct instructions of the Ministers Conference of European countries education that signed The Bologna Declaration. Russia also signed The Bologna Declaration, which means that, along with other European countries, it has committed itself to building a unified educational space (Flores, M.A., Brown, G., Pereira, D. et al., 2020).
In this regard, these standards and recommendations should be used in higher education institutions as a single reference guide for the formation of systems of external and internal quality assurance in higher education.
Methodology for evaluating the formation of learning outcomes. Assessment of control the presence of students formed learning outcomes of previously studied disciplines (modules) in the part of formation of competences is carried out in the framework of formation in the BSU system of internal evaluation of the quality of education (the basis of methodical recommendations of the Ministry of Science and Higher Education of the Russian Federation).
The purpose of assessing the quality of training of students is to achieve a positive dynamics in the quality of training of students by forming an objective assessment of the results of the development of educational programs implemented by an educational organization. The main principles of internal independent quality assessment should include: objectivity, collegiality, independence, and frequency. These principles determine the following requirements for the preparation and implementation of measures to control the availability of students' formed learning results in previously studied disciplines (modules):
- a two-criteria approach (competence-based and interdisciplinary);
- conducting the assessment procedure no earlier than 6 months after the completion of the discipline;
- the form of conducting-computer testing, based on evaluation materials for disciplines that form a certain competence;
- when forming the test, the classical model2 is used; the test is formed from 30 tasks, each of which corresponds to the chosen competence for the study; the test is given 45 minutes to complete;
- the quality of evaluation materials is confirmed by an expert opinion (review) of a representative of the relevant organization or enterprise;
- disciplines, that form the competence are selected for research at random, taking into account the above requirements. Formation of competence depending on the complexity and duration of skills development may include several stages, if the competence has not yet been formed, the assessment is made at the stage of competence formation;
2 The classical model is characterized by the following criteria. There are n jobs in a particular field of knowledge, several knowledge areas or parts of area of expertise (section, topic, etc.). From this set of tasks is randomly chosen k jobs (k<n) which are available to the learner. The trainee selects or enters the correct answer, in their opinion. The result of the response to each task is evaluated as "correct" or "incorrect". The result of testing is the percentage of correct responses of the trainee. The rating depends only on the number of correct answers and does not take into account the complexity of tasks.
- the procedure is organized by an independent Department responsible for managing the quality of educational activities at the University;
- to form objective results, in terms of the completeness of coverage of the elements of the studied population, continuous observation is used (actual participation of at least 80 % of students in the studied field of training /
specialty, taking into account the valid reasons for absence);
- representatives of student self-government bodies, representatives of specialized enterprises and organizations are invited as independent observers of the course of computer testing.
To study the level of competence formation, it is proposed to use the following assessment scale (table 1).
Table 1
Scale of assessment of the formation of learning outcomes
Percentage of correctly completed tasks Characteristics of competence formation level The level of competence formation
(90 - 100) Analytical level. The student has intuitive knowledge, the ability to demonstrate them in nonstandard ways, to determine the properties, belonging and relationship of the studied objects, their advantages and disadvantages. Advanced
(70 - 90) System level. The student has clear and systematic knowledge and ideas, is able to successfully and systematically apply knowledge, skills, and abilities. Profound
(50 - 70) Reproductive level. The student has an idea of the content of the studied object, has the ability to reproduce their knowledge orally, in writing or in demonstrated actions on a sample, according to instructions. Threshold
(0 - 50) The level of orientation, spatial representations. Reflection of either a complete lack of knowledge, skills, or the presence of the student's fragmentary knowledge of the main educational material Low
The threshold value (the minimum acceptable percentage) when evaluating whether students have formed learning outcomes in previously studied disciplines (modules) in terms of competence formation is defined as 50 %.
Results of the method testing. The
summary results of the diagnostic study of the formation of learning outcomes in the framework of determining the level of competence formation based on the proposed methodology
are reflected in the analytical table 2, presented below.
This fragment of the study presents the results of assessing the level of formation of General Cultural Competence-2 (further GCC-2) in the field of training 38.03.01 Economics. 14 students of the 3rd year of the training direction 38.03.01 Economics took part in the diagnostic testing. The results of the study sample are checked for reliability, validity, representativeness, and reliability requirements and can be applied to the entire general population.
Table 2
Assessment results of the level of GCC-2 formation in the field of training 38.03.01 Economics
№ The level of competence formation Indicator of the level of competence formation, (%) Number of students (respondents), (pers.) Percentage of respondents, (%)
1 Advanced (90 - 100) 11 78,6
2 Profound (70 - 90) 3 21,4
3 Threshold (50 - 70) 0 0,0
4 Low (0 - 50) 0 0,0
The obtained results of diagnostic testing allow to conclude that most of the students who took part in the testing have an advanced level of competence formation. The percentage of students who showed the level of competence formation below the threshold value (0-50) was 0 %, which demonstrates the formation of competence in students of the training direction 38.03.01 Economics.
Conclusions. Testing of the method allows to make a number of generalizing conclusions and recommendations. Assessment of the level of formation of learning outcomes makes it possible to determine the level of implementation of educational programs by the University. Assessment of the trajectory of competence formation allows to conclude that the main principle of phasing in the formation of competencies (increasing complexity from basic knowledge to advanced level) is observed. The analysis of the structure and content of educational programs in terms of curricula in the field of training/ specialties and the matrix of competence formation in the disciplines of the curriculum allows to determine the areas of possible improvements.
The matrix of competence formation is a key element of the main professional educational program, which allows to clearly reflect the formation of competencies within the disciplines of the curriculum. Based on it, lining up the passport of competencies, competence maps disciplines and evaluation funds, as well as the evaluation of competence and as a consequence the quality and the level of development of the basic professional educational program. During the design trajectory of competence formation should note: the basic approach
of the pedagogical design in the formation of educational outcomes (the result is a tool - the content); developed evaluation funds (evaluation materials) in the discipline should be structured (to correlate with the theme of sections of discipline code controlled competencies and indicators). This will help to more correctly prepare tasks for assessing the formation of competencies.
An important condition that precedes the procedure for assessing the formation of competencies is the quality control of test tasks in the e-learning system used. However, during testing, it is advisable to record the facts of students' complaints about the presence of errors/typos in the proposed tasks: technical errors (incorrect display of test tasks, drawings, formulas, etc.); incorrect wording of questions, suggesting ambiguous answers; the absence of specific rules for filling in answers in the open form before the task; the absence of all possible variants of correct answers in questions such as "Short answer" / "Missing word". As part of the determination of opportunities to improve assessment tools created by University teachers and placed in the e-learning system, it is recommended to cross-check test tasks by teachers at the Department level, form a database of test tasks from tasks of different types, varying degrees of complexity, use competence-oriented tasks (cases, creative tasks, project tasks).
In terms of strengthening the interaction of the Institute with specialized enterprises and organizations on improving the educational process for the formation of the most objective assessment of training results, it is recommended to conduct an examination of evaluation tools with the involvement of representatives of
specialized enterprises and organizations. The form of confirmation of passing the examination can be the receipt of a review.
To increase the motivation of students to successfully master educational programs, it is recommended to use the developed evaluation materials as part of the current and intermediate control, as an element of the point-rating system, as well as self-control of students during the semester.
Testing of the methodology for assessing the formation of competence based on a disciplinary approach has shown that it can be used to assess the formation of learning results in determining the level of competence formation. Nevertheless, it is necessary to move from the classical model of forming evaluation materials used in the research to the use of more advanced models, for example, to adaptive testing. Since the testing revealed the shortcomings of the classical model of forming the final test, namely, due to a random selection of questions, it is impossible to determine in advance what tasks of complexity will be formed for the student. When using the classic model, a combination of easy tasks for one student and difficult tasks for another student is possible. Therefore, the result depends only on the number of correct answers and does not take into account the complexity of tasks, which has low reliability, since the lack of consideration of task parameters often makes it impossible to objectively assess the student's knowledge.
At the moment, the proposed method for evaluating the formation of learning outcomes has been finalized by the areas of improvement identified during the study and is at the stage of re-testing.
References
Abdallah K M. (2019), "Actual problems of evaluating the quality of education in higher education", Young scientist, №17, 203-204. (In Russian)
Aguado, M.L. (2018), "Quality assessment of university degrees. Difficulties perceived from the quality assurance systems", Education XX1, 21, 263-284.
Cechova, Ivana, Neubauer, Jin and Sedlacik, Marek, (2019), "Tracking the University Student Success: Statistical Quality Assessment", Journal
on Efficiency and Responsibility in Education and Science, 12, 12-25.
Deng, S. and Que, X. (2019), "Research on the teaching assessment of students of science and engineering teachers in a university", Comput. Appl. Eng. Educ. 27: 5-12, available at: https://doi.org/10.1002/cae.22051 (Accessed 28 March 2020).
Flores, M.A., Brown, G., Pereira, D. et al. (2020), "Portuguese university students' conceptions of assessment: taking responsibility for achievement", High Educ. 79, 377-394, available at: https://doi.org/ 10.1007/s10734-019-00415-2 (Accessed 26 March 2020).
Gerritsen-van Leeuwenkamp, Karin & Brinke, Desiree & Kester, Liesbeth. (2019), "Students' perceptions of assessment quality related to their learning approaches and learning outcomes", Studies in Educational Evaluation, 63, 72-82.
Goos, M. and Salomons, A. (2017), "Measuring teaching quality in higher education: assessing selection bias in course evaluations". Res. High Educ., 58, 341-364, available at: https://doi.org/10.1007/s11162-016-9429-8 (Accessed 28 March 2020).
Guidelines for conducting an independent assessment of the quality of educational activities of organizations engaged in educational activities: letter of the Ministry of Education and Science of the Russian Federation dated 03.04.2015 no. AP-512/02. (In Russian).
Gerritsen-van Leeuwenkamp, K. J., Joostenten Brinke, D., and Kester, L. (2019), "Taking Stock of Initiatives to Improve Learning Quality in American Higher Education Through Assessment", Higher Education Policy, 32, 577-595.
Gerritsen-van Leeuwenkamp, K. J., Joostenten Brinke, D., and Kester, L. (2017), "Assessment quality in tertiary education: An integrative literature review", Studies in Educational Evaluation, 55, 94-116.
Ledovskaja T V. and Solynin N.Je. (2019), "Main approaches to assessing the results of students' development of basic educational programs", Yaroslavl pedagogical Bulletin, №1 (106), 49-55. (In Russian)
Liu, Shuiyun (2016), "Quality Assessment and University Change in China", Higher Education in Asia: Quality, Excellence and Governance, available at:https://doi.org/10.1007/978-981-10-0789-7_7 (Accessed 26 March 2020)
Razov P.V. and Amiraslanova A.N. (2018), "Assessment of the quality and demand for educa-
tional services of higher educational institutions, taking into account the opinion of consumers", Vlast, №9, 162-167. (In Russian)
Tjurikov A.G., Borovinskih O.S., Golubeva K.A. and Kunizheva D.A. (2018), "Formation of a system of internal and external independent assessment of the quality of educational services at the University", Vlast, №9, 147-152.(In Russian)
Wang, Dongfang & Sun, Yuting & Jiang, Ting. (2018), "The Assessment of Higher Education Quality from the Perspective of Students through a Case Study Analysis", Frontiers of Education in China. 13. 267-287.
Zhuravleva M.M., Nakonechnyh V.N. (2019), "Scientific and methodological approaches to the formation of professional competence of students in higher education", Pedagogical image, № 2 (43), 227-239. (In Russian)
Информация о конфликте интересов: автор не имеет конфликта интересов для декларации. Conflicts of Interest: the author has no conflict of interests to declare.
Данные авторов:
Кучерявенко Светлана Алексеевна, директор центра менеджмента качества; доцент, доцент кафедры инновационной экономики и финансов; кандидат экономических наук, Белгородский государственный национальный исследовательский университет. ORCID: 0000-00023642-2948
Стенюшкина Светлана Геннадьевна, специалист по менеджменту качества центра менеджмента качества, кандидат социологических наук,
Белгородский государственный национальный исследовательский университет. ORCID: 0000-GGG1-8322-6424.
Назарова Анастасия Николаевна, начальник отдела системы менеджмента качества центра менеджмента качества; старший преподаватель кафедры инновационной экономики и финансов, Белгородский государственный национальный исследовательский университет. ORCID: GGGG-GGG1-7578-5461.
Бондаренко Байрамкиз Агабалайевна,
начальник отдела качества образования центра менеджмента качества, Белгородский государственный национальный исследовательский университет. ORCID: GGGG-GGG2-9683-4725.
About the authors:
Svetlana Al. Kucheryavenko, Director of the Center for Quality Management, Associate Professor, Associate Professor of the Department of Innovative Economics and Finance, Candidate of Economic Sciences, Belgorod State National Research Uni-versity.ORCID:GGGG-GGG2-3642-2948. Svetlana G. Stenyushkina, Specialist in Quality Management, Center for Quality Management, Candidate of Sociological Sciences, Belgorod State National Research University. ORCID: GGGG-GGG1-8322-6424.
Anastasia N. Nazarova, Head of the Department of Quality Management System, Centerfor Quality Management, Senior Lecturer of the Department of Innovative Economics and Finance, Belgorod State National Research University. ORCID: GGGG-GGG1-7578-5461.
Bayramkiz A. Bondarenko, Head of the Department of Education Quality, Center for Quality Management, Belgorod State National Research University. ORCID: GGGG-GGG2-9683-4725.
