CC BY
41
DOI: 10.14529/ped200108
EXPERT ASSESSMENT METHOD IN TRAINING LECTURERS FOR WORK IN INTERNATIONAL GROUPS OF STUDENTS
E.V. Evsina, elenachel@mail.ru
South Ural State University, Chelyabinsk, Russian Federation
Nowadays more and more universities recruit international students due to the process of internationalization in higher education. The author determined the main problems that lecturers who start teaching in international groups of students may come across. Expert assessment method was used. Working in international groups for at least several years, experts are able to share the most valuable data on international students' behavior and to anticipate some conflict situations that may occur. The article gives a brief literature review of the expert assessment method to prove that this method is well developed and widely used in different spheres of research. The results of the expert survey show that the main problems for the lecturers are the non-sufficient level of English as the language of instruction and the basic rules of intercultural communication. The conclusion was made that the lecturers who are going to teach in the international groups of students should have the level of English B2 and higher, should be certified in EMI, and should know the basics of intercultural communication.
Keywords: qualitative research, expert assessment method, competence and consistency oj experts, ascertaining experiment.
Introduction
At present, almost all types of professional activity are somehow related to the examination or assessment. Doctors and teachers, managers and engineers, lawyers and economists and other specialists, to one degree or another, act as experts and/ or those who are being assessed or examined. As a result of the examination (assessment), experts determine the winners of contests and competitions, create ratings of organizations (enterprises, banks), politicians, businessmen and athletes, approve indices (Dow Jones and other.) etc. The system of Higher Education is not an exception.
Speaking about examination/ assessment in education, it should be mentioned that examination is the word of Latin origin, meaning research, resolution with the help of knowledgeable, experienced people in any field of science and technology that requires special knowledge. In turn, an expert means an experienced person invited in controversial or difficult cases for examination or assessment [20]. In pedagogy, the scope of scientific and pedagogical expertise is very extensive: competent assessment of new textbooks and manuals, expert assessment of the achievement test, expert evaluation of projects and creative works, etc.
The number of expert assessments today is increasing, since this method helps in solving
complex scientific, technical and socio-economic problems in conditions where there is no information about solving similar problems in the past.
Subject of the study
The subject of our study is the process of training lecturers who are going to work with international groups of students. In our article we try to identify problems associated with foreign students teaching that may be faced by lecturers, who are just starting to work in international groups of students, using the expert assessment method. Teaching in international group of students in the situation of global changes in the system of Higher Education is rather new due to the fact that Russia has signed the Bologna Declaration in 2003. And since then the programs as well the methods of teaching have changed drastically. Although Russia has a very rich experience in teaching foreign students during the years of the Soviet Union era, the "imperatives of internationalization" demand absolutely new approach to the process of education in international groups of students at our universities.
Methods
We apply the method of expert assessments as it is rather popular and deeply studied in modern science. To prove it, we would like to mention briefly what has been done in this area. Nowadays, there is not only a great number of scientists, like L.G. Evlanov [9], A.A. Frenkel [11],
E.A. Shashenkova [14] etc., who contributed articles and books on the methods of expert assessments, but also the authors [10, p. 14], who provide an overview of what has been done in this area. Such a review is presented in the works of G.G. Azgaldov [1], as well as in G.G. Azgaldov and E.P. Raikhman's work [2], D.S. Schmerling [18], V.S. Cherepanov [6] and others. As the first steps in the development of expert assessment methods, one can consider the work of a group of American psychologists who began to study the influence of the expert's interest on the nature of his/her judgment on future socio-political events in 1930. Studies have shown that the level of expert's special knowledge has the greatest impact on the reliability of his/her statements [3, p. 8].
It can be stated that at present the statistical and mathematical foundations of this method are substantiated in sufficient detail [5], as well as its role and place in the pedagogical expertise [12, 13] etc. Fundamental research in this area was conducted by V.S. Cherepanov [6].
In addition, there has been a tendency abroad not to explore the expert method itself, but the expert systems, such as Artificial Intelligence.
A few years later the study of expert systems done by R. Benfer, E. Brenta, L. Furbi (R.A. Ben-fer, E.E. Brent, L. Furbee, 1991) becomes fundamental [4]. In their book, the authors have presented a detailed model for developing expert systems. This model involves the acquisition, storage, presentation of knowledge, as well as the assumption that this knowledge is limited in diagnosing and predicting of human behavior.
G. Strube, D. Janetzko, M. Knauff (1996) investigated the structure of generalized expert knowledge [19]. This study also outlined a transition from the analysis of the knowledge of individual experts to the analysis of their generalized opinion, which is permeated by psychological laws. The authors proposed a technique for collecting knowledge of experts.
Methods of expert assessments are continuously developed and improved.
Expert methods, including the method of group expert assessments have been increasingly used in pedagogical research in recent years [15, 17]. The method of group expert assessments is fundamental in pedagogical qualimetry -the science of measuring and assessing the quality of pedagogical objects and processes [2]. We have chosen this method for conducting our ascertaining experiment (at the beginning of our study).
Expert assessment is a procedure for obtaining an assessment of a problem based on the opinion of specialists-experts in order to come to a certain decision / conclusion [7, p. 183-187]. The method of expert assessment allows to get the most reliable results and conduct an analysis on the problem we are studying: the problem that lecturers (who are just beginning to teach in international groups of students) may come across. Experts are the competent colleagues who have been working in international groups of students and can point out such problems.
From the standpoint of pedagogical quali-metry, the validity of the expert assessment method application is confirmed by the fact that the methodologically correctly obtained expert judgments satisfy the criteria of accuracy and reproducibility of the examination result [2]. Moreover, a technology for the transition from individual expert to collective assessments has been developed within the framework of this method.
The competence of an expert in the field of pedagogical research was determined by such requirements to him/ her as: awareness, objectivity, skill level, creativity, etc. The more experts' qualities are taken into account, the less is the variance of their individual assessments. And this (small variance), in turn, improves the quality of pedagogical expertise. All of the experts were lecturers from the South Ural State University (SUSU) with the sufficient experience of working in groups with foreign students.
Discussion
The selection of experts with the necessary qualities was carried out using the following methods: mutual recommendations, self-assessment, assessment of argumentation, personal data, and assessment of consistency [16].
The method of mutual recommendations is similar to the method of voting. Each expert had to give information about his/her place of employment, position, academic degree and rank, teaching experience and other personal data. The experts were also asked to answer the question: 'Who would you choose as an expert from the proposed list?" The coefficient of mutual recommendations was determined on the basis of the questionnaire. The more votes were given for an expert, the greater was his coefficient of mutual recommendations [3].
The method of self-assessment is based on the subjective assessment of an expert candidate himself/ herself as a specialist in the pedagogical
Table 1
Method of self-assessment of candidates for experts (Ques = question; Ex = expert)
N Ques Ex1 EX2 Ex3 Ex4 Ex5 Ex6 Ex7 Exs
1 5 5 3 5 3 5 5 5
2 3 3 2 3 2 3 3 3
3 3 3 3 3 3 3 3 3
4 3 3 3 3 3 3 3 3
problem under consideration. Candidates for experts needed to rank the indicators of the questionnaire and assess themselves on the following proposed scale (table 1).
1. I have been working in groups with international students: for 1-3-5 years.
2. I understand the problem of intercultural interaction between a lecturer and foreign students: 1-2-3 point-scale; (1 - is minimum; 3 -is maximum).
3. I am familiar with the problem of appli-cants'(who arrived from different countries) different levels of knowledge 1-2-3 point-scale; (1 - is minimum; 3 - is maximum).
4. I am familiar with the problem of a different attitude towards the educational process of students from various countries. 1-2-3 point-scale; (1 - is minimum; 3 - is maximum).
From the number of experts selected by us for this survey, all candidates chose maximum scores/points (2 and 3). This result allowed us to conclude that the choice of experts was correct as the main selection bias is the competency in the research field. The specialist involved as an expert should be a person who himself/herself has experienced the difficulties of problem situations in the subject area of social reality, who has judgments about the nature of these problem situations, and what, in his/her opinion, is necessary to do to find a solution of the problem.
The methods for assessing competence include collection of personal data, which takes into account the relationship between the individual documented characteristics of an expert candidate and the properties that determine his/her quality as an expert (academic degree, title, number of methodical works, etc.). All the experts who took part in the assessment have academic degrees (six lecturers holding a PhD degree) (two lecturers holding a title of a Full Professor). All the experts have scientific, methodological and educational publications on the problem under review for this study.
The main types of expert interviews are: questioning, interviewing, Delphi method, brains-
torming, and discussion. For our study, we chose a questionnaire. The main advantages of the questionnaire survey method are the ability to collect a large amount of information for a sufficiently short period and the independence of expert judgments. The disadvantages include the difficulty of developing an effective questionnaire that contains a reasonable list of questions.
The procedure for interviewing experts excluded the possibility of exchanging views between them in order to avoid mutual influence on the final decision on a particular issue. During the ascertaining (stating) experiment we used questionnaires, to which a number of requirements were presented:
- the questionnaires had to be valid. We adhere to the concept of "validity" after I.F. De-vyatko and we consider that the validity of a measurement, in the most general sense, characterizes the conformity of a measurement to its purpose. The empirical indicator is valid to the extent that it accurately reflects the value of the theoretical variable that was supposed to be measured [8]. The measurement is valid as it characterizes the conformity of our measurement to its purpose. Besides, we have conducted a pilot study among the lecturers and asked them to answer the question: 'What are main difficulties of working in international groups of students?" They all understood the question and found it correct.
- the questionnaires had to have measuring properties, which ensure the inclusion of important factors of indicators and the choice of scale for assessing the quality of the object being assessed.
There were eight experts (lecturers from the SUSU) who took part in the survey. The questionnaire consisted of two parts: Part 1 - information about the expert (work experience at a university, academic degree / title, availability of scientific, educational and methodological publications on topics under review in this study, and experience in international groups); Part 2 -questions regarding the clarification of the state of the problem in practice, i. e. questions regarding
the main problems that the lectures (who will start working in international groups of students) may come across.
In the course of administrating the survey to the experts (ascertaining/stating experiment) we obtained the following results of the state of the problem (pedagogical assistance in the development of professional competence of lecturers in groups with foreign students):
1. Differences in educational process in general.
1.1. Students enter the SUSU with different levels of knowledge obtained at the previous stage of education. (Upon admission to SUSU, foreign students do not pass entrance exams.)
1.2. Adaptation to the requirements of lecturers, living and weather conditions, and cultural traditions etc. is different and takes a long time, despite the fact that SUSU holds an annual information day for foreign students. There is also an association of foreign students and other events that help foreign students to get used to living in Russia.
1.3. Differences in behavior - the intolerance of some nations in relation to others.
2. Different behavior in the classroom (from complete "inaction" to "excessive activity"). Students' attitudes on educational process are divergent. To a greater extent, this indicator is affected by the social status, as well as the level of knowledge (+ self-organization) obtained at the previous stage of education and nationality; (age has a lesser degree of effect).
3. Lecturers who conduct classes with foreign students communicate with their students after classes as well, helping to solve organizational issues, helping to adapt, and also advising on their academic discipline. (Some of the students are embarrassed to ask questions on academic discipline during the lesson.)
After conducting the survey we processed the results. The initial information for processing was numerical data expressing the preferences of experts, and substantiation of their preferences. The purpose of the processing was to obtain generalized data and new information contained in a hidden form in expert assessments. Based on the processing results, we formed the solution to the problem.
During the ascertaining (stating) experiment, we evaluated the consistency of experts' opinions. The collected expert opinions were processed both quantitatively (numerical data) and qualitatively (meaningful information).
The main difficulties of working in international groups (II part of the expert questionnaire, where they had to choose options 1, 2 or 3; 1 -not very important; 2 - important; 3 - very important.
Questionnaire
I. Personal data:
- Age
- General work experience
- Academic degree
- Availability of publications on the issue under study
- Experience in international student groups.
II. The main difficulties of working in international groups:
N Description of the problem Points
1 Different levels of pre-university training of foreign students 1 2 3
2 The presence of an adaptation period both in studies, weather, and living conditions 1 2 3
Behavior difference during class
3 time (levels of active and passive participations) 1 2 3
Different attitude to the educa-
4 tional process (full compliance with the requirements of the lecturer - lack of interest in the educational process; partial implementation of the requirements of the lecturer) 1 2 3
5 Lecturer's knowledge of the main cultural features when communi- 1 2 3
cating with students from different countries
6 Weak foreign language proficiency on the part of the lecturer (CPD) 1 2 3
7 Your own variant 1 2 3
III. What, in your opinion, is the most important problem. Arrange the numbers of the previous questions in descending order of importance (1-7).
Table 2
Main difficulties of working in international groups (1-3)
Importance
1-3 1 2 3 4 5 6
N Question
1 0 1 0 1 0 0
2 6 5 3 6 2 2
3 2 3 5 1 6 6
From table 2 we observe that the most important issues are the ones of intercultural and language training.
Table З
Ranking of the experts' judgment importance
Imp. of the ques Exn
Exl Ex2 Ex3 Ex4 Ex5 EX6 Ex7 Ex8
l б 5 5 5 5 5 5 5
2 5 б б б б 6 6 6
3 3 3 l 3 3 l 3 l
4 l 4 3 l 4 4 l 3
5 4 l 4 4 l 3 4 4
б 2 2 2 2 2 2 2 2
For our study, questions 5 and 6 are the most important (table 3). After processing the data, we conclude that knowledge of the main cultural features and good command of a foreign language (+ EMI) are the most important aspects in preparing lecturers for working in international groups.
Results
After conducting the ascertaining (stating) experiment, we identified the main problems that the lecturers who are going to teach in international groups of students may come across. We also suggest the solution of these problems: lecturers newly embarking on work with foreign students should be able to conduct conversations on the taught course in a foreign (English) language (the level of the language cannot be lower than B2); lecturers should also know the EMI course for conducting classes in international groups, as well as to understand the intercultural peculiarities of communication with students from different countries.
References
1. Azgal'dov G.G. Teoriya i praktika otsenki kachestva tovarov. Osnovy kvalimetrii [Theory and Practice of the Goods Quality Assessment. Qualimetry Basics]. Moscow, 1982. 235 p.
2. Azgal'dov G.G., Raikhman E.P. O kvalimetrii [About Qualimetry]. Moscow, Izdatel'stvo standartov Publ., 1973. 172 p.
3. Anokhin A.N. Metody ekspertnykh otse-nok [Methods of Expert Assessments]. Obninsk, IATE Publ., 1996. 148 p.
4. Benfer R.A. Expert Systems. Columbia, 1991. 492 p.
5. Beshelev S.D., Gurevich V.G. Matema-tiko-statisticheskie metody ekspertnykh otsenok [Mathematical and Statistical Methods of Expert Assessments]. Moscow, 1980.
6. Cherepanov V.S. Ekspertnie otsenki v pe-dagogicheskikh issledovaniyakh [Expert Evaluations of Educational Research]. Moscow, Peda-gogika Publ., 1998. 152 p.
7. Danelyan T.Ya. [Formal Methods of
Expert Assessments]. Modern Problems of Science and Education, 2015, no 1, pp. 183-187. (in Russ.)
8. Devyatko I.F. Metody sotsiologicheskogo issledovaniya [Methods of Sociological Research]. Ekaterinburg, izdatel'stvo Ural'skogo universiteta Publ., 1998. 208 c.
9. Evlanov L.G., Kutuzov V.A. Ekspertnye otsenki v upravlenii [Expert Assessments in Management]. Moscow, 1978.
10. Enakaeva R.R. Psikhologicheskie re-zervy sovershenstvovaniya ekspertnykh otsenok pri provedenii akmeologicheskikh ekspertiz. Dis. kand. psikh. nauk [The Psychological Reserves of Improvement of Expert Assessments in the Conduct of Acmeological Examinations. Diss. Cand. (Psychology)]. Moscow, 1999. 175 p.
11. Frenkel' A.A. Analiz faktorov rosta proizvoditel'nosti truda s pomoshch'yu ekspert-nykh otsenok. Statisticheskie metody analiza eks-pertnykh otsenok [Analysis of Factors of Labor Productivity Growth by Means of Expert Estimations. Statistical Methods of Analysis of Expert Assessments]. Moscow, 1977. 384 p.
12. Kiselev N.I. Ekspertno-statisticheskiy metod opredeleniya funktsii predpochteniya po rezul'tatam parnykh sravneniy ob"ektov [Expert-statistical Method for Determining the Preference Function Based on the Results of Paired Comparisons of Objects]. Moscow, Nauka Publ., 1980.
13. Kitaev N.N. Gruppovye ekspertnye otsenki [A Group Expert Evaluation]. Moscow, Nauka Publ., 1975.
14. Shashenkova E.A. Issledovatel'skaya deyatel'nost' [Research Activity]. Moscow, UTs Perspektiva Publ., 2010. 88 p. (in Russ.)
15. Shikhov U.A., Shikhova O.F. [Model of Monitoring the Quality of Education in Terms of Competence Approach]. Modern Fundamental and Applied Research, 2013, no. 4 (11), pp. 35-39.
16. Shikhova O.F., Shikhov U.A. [Quality Criteria of Competence-oriented Pedagogical Control Materials]. Modern Fundamental and Applied Research, 2014, no. 1 (12), pp. 48-52.
17. Shikhova O.F., Shikhov U.A. [Qualime-tric Approach to the Diagnosis of Competencies of High School Graduates]. Education and Science, 2013, no. 4, pp. 40-56. (in Russ.)
18. Shmerling D.S., Dubrovsky S.A., Arzha-nova T.D. [Ekspert Estimates. Methods and Application]. Statistical Metods for the Analysis of Expert Assessments, 1977, pp. 290-382. (in Russ.)
19. Strube G., Janetzko D., Knauff M. Cooperative Construction of Expert Knowledge: The Case of Knowledge Engineering. Freiburg, 1996. 576 p.
20. Ushakov D.N. (Ed.) Tolkovyi Slovar'
russkogo yazyka [Explanatory Dictionary of the Russian Language]. Moscow, Izdatel'stvo Astrel' Publ., 2000. Available at: https://feb-eb.ru/feb/ ushakov/ush-abc/Oush.htm (accessed 12.07.2019)
Received 15 December 2019
УДК 378.18 DOI: 10.14529/ped200108
МЕТОД ЭКСПЕРТНЫХ ОЦЕНОК В ПОДГОТОВКЕ НПР ДЛЯ РАБОТЫ С ИНОСТРАННЫМИ СТУДЕНТАМИ
Е.В. Евсина
Южно-Уральский государственный университет, г. Челябинск, Россия
В наши дни благодаря интернационализации высшего образования все большее количество университетов обучают иностранных студентов. В статье мы попытались определить основные проблемы, с которыми сталкиваются преподаватели, только начинающие работать в интернациональных группах студентов. В нашем исследовании мы обратились к методу экспертных оценок, считая, что в данной ситуации он является наиболее валидным, так как эксперты хорошо знакомы с проблемами обучения студентов в интернациональных группах. Имея опыт работы в таких группах, эксперты способны предоставить наиболее ценную информацию о поведении студентов в интернациональных группах и помочь только начинающим работать преподавателям, предупредить вероятные конфликтные ситуации, которые могут произойти в таких группах. В статье дается краткий обзор литературы по методам экспертных оценок, чтобы показать, что данный метод довольно хорошо разработан и применяется во многих исследованиях. После опроса экспертов мы получили следующие результаты: основные проблемы, с которыми могут столкнуться преподаватели, начиная работать в интернациональных группах студентов, это: низкий уровень владения иностранным (английским) языком, как средством передачи знаний, информации, а также слабый уровень владения основными аспектами межкультурной коммуникации. Следовательно, преподаватели, собирающиеся вести занятия в группах с иностранными студентами должны владеть иностранным (английским) языком на уровне не ниже В2, овладеть курсом EMI (английский как «инструмент» передачи знаний), а также знать особенности межкультурного общения.
Ключевые слова: педагогическая квалиметрия, метод групповых экспертных оценок, компетентность и согласованность экспертов, констатирующее исследование.
Литература
1. Азгальдов, Г.Г. Теория и практика оценки качества товаров (основы квалиметрии) / Г.Г. Азгальдов. - М.: 1982. - 235 с.
2. Азгальдов, Г.Г. О квалиметрии /Г.Г. Азгальдов, Э.П. Райхман. - М., 1973. - 172 с.
3. Анохин, А.Н. Методы экспертных оценок /А.Н. Анохин. - Обнинск: ИАТЭ, 1996. - 148 с.
4. Бенфер, Р.А. Экспертные системы /Р.А. Бенфер. - Колумбия, 1991. - 492 с.
5. Бешелев, С.Д. Математико-статистические методы экспертных оценок / С.Д. Бешелев, В.Г. Гуревич. - М., 1980. - 263 с.
6. Черепанов, В. С. Экспертные оценки в педагогических исследованиях / В.С. Черепанов. -М. : Педагогика, 1989. - 152 с.
7. Данелян, Т.Я. Формальные методы экспертных оценок / Т.Я. Данелян // Современные проблемы науки и образования. - 2015. - № 1. - С. 183-187.
8. Девятко, И. Ф. Методы социологического исследования / И. Ф. Девятко. - Екатеринбург: Изд-во Урал. ун-та, 1998. - 208 с.
9. Евланов, Л.Г. Экспертные оценки в управлении /Л.Г. Евланов, В.А. Кутузов. - М., 1978. -133 с.
10. Енакаева, Р.Р. Психологические резервы совершенствования экспертных оценок при проведении акмеологических экспертиз: дис. ... канд. психол. наук /Р.Р. Енакаева. -М., 1999. - 175 с.
11. Френкель, А.А. Анализ факторов роста производительности труда с помощью экспертных оценок. Статистические методы анализа экспертных оценок / А.А. Френкель. - М., 1977. -241 с.
12. Киселев, Н.И. Экспертно-статистический метод определения функции предпочтения по результатам парных сравнений / Н.И. Киселев // Алгоритмическое и программное обеспечение прикладного статистического анализа. - М., 1980. - 422 с.
13. Китаев, Н.Н. Групповые экспертные оценки /Н.Н. Китаев. - М. : Знание, 1975.
14. Шашенкова, Е.А. Исследовательская деятельность: слов. / Е.А. Шашенкова. - М. : УЦ «Перспектива», 2010. - 88 с.
15. Шихов, Ю.А. Модель мониторинга качества образования в условиях компетентностного подхода / Ю.А. Шихов, О. Ф. Шихова // Современные фундамент. и приклад. исследования. -2013. - № 4 (11). - С. 35-39.
16. Шихова, О.Ф. Критерии качества компетентностно-ориентированных педагогических контрольных материалов / О. Ф. Шихова, Ю.А. Шихов // Современные фундамент. и приклад. исследования. - 2014. - № 1 (12). - С. 48-52.
17. Шихова, О.Ф. Квалиметрический подход к диагностике компетенций выпускников высшей школы /О.Ф. Шихова, Ю.А. Шихов //Образование и наука. - 2013. - № 4. - С. 40-56.
18. Шмерлинг, Д.С. Экспертные оценки. Методы и применение /Д.С. Шмерлинг, С.А. Дубровский, Т.Д. Аржанова // Статистические методы анализа экспертных оценок. - М., 1977. -С 290-382.
19. Штрубе, Г. Совместное построение экспертных оценок / Г. Штрубе, Д. Жанецко, М. Кнауфф. - Фрайбург, 1996. - 576 с.
20. Толковый словарь русского языка / под ред. Д.Н. Ушакова. - М.: ООО «Издательство Астрель», 2000. - http://feb-eb.ru/feb/ushakov/ush-abc/0ush.htm (дата обращения 12.07.2019).
Евсина Елена Валерьевна, аспирант кафедры безопасности жизнедеятельности, ЮжноУральский государственный университет, г. Челябинск, elenachel@mail.ru.
Поступила в редакцию 15 декабря 2019 г.
ОБРАЗЕЦ ЦИТИРОВАНИЯ
Evsina, E.V. The Expert Assessment Method in Training Lecturers for Work in International Groups of Students / E.V. Evsina // Вестник ЮУрГУ. Серия «Образование. Педагогические науки». - 2020. - Т. 12, № 1. -С. 83-89. DOI: 10.14529/ped200108
FOR CITATION
Evsina E.V. The Expert Assessment Method in Training Lecturers for Work in International Groups of Students. Bulletin of the South Ural State University. Ser. Education. Educational Sciences. 2020, vol. 12, no. 1, pp. 83-89. DOI: 10.14529/ped200108
