CC BY
64
ИНФОРМАЦИОННЫЕ ТЕХНОЛОГИИ И ОБРАЗОВАНИЕ X
udc 004.032.2 Articles
doi:10.31799/1684-8853-2020-5-50-61
System for monitoring and adjusting the learning process of primary schoolchildren based on the EEG data analysis
S. A. Kurkina, Dr. Sc., Phys.-Math., Professor, orcid.org/0000-0002-3438-5717, [email protected]
V. V. Grubova, PhD, Phys.-Math., Senior Researcher, orcid.org/0000-0003-2491-2592
V. A. Maksimenkoa, PhD, Phys.-Math., Senior Researcher, orcid.org/0000-0002-4632-6896
E. N. Pitsika, Junior Researcher, orcid.org/0000-0003-1850-2394
M. V. Khramovab, PhD, Ped., Associate Professor, orcid.org/0000-0002-6392-4580
A. E. Hramova, Dr. Sc., Phys.-Math., Professor, orcid.org/0000-0003-2787-2530
alnnopolis University, 1, Universitetskaya St., 420500, Innopolis, Russian Federation
bSaratov State University named after N. G. Chernyshevsky, 83, Astrakhanskaya St., 410012, Saratov,
Russian Federation
Introduction: Monitoring the learning process usually involves an analysis of the higher mental functions of the student: imagination, memory, thinking, attention, etc. Currently, there are wide opportunities for objective monitoring and evaluation of these characteristics based on the achievements of modern neuroscience. Analysis of the personal characteristics of the student allows to perform personalization of educational process. ln this context, the approaches based on the electroencephalography analysis are particularly promising. Currently, one of the main barriers to the widespread introduction of EEG based systems in the school educational process is the lack of ready-made solutions to such systems for use in the learning process of schoolchildren, as well as scenarios for their application. Purpose: Development and initial testing of the system for monitoring and adjusting the learning process, as well as strategies for its use for primary school students. Methods: EEG analysis, wavelet analysis of recorded multichannel EEG data, analysis of changes in the energies of the main rhythms of the neural activity of the brain, the feedback provided by the system. Results: The concept and diagram of the developed system for monitoring and adjusting the learning process and the strategy for its use for elementary school students are described in detail. An increase in the efficiency of learning process with developed system was demonstrated as a consequence of feedback based on the assessment of changes in the energies of the characteristic rhythms of the student's brain activity. Practical relevance: The use of EEG systems for an objective analysis of the personal characteristics of students and the personalization of the learning process through the implementation of feedback based on the data obtained will increase the quality of the educational process and the efficiency of learning new material.
Keywords — personalized learning, EEG, elementary school students, brain-computer interface, learning process control, electroencephalography, EEG signal analysis, brain activity, neural activity.
For citation: Kurkin S. A., Grubov V. V., Maksimenko V. A., Pitsik E. N., Khramova M. V., Hramov A. E. System for monitoring and adjusting the learning process of primary schoolchildren based on the EEG data analysis. Informatsionno-upravliaiushchie sistemy [Information and Control Systems], 2020, no. 5, pp. 50-61. doi:10.31799/1684-8853-2020-5-50-61
Introduction
The control of the learning process on the all levels of education has always been an demanded topic. The control has been considered in the context of the evaluation the learning process efficiency by teacher as well as by the students. The pedagogical control implies correction of the learning process based on the analysis of the data obtained by the teacher. The procedure of adjustment itself can be carried out at each step of the learning process, but it has the most important value at the completion of the learning new material. Thus, the pedagogical control is a certain way to assess information about the quality of the learning process and the readiness of the student.
Traditionally, the Russian and foreign pedagogy of XX century was focused on the quality of the learning, skills and attitudes. Various forms and methods of control were able to not only to verify
learning, skills and attitudes, but also change the system of teaching in the process of learning, which allows to individualize and differentialize it [1-5].
Introduction of competency approach in the school education highlighted certain dissatisfaction with the traditional methods of learning process control. The search for the new methods and techniques was initiated for verifying not only traditional learning, skills and attitudes, but also personal and mental features of the student.
Since the works of L. S. Vygotskiy, the educational psychology implied the analysis of the development of the higher mental functions, such as imagination, memory, thinking, attention, etc. [2, 6] Russian and foreign teachers and psychologists tried to consider corresponding neuropsychological knowledge in the process of the learning and control [5, 7]. Recently, the great opportunities for objective control and evaluation of such features were discovered based on the achievements of modern
neuroscience. In [8, 9] the relationships between academic progress, cognitive abilities and features were studied. The modern techniques of neuroim-aging (EEG, MEG, fNIRS, fMRI) have broad potential to consider the development of memory and attention of preschoolers [10], as well as to visualize the cognitive processes while solving the spatial reasoning mathematical tasks [11], arithmetic tasks [12], or the tasks that require auditory processing during reading and spelling [13].
Introduction of the digital learning devices and remote educational technologies, as well as transition of the learning in the digital environment, highlighted the issue of relevance of the learning process quality control. On the one hand, the analytics of a large amount of data obtained in the process of mass training and, on its basis, the personalization of the learning process has become in demand [14]. On the other hand, personalization became possible based on the analysis of the cognitive characteristics of the students [15]. EEG based research on the analysis of brain activity seems to be especially promising. First of all, this is due to the fact that EEG, being a relatively inexpensive, affordable, easy-to-use and safe technology, allows obtaining objective information about the brain's work with good time resolution [16]. Moreover, when using EEG, preliminary analysis and optimization usually allows to significantly reduce the number of electrodes, as well as the duration of recordings of EEG signals when solving a specific problem [17]. For example, in [18], it was shown that the state of the subject upon perception of an ambiguous visual stimulus can be identified with high accuracy using signals recorded with only two EEG channels.
Electroencephalogram studies have shown [19] that children with lower math skills need more careful monitoring and processing of resources. In [20], it was revealed how the type and complexity of the task affect the information processing mechanism and how the approaches to solving linguistic and visual-spatial tasks differ in individuals of different intelligence levels. However, such works are rather in the nature of scientific research and are not intended for the direct introduction of EEG into the learning process. The 2018 survey mentions only 22 examples of publications on this topic [16].
At the same time, it became clear that the use of EEG for an objective analysis of the cognitive characteristics of students and personalization of the learning process through the implementation of feedback based on the data obtained will signif icant-ly improve the quality of the educational process and the efficiency of learning new material [16]. It is expected that the proposed approach will give the best results when used on primary schoolchildren. This is due to the higher lability of children and the presence of simpler scenarios for using such a sys-
tem on the relatively simple and easily systematized educational material of primary school.
Currently, there are two main barriers that prevent the widespread introduction of EEG systems into the school educational process. Firstly, the technical difficulties of using EEG, which include the relatively long process of mounting electrodes on the head and the need to use special gel and paste. Secondly, the lack of ready-made solutions for systems based on EEG to use in the learning process of schoolchildren, as well as scenarios for their use. However, the development of compact and relatively inexpensive electroencephalographs and "dry" electrodes already makes it possible to overcome the first barrier [16].
In present paper, we took a step towards overcoming the second barrier by describing the concept and scheme of the developed system for monitoring and adjusting the learning process and the strategy of its use for primary schoolchildren. The results of the first tests of the implemented system are also given.
General description of the system for monitoring and adjusting the learning process
Functional diagram and main blocks
A diagram of the proposed prototype (briefly — a learning control system, LCS) is shown in Fig. 1. It includes three main blocks.
1. Electroencephalograph for recording the electrical activity of the student's brain in the process of solving cognitive tasks.
2. A tablet with a developed electronic educational environment (EEE). EEE provides an interface for the student to interact with the LCS using a tablet, solving the proposed tasks, studying the proposed educational material, or perceiving other accompanying information.
3. Control computer (personal computer or laptop) with a software module for reading and primary processing of data coming from the electroen-cephalograph and a software module of the control system that processes all incoming information, controls the effectiveness of the learning process and develops a strategy for its correction.
There are three main streams of information organized in LCS.
1. The EEG data are transmitted to the control computer (to the software module for reading EEG data and then to the software module of the control system) via wireless communication channel (for example, Bluetooth).
2. The results of the students' testing are transmitted from the tablet with EEE to the control computer (to the software module of the control sys-
■ Fig. 1. Functional diagram of the proposed system
Monitoring of the current state:
a) psychoemotional state
b) psychophysiological state
c) cognitive fatique
Brain activity recording
Tksk
1 i, ¿a -^Г/Tl
л
ù Task difficulty adjustment
Users' answers recording
■ Fig. 2. Qualitative illustration of the developed learning control system
tem). The results can be both the correctness and speed of the solution, as well as the reaction time, the number of errors and any other behavioral specifications; a wireless communication channel is used for data transmission.
3. Feedback in the form of control commands transmitted via wireless channel from the software module of the control system to the tablet with EEE. Feedback is necessary to adjust the tasks and educational material (their level of difficulty, type, rest time and other parameters) offered to the student through the EEE. The control system determines the direction of changing the task parameters based on the analysis of incoming information (see items 1 and 2) using original algorithms. Moreover, control or adjustment can be carried out by an operator (teacher, psychologist) in manual
mode based on information displayed in real time on the monitor of the control computer.
Figure 2 shows an illustration of the developed system for monitoring and adjusting the learning process of primary schoolchildren based on the analysis of EEG data, which schematically demonstrates the variant of its use.
The principle of functioning of the developed system
Let's consider the principle of operation of the proposed LCS system on a simple example of its use during one student control session. In this case, the student is invited to pass a set (battery) of tests in the EEE on a tablet, consisting of mandatory and variable parts. The obligatory part includes tests for assessing the student's cognitive resource, in
particular, Schulte tables, memory test, proofreading test, etc. These tests allow assessing the student's current cognitive resource, as well as his psycho-emotional and psychophysiological state and the level of cognitive fatigue. The variable part of the tests includes subject-oriented test assignments in the frames of the taught discipline. Tests can be both input and intermediate, or final verification tests for the chosen topic.
A fundamental feature of the developed system is the feedback (see Figs. 1 and 2), based on continuous monitoring and analysis of the student's brain activity and adjusting the complexity of test tasks based on the aggregate information about the correctness of tasks and the characteristics of brain activity. The latter allows to objectively and continuously assess such components of the cognitive resource as the level of attention, cognitive fatigue, and working memory. Thus, the following main scenarios of feedback have been implemented:
— The student successfully copes with the current test task (the success criterion is set by the teacher) — the feedback commands EEE to increase the complexity of the task. For example, the student copes well with the implementation of the Schulte table consisting of 25 digits, then the feedback will lead a Schulte table of 36 digits, on the next step of testing.
— The student does not cope with the current test task, the analysis of EEG data shows a high level of cognitive fatigue — the student is given the rest necessary to reduce cognitive fatigue, for example, an educational game or cartoon is offered. The test then continues.
— The student does not cope with the current test task, the analysis of EEG data shows a low level of attention, while cognitive fatigue is at a normal level — the student is given comments that motivate him to perform the test more attentively.
— The student does not cope with the current test task, while the analysis of the EEG data shows that cognitive fatigue and the degree of attention are at normal levels. This means that the student is trying to solve the task proposed to him, effectively using his cognitive resource. However, the complexity of the task is too high for him. As a result, the feedback gives the EEE command to reduce the level of task complexity.
The described adaptive adjustment of the complexity of the test tasks based on the analysis of the student's EEG data, allows to determine the maximum task complexity for each student, which is determined by his cognitive resource and abilities. This adjustment is carried out both for mandatory and optional tests. In the first case, feedback allows to assess the most correct and objective state of the cognitive resource of the student, as well as his psychoemotional and psychophysiological state.
In the second case, it is possible to determine the current level of the student on the topics covered by the tests.
Relationships between EEG and cognitive resources
As mentioned in the previous subsection, the feedback in LCS based on a continuous assessment of the components of a student's cognitive resource, which are attention level, cognitive fatigue, working memory load, using recorded EEG in the process of passing test tasks. It is well known that M/EEG signals reflect the activity of the brain in different frequency ranges [17, 21, 22], mainly 8-rhythm (1-4 Hz), 9-rhythm (4-8 Hz), a-rhythm (8-12 Hz), p-rhythm (13-30 Hz) and y-rhythm (> 30 Hz). The ratio of given frequency ranges reflects various cognitive processes in the human brain [23-26]. To assess the cognitive resource (in particular, indicators of attention and cognitive fatigue), the most important is the ratio of the energies of the a- and p-rhythms [27-32]. For example, an increase in the level of attention and concentration on solving a problem in most cases is accompanied by a significant decrease in the energy of the a-rhythm with a simultaneous increase in activity in the p-range [29, 31, 33]. Note that it is not the absolute values of the energies of different rhythms that matter, but how they change relatively to a certain basic level (for example, the corresponding energies at rest), or their dynamics in the process of solving problems. Thus, when using EEG signals to assess the cognitive characteristics of a student, a necessary step is to calibrate the system for a particular student, which is carried out during resting state [34]. Further, the system monitors changes in various rhythms of brain activity in the process of performing test tasks by students, as well as during rest between tasks. To assess the energies of various rhythms in EEG signals, wavelet analysis is used [35].
The methodological basis of the algorithms that establish the relationship between the levels of the main components of the student's cognitive resource and changes in rhythm energies in EEG signals is described in more detail in [31, 36, 37] and presented in registered computer programs [38, 39].
The strategy of using the developed system to control and adjust the learning process
The developed system in the educational process monitors and adjusts the learning process at certain points in time.
Suppose that some educational course consists of N topics (Fig. 3), and the level of complexity of the taught material within each of the topics may vary. For simplicity, let's assume that there are only
Difficulty
То
Q-
-9
г,
-О-
Т1
■f---—---
То
Topic 1
Topic 2
1 2N-2
! 2N-1
О
-о
Topic N
г 2N
-О
Time
Т
4
■ Fig. 3. A qualitative illustration of the principle of application of the developed system for monitoring and adjusting the learning process: Tl symbols and circles indicate reference points in time at which testing is performed using the learning control system, while white circles correspond to entrance tests, and gray circles correspond to intermediate ones; a thick line with arrows shows the "educational path" in terms of changes in the complexity of the taught material over time, while the arrows indicate the direction of the change in complexity at reference points in time
three difficulty levels (the dashed horizontal line in Fig. 3 corresponds to the average level), and each topic is two weeks long. Then, a possible effective scenario for using LCS would be testing each student using this system twice during one topic: conducting an entrance test before the beginning of a new topic (white circles in Fig. 3) and an intermediate test in the middle (in this case, a week after the start topics; gray circles in Fig. 3).
As mentioned, entrance testing with the help of LCS allows to objectively determine the current level of a student according to the topics covered by the tests, which makes it possible to choose the most suitable level of complexity of the material planned for teaching. During intermediate testing (in this case, after a week), the difficulty level is adjusted based on the test results. Thus, an "educational trajectory" personalized for a particular student is built in the chosen discipline (see the thick line with arrows in Fig. 3). The described adaptive adjustment of the complexity of the material taught to the student in accordance with its current level is aimed at increasing the efficiency of the educational process and the success of mastering the necessary material.
Let's take a closer look at the structure of EEE tests. The variable part of the tests should consist of several blocks (usually from three to five), which allows to change the difficulty level from block to block in accordance with the feedback commands and, thus, to implement adaptive adjustment. A set of related tasks can act as a block. Also, the block structure turns out to be convenient for the mandatory part of the test. In this case, for example, one Schulte table or a proof test can act as a block. Moreover, the cognitive characteristics of the student, as well as the success of his solution of tasks, is assessed throughout the block, and not when solving individual tasks. Such averaged characteristics
are more objective and less susceptible to random fluctuations.
Note that when completing tasks from the first block of the variable part, the difficulty level is selected based on the assessment of the components of the cognitive resource based on the test results for the mandatory part and the difficulty level determined in the previous testing session, if the current test is intermediate. The result of testing for the variable part is an assessment of the level of complexity of the material that will be most appropriate for a given student at a given period of time, taking into account his current training and abilities. Thus, the level of complexity of the taught material is set in accordance with the test results in the period from the next lesson to the next reevaluation, adjusting the level of complexity using the LCS.
For the system to work successfully, it is necessary to foresee in advance the possibility of changing the level of complexity of the taught material for different students, for example, by combining students into different subgroups according to the level of difficulty. Also, blocks of tasks of various difficulty levels must be included in the base of LCS tests.
Note that the number of difficulty levels can be more than three. Also, instead of difficulty, another scale can act, for example, the type of material taught, the strategy of material presentation, etc. Moreover, the scale can consist of several parameters, i. e. be multidimensional.
Algorithm of functioning of the developed system for monitoring and adjusting the learning process
Let us consider in more detail the algorithm underlying the functioning of the developed system for monitoring and adjusting the learning process of a primary school student. Algorithm consists of
a)
b)
Input
Interactive block with variable test
Testing cycle — one test block
EEG settings -►
Task difficulty, defined using the results of the
obligatory tests -►
EEG analysis Evaluation of effectivenes of task performance based on EEG data
Transition to the next test block
Positive rate — increase the rask difficulty and go to the next block
Output
Negative rate
Reducing difficulty or taking actions to restore the cognitive resource
■ Fig. 4. The algorithm for the functioning of the learning control system: a — the preparatory part related to the calibration of the system and passing the required tests; b — the main part associated with the passage of tasks from the variable tests
two main parts, preparatory and main, shown in Fig. 4, a and b, respectively. The first step is to calibrate the system, which consists in assessing the base levels of the basic rhythms of neural activity using EEG data under rest condition. This calibration must be done for every new user; then the calibration information is saved in users' profile. To improve the accuracy of the system, it is necessary to periodically recalibrate it (for example, once in a month) for already registered users. Based on the calibration information, the parameters of the time-frequency analysis of EEG signals are also adjusted.
The next step starts the obligatory part of the test, which is implemented using feedback based on the analysis of changes in the energies of the rhythms of neural activity according to the recorded EEG data. As a result of the test,
the levels of the components of the student's cognitive resource are determined, and on the basis of these data, the initial level of difficulty and other settings of the tasks of the variable part of the test. Then control is transferred to the block with the variable part of the test (see Fig. 4, b). Here, the student is asked to solve a number of test tasks, grouped into separate blocks, while the complexity of the tasks varies from block to block in accordance with the principle of feedback implementation (for more details, see subsection "The principle of functioning of the developed system"). As a result, the system generates a summary of the test results, the level of the components of the student's cognitive resource and recommendations on the optimal parameters (for example, the level of difficulty) of the material planned for teaching on the test topic.
Description and testing of the implemented system to control and adjust the learning process
System was implemented with electroencephalo-graph "ActiCHamp" (Brain Products, Germany), a tablet with a stylus and a laptop as a control computer. EEG signals were recorded from 31 channels using active electrodes "ActiCap" (Brain Products) based on high-quality Ag/AgCl-sensors, while the electrodes were placed using a special cap in accordance with the "10-10" scheme with a ground electrode on the position "Fpz" and one reference electrode in the area of the right mastoid. An abrasive "NuPrep" gel was used to increase the conductivity of the scalp, and the "SuperVisc" conductive gel was applied to the EEG electrodes.
Electroencephalogram recordings were sampled at 250 Hz and filtered with a bandpass filter from 0.016 to 70 Hz and a notch filter at 50 Hz. An illustration of the learning process using developed LCS is shown on Fig. 5.
To interact with the student in the process of passing the tests, an EEE was developed in an interactive form familiar to the child, in which a special fictional character conducts communication with the child (Fig. 6). The type and character of this character were developed in cooperation with psychologists in terms of the best communication with a primary school child.
Working process with EEE consists of three stages:
■ Fig. 5. The learning process with the developed system
1) the procedure for calibrating and configuring the system (Fig. 6, a);
2) the obligatory part of the test (in Fig. 6, b and c, the task with the Schulte table is shown as an example);
3) the variable part of the test. In the developed EEE it consists of logic tasks (an example is shown in Fig. 6, d).
1 7 25 11 16
21 23 9 22 4
12 IS 3 10 14
5 24 19 20 6
13 17 g IS 2
d)
Cheburashka wefit to school. The leacher put him a( (he secorKl desk, if you ccujiit fjom the front, arit the fourth, if you counl trom the ::!::k How many des'is are there in a raw?
■ Fig. 6. Screenshots of the developed e-learning environment: a — the stage of calibration and tuning of the system; b and c — an example task with a Schulte table from the mandatory part of the test; d — an example of a logical problem from the variable part of the test
The system gives out test results, an assessment of the cognitive resource and individual recommendations for teaching material for the student who passed the tests.
We now consider its functioning when a student completes an assignment with a Schulte table, which is a simplified version of the Zahlen-VerbindungTest [40]. The completion of this task allows to determine the effectiveness of the student's work and his ability to work, as well as resistance to external distractions. By default, the Schulte table is a 5 x 5 matrix with randomly located numbers from 1 to 25. However, its size can be increased or decreased by the system as the student passes the test. By default, the student should select numbers from 25 to 1 in the table in descending order by clicking on them on the tablet. The system registers time intervals Atm between two consecutive clicks on adjacent numbers. For example, At1 is the time interval between the moment the task starts and the moment you press the digit "25", At2 is between pressing the non-digit "25" and "24", etc.
The participants in the experiment were 10 schoolchildren aged 7 to 10 years (grades 1-3 of the school). Each student completed N = 5 Schulte tables, the execution of 1 table took from 50 to 90 s, we denote the execution time as t1, t2, ..., At^ Between the tables there was a short break of 10-20 s. As a result, the following individual cognitive characteristics of schoolchildren can be determined:
1. Efficiency of work, which is the arithmetic mean of the execution times of all tables: WE = (t1 + + t2 + ... + tN / N. It characterizes consistency of attention and productivity.
2. The indicator of working capacity, which is the ratio of the time of execution of the first table to the efficiency of work: WU = tl / WE. WU < 1 indicates good working capacity, otherwise, it indicates that the student needs a longer preparation for the main job.
3. Psychological stability (a person's ability to maintain work activity for a long time) PS = = tN - 1 / WE. If PS < 1, then this indicates good psychological stability.
Let's consider the main steps that the developed LCS takes, and the results and conclusions obtained in the process of executing the Schulte table. First, the system analyzes the dependence of the time required for a student to search for each subsequent number on its number: Atm(m). An example of such a dependence is shown in Fig. 7. Its analysis allows to draw conclusions about the efficiency and speed of execution of a given table.
Second, according to EEG data, the dependences of the energy values of a- and p-rhythms on the number m, averaged over the corresponding intervals Atm: Ea p(m), are estimated for each of the 31 channels. Analysis of these dependencies allows
Atm, S
7 6 5 4 3 2 1
25 m
■ Fig. 7. The time Atm required for the student to search for each subsequent number on his number m
you to determine the level of concentration on solving the problem when choosing each of the numbers. Thus, an increase in the p-rhythm with a simultaneous decrease in the a-rhythm relative to the activity of the brain in a calm state, recorded during the calibration process, indicates an increase in the level of attention and concentration on solving the problem [41].
However, more objective assessments are given by the values of the energies of the corresponding rhythms averaged over one table (in the general case, one test block). Fig. 8 shows the values of the efficiency of each of the five Schulte tables, which is the inverse to the time spent on its execution (effn = 1/tn), and the corresponding ratios of the p-and a-rhythm energies (Ep/Ea) in the temporal brain area, averaged over period of execution of each of the tables. The task consisted of five 5 x 5 Schulte tables in total. Fig. 8, a corresponds to the typical case when LCS is used in the process of completing a task. One can see that for the second table, the effectiveness of solving the assigned task decreases (solid line). This is accompanied by a decrease in the p- and the a-rhythm energies ratio (dashed line), which indicates a decrease in the level of attention and concentration. LCS reveals this feature and, in accordance with the described above algorithm, gives the student a feedback, motivating him to perform the task more attentively. This leads to an increase in the efficiency of the task, starting from the fourth Schulte table. As a result, the efficiency of the last table exceeds the efficiency of the first one.
On the contrary, Fig. 8, b corresponds to a situation when LCS is not used in the process of a schoolchild's fulfillment of an assignment. In this case, starting from the fourth table, the efficiency of task performance is significantly reduced, and for the last table it becomes much lower than for the
0
a)
Ep/E
eff
2.2 2.1 2 1.9 1.8
1.8
1.6
1.4
1.2
b)
Ep/E
eff
1.6
1.5
1.4
1.3
■ Fig. 8. Efficiency of each of the five Schulte tables (solid line) and the corresponding values Ep/E„ in the temporal region, averaged over the period of execution of each of the tables (dashed line): a — the characteristic case when the student uses the learning control system; b — the case without the use of the learning control system (for another subject)
first one. This is due to a decrease in the level of attention and concentration of the student on solving the task (see the dependence Ep/Ea in Fig. 8, b).
Comparing the Fig. 8, a and b, we demonstrate that in case with Schulte tables the LCS increased the efficiency and the stability of the results of the student's performance due to the presence of feedback.
Additionally, the analysis demonstrated that the tendency towards a high p/a-rhythms ratio is most pronounced in the occipital and temporal areas of the brain. Excitation of high-frequency p-activity and a decrease in the a-rhythm in the occipital region indicates an increased load on the visual cortex, while a similar trend in the temporal cortex indicates the activity of the brain associated with learning and processing new information [42, 43].
In conclusion, note that in order to reliably confirm the effectiveness of the proposed system and demonstrate the statistical significance of the differences obtained, it is necessary to conduct its versatile testing on a group of schoolchildren, including various types of tasks and scenarios. The results of such testing will be presented in our subsequent works.
Conclusion
We described in detail the concept and scheme of the developed system for monitoring and adjusting the learning process, as well as the strategy its' application for primary schoolchildren. The functional diagram and the main blocks of the system, the principle and algorithms of its functioning are described. The relationship of the characteristics of EEG signals with the components of the cognitive resource and the implementation of this concept in the LCS system are discussed. The strategy of using the developed system for monitoring and
adjusting the learning process in the educational process is described. A description of a specific implemented system along with an electronic educational environment is given. The results of the first tests of the implemented system are presented when schoolchildren perform a simple task consisting of N Schulte tables. The results of comparing the effectiveness of solving the proposed tasks with and without active LCS system demonstrated using a specific example of the increase in the efficiency of a schoolchild's work when using LCS system. This is a consequence of the influence of feedback and personalization of the testing process, based on the assessment of changes in the energies of the characteristic rhythms of the student's brain activity.
Financial support
This work was supported by the RFBR (grant No. 19-29-14101). Work on the development of algorithms for time-frequency analysis was supported by the grant of the President of the Russian Federation (MD-1921.2020.9).
Reference
1. Basham J. D., Hall T. E., Carter Jr. R. A., Stahl W. M.
An operationalized understanding of personalized learning. Journal of Special Education Technology, 2016, vol. 31, no. 3, pp. 126-136. doi: 10.1177/ 0162643416660835
2. Bernacki M. L., Walkington C. The role of situational interest in personalized learning. Journal of Educational Psychology, 2018, vol. 110, no. 6, pp. 864. doi:10.1037/edu0000250
3. Kovaleva ^ M. New didactic resources for modern school. Science and school, 2015, no. 1, pp. 88-94.
4. Pardo A., et al. OnTask: Delivering data-informed, personalized learning support actions. Journal of Learning Analytics, 2018, vol. 5, no. 3, pp. 235-249. doi: 10.18608/jla.2018.53.15.
5. Bulger M. Personalized learning: The conversations we're not having. Data and Society, 2016, vol. 22, no. 1, pp. 1-29.
6. Vygotsky L. S. Myshlenie i rech' [Thinking and Speech]. Moscow, Labirint Publ., 1999. 352 p. (In Russian).
7. Stepanov V. G. Nejropedagogika. Mozg i effektivnoe razvitie detej i vzroslyh: vozrast, obuchenie, tvorchestv, proforientaciya [Neuropedagogy. The brain and the effective development of children and adults: age, education, creativity, career guidance]. Moscow, Akademich-eskij proekt Publ., 2020. 345 p. (In Russian).
8. Demetriou A., Kazi S., Spanoudis G., Makris N. Predicting school performance from cognitive ability, self-representation, and personality from primary school to senior high school. Intelligence, 2019, vol. 76, September-October, 101381. doi:10.1016/J. Intell.2019.101381
9. Demetriou A., Makris N., Taxmatzidis D., Kazi S., Spanoudis G. Decomposing the influence of mental processes on academic performance. Intelligence, 2019, vol. 77, November-December, 101404. doi:10. 1016/j.intell.2019.101404
10. Anjolii Diaz, Tashauna L. Blankenship, Martha Ann Bell. Episodic memory in middle childhood: Age, brain electrical activity, and self-reported attention. Cognitive Development, 2018, vol. 47, July-September, pp. 63-70. doi:10.1016/j.cogdev.2018.03.003
11. Murat Perit £akir, Hasan Ayaz, Meltem Izzetoglu, Patricia A. Shewokis, Kurtulu? Izzetoglu, Banu Onaral. Bridging brain and educational sciences: An optical brain imaging study of visuospatial reasoning. Procedia — Social and Behavioral Sciences, 2011, vol. 29, pp. 300-309. doi:org/10.1016/j.sb-spro.2011.11.243
12. Anna A. Matejko, Daniel Ansari. The neural association between arithmetic and basic numerical processing depends on arithmetic problem size and not chronological age. Developmental Cognitive Neuroscience, 2019, vol. 37, June, 100653. doi:10.1016/j.dcn.2019. 100653
13. Debskaa A., Chyla K., Dzi^giela G., Kacprzakab A., Luniewskaa M., Plewkoa J., Marchewkac A., Grabowskad A., Jednoroga K. Reading and spelling skills are differentially related to phonological processing: Behavioral and fMRI study. Developmental Cognitive Neuroscience, 2019, vol. 39, October, 100683. doi:10.1016/j.dcn.2019.100683
14. Mamedova G. A., Zeynalova L. A., Melikova R. T. Big data technologies in e-learning. Open Education, 2017, no. 6, pp. 41-48 (In Russian). doi:10.21686/1818-4243-2017-6-41-48
15. Yawei Cheng, Chen Yi Chen, Jean Decety. An EEG/ ERP investigation of the development of empathy in
early and middle childhood. Developmental Cognitive Neuroscience, 2014, vol. 10, October, pp. 160-169. doi:10.1016/j.dcn.2014.08.012
16. Jiahui Xu, Baichang Zhong. Review on portable EEG technology in educational research. Computers in Human Behavior, 2018, vol. 81, April, pp. 340-349. doi:10.1016/j.chb.2017.12.037
17. Maksimenko V. A., Kurkin S. A., Pitsik E. N., Musa-tov V. Y., Runnova A. E., Efremova T. Y., Hramov A. E., Pisarchik A. N. Artificial neural network classification of motor-related EEG: An increase in classification accuracy by reducing signal complexity. Complexity, 2018, vol. 2018, 9385947. doi:10.1155/2018/ 9385947
18. Hramov A. E., Maksimenko V. A., Koronovskii A., Runnova A. E., Zhuravlev M., Pisarchik A. N., Kurths J. Percept-related EEG classification using machine learning approach and features of functional brain connectivity. Chaos, 2019, vol. 29, 093110. doi:10.1063/1.5113844
19. Gonzalez-Garrido A., Gomez-Velazquez F. R., Sali-do-Ruiz R. A., Espinoza-Valdez A., Velez-Perez H., Romo-Vazquez R., Gallardo-Moreno G. B., Ruiz-Stov-el V. D., Martinez-Ramos A., Berumen G. The analysis of EEG coherence reflects middle childhood differences in mathematical achievement. Brain and Cognition, 2018, vol. 124, July, pp. 57-63. doi:10. 1016/j.bandc.2018.04.006
20. Jun-Su Kang, Amitash Ojha, Giyoung Lee, Minho Lee. Difference in brain activation patterns of individuals with high and low intelligence in linguistic and visuo-spatial tasks: An EEG study. Intelligence, 2017, vol. 61, March-April, pp. 47-55. doi:10.1016/j. intell.2017.01.002
21. Fries P. Rhythms for cognition: communication through coherence. Neuron, 2015, vol. 88, no. 1, pp. 220-235. doi:10.1016/j.neuron.2015.09.034
22. Chholak P., Niso G., Maksimenko V. A., Kur-kin S. A., Frolov N. S., Pitsik E. N., Hramov E. A., Pisarchik A. N. Visual and kinesthetic modes affect motor imagery classification in untrained subjects. Scientific Reports 9, 2019, 9838. doi:10.1038/s41598-019-46310-9
23. Marrufo V., Vaquero E., Cardoso M. J., Gomez C. M.
Temporal evolution of a and p bands during visual spatial attention. Cognitive Brain Research, 2001, vol. 12, no. 2. pp. 315-320. doi:10.1016/s0926-6410(01)00025-8
24. Clayton M. S., Yeung N., Kadosh R. C. The roles of cortical oscillations in sustained attention. Trends in Cognitive Sciences, 2015, vol. 19, no. 4, pp. 188-195. doi:10.1016/j.tics.2015.02.004
25. Khorev V. S., Maksimenko V. A., Pitsik E. N., Runnova A. E., Kurkin S. A., Hramov A. E. Analysis of motor activity using electromyogram signals. Infor-matsionno-upravliaiushchie sistemy [Information and Control Systems], 2019, no. 3, pp. 114-120 (In Russian). doi:10.31799/1684-8853-2019-3-114-120
26. Kurkin S. A., Pitsik E. N., Hramov A. E. Classifying electrical activity of the brain during imaginary movements of untrained subjects using artificial neural networks. Informatsionno-upravliaiushchie sistemy [Information and Control Systems], 2019, no. 6, pp. 77-84 (In Russian). doi:10.31799/1684-8853-2019-6-77-84
27. Carp J., Compton R. J. Alpha power is influenced by performance errors. Psychophysiology, 2009, vol. 46, no. 2, pp. 336-343. doi:10,1111/j.1469-986.2008. 00773.x
28. Mazaheri A., Nieuwenhuis I. L., van Dijk H., Jensen O. Prestimulus alpha and mu activity predicts failure to inhibit motor responses. Human Brain Mapping, 2009, vol. 30, no. 6. pp. 1791-1800. doi:10.1002/hbm. 20763
29. Klimesch W. Alpha-band oscillations, attention, and controlled access to stored information. Trends in Cognitive Sciences, 2012, vol. 16, no. 12, pp. 606-617. https://doi.org/10.1016/j.tics.2012.10.007
30. Novikov N. A., Bryzgalov D. V., Chernyshev B. V. Theta and alpha band modulations reflect error-related adjustments in the auditory condensation task. Frontiers in Human Neuroscience, 2015, vol. 9, p. 673. doi:10.3389/fnhum.2015.00673
31. Maksimenko V. A., Runnova A. E., Zhuravlev M. O., Makarov V. V., Nedayvozov V., Grubov V. V., Pche-lintceva S. V., Hramov A. E., Pisarchik A. N. Visual perception affected by motivation and alertness controlled by a noninvasive brain-computer interface. PloS One, 2017, vol. 12, no. 12. https://doi.org/10. 1371/journal.pone.0188700
32. Stam C. J. Brain dynamics in theta and alpha frequency bands and working memory performance in humans. Neuroscience Letters, 2000, vol. 286, no. 2, pp. 115-118. doi:10.1016 / s0304-3940 (00) 01109-5
33. Sacchet M. D., LaPlante R. A., Wan Q., Pritchett D. L., Lee A. K., Hamalainen M., Moore C. I., Kerr C. E., Jones S. R. Attention drives synchronization of alpha and beta rhythms between right inferior frontal and primary sensory neocortex. Journal of Neuroscience, 2015, vol. 35, no. 5, pp. 2074-2082. doi:10.1523/ JNEUR0SCI.1292-14.2015
34. Roslan N. S., et al. Review of EEG and ERP studies of extraversion personality for baseline and cognitive tasks. Personality and Individual Differences, 2017, vol. 119, pp. 323-332. doi:10.1371/journal.pone. 0219839
35. Hramov A. E., Koronovskii A. A., Makarov V. A., Pavlov A. N., Sitnikova E. Wavelets in Neuroscience.
Springer Berlin Heidelberg, 2015. 318 p. https://doi. org/10.1007/978-3-662-43850-3
36. Maksimenko V. A., Runnova A. E., Kulanin R. A., Protasov P. A., Zhuravlev M. O., Chholak P., Pisarchik A. N., Hramov A. E. Algorithm for automatic estimation of human brain activity features during mental task evaluation. Informatsionno-upravliai-ushchie sistemy [Information and Control Systems], 2018, no. 5, pp. 104-111. doi:10.31799/1684-8853-2018-5-104-111
37. Maksimenko V. A., Runnova A. E., Zhuravlev M. O., Protasov P., Kulanin R., Khramova M. V., Pisar-chik A. N., Hramov A. E. Human personality reflects spatio-temporal and time-frequency EEG structure. PloS One, 2018, vol. 13, no. 9. https://doi.org/10.1371/ journal.pone.0197642
38. Badarin A. A., Runnova A. E., Karavaev A. S., Zhuravlev M. P., Astahov S. V., Hramov A. E. Ustro-jstvo dlya opredeleniya v rezhime real'nogo vremeni stepeni koncentracii vnimaniya operatora pri vospri-yatii i obrabotke informacii. [The device for real-time detection of operators' concentration level during the perception and processing of information]. Patent for invention № 2704562, 2019.
39. Maksimenko V. A., Nedaivozov V. O., Hramov A. E. Ocenka koncentracii vnimaniya cheloveka po dannym EEG-izmerenij. [Evaluation of human attention using EEG]. Certification of official registration of computer program № 2019661003, 2019.
40. Vernon P. A. Der Zahlen-Verbindungs-Test and other trail-making correlates of general intelligence. Personality and Individual Differences, 1993, vol. 14, no. 1, pp. 35-40.
41. Grubov V. V., Maksimenko V. A., Kurkin S. A., Khramova M. V., Hramov A. E. Maintaining attention state of children during cognitive load. Proc. SPIE 11459, Saratov Fall Meeting 2019: Computations and Data Analysis: from Nanoscale Tools to Brain Functions, 114590A. doi:10.1117/12.2564349.
42. Papanicolaou A. C., Loring D. W., Deutsch G., Eisenberg H. M. Task-related EEG asymmetries: a comparison of alpha blocking and beta enhancement. International Journal of Neuroscience, 1986, vol. 30, no. 1-2, pp. 81-85. doi:10.3109/00207458608985658
43. Morozova L. V., Zvyagina N. V., Terebova N. N. Characteristics of visual perception in seven-year-old children differing in functional maturity of brain structures. Human Physiology, 2008, vol. 34, no. 1, pp. 14-21. doi:10.1134/S0362119708010027
УДК 004.032.2
doi:10.31799/1684-8853-2020-5-50-61
Система для контроля и корректировки процесса обучения младших школьников на базе анализа данных ЭЭГ
С. А. Куркина, доктор физ.-мат. наук, профессор, orcid.org/0000-0002-3438-5717, [email protected]
В. В. Грубова, канд. физ.-мат. наук, старший научный сотрудник, orcid.org/0000-0003-2491-2592
В. А. Максименкоа, канд. физ.-мат. наук, старший научный сотрудник, orcid.org/0000-0002-4632-6896
Е. Н. Пицика, младший научный сотрудник, orcid.org/0000-0003-1850-2394
М. В. Храмова6, канд. пед. наук, доцент, orcid.org/0000-0002-6392-4580
А. Е. Храмова, доктор физ.-мат. наук, профессор, orcid.org/0000-0003-2787-2530
аУниверситет Иннополис, Университетская ул., 1, Иннополис, 420500, РФ
6Саратовский национальный исследовательский государственный университет им. Н. Г. Чернышевского, Астраханская ул., 83, Саратов, 410012, РФ
Введение: контроль процесса обучения обычно подразумевает анализ развития высших психических функций обучающегося: воображения, памяти, мышления, внимания и др. В настоящее время открываются широкие возможности объективного контроля и оценки данных характеристик благодаря достижениям современной нейронауки. Персонализация становится возможной на основе анализа личностных характеристик обучаемых. Особенно перспективными представляются подходы с использованием электроэнцефалографии. Одним из главных барьеров, препятствующих широкому внедрению ЭЭГ-систем в школьный образовательный процесс, является отсутствие готовых решений таких систем для применения в ходе обучения школьников, а также сценариев их приложения. Цель: разработка и тестирование системы для контроля и корректировки процесса обучения, а также стратегии ее использования для школьников младших классов. Методы: ЭЭГ-анализ, вейвлет-анализ регистрируемых многоканальных ЭЭГ-данных, анализ изменений энергий характерных ритмов нейронной активности головного мозга, а также реализуемая системой обратная связь. Результаты: детально описана концепция и схема разработанной системы для контроля и корректировки процесса обучения, стратегия ее использования для школьников младших классов. Продемонстрировано существенное повышение эффективности работы школьников при применении разработанной системы, что является следствием влияния обратной связи и персонализации процесса тестирования и обучения, базирующихся на оценке изменения энергий характерных ритмов мозговой активности школьника. Практическая значимость: использование ЭЭГ-систем для объективного анализа личностных характеристик обучаемых и персонализации процесса обучения путем реализации обратной связи на основании получаемых данных существенно повысит в перспективе качество образовательного процесса и эффективность усвоения нового материала.
Ключевые слова — персонализированное обучение, ЭЭГ, младшие школьники, интерфейс мозг-компьютер, контроль процесса обучения, электроэнцефалография, анализ сигналов ЭЭГ, мозговая активность, нейронная активность.
Для цитирования: Kurkin S. A., Grubov V. V., Maksimenko V. A., Pitsik E. N., Khramova M. V., Hramov A. E. System for monitoring and adjusting the learning process of primary schoolchildren based on the EEG data analysis. Информационно-управляющие системы, 2020, № 5, с. 50-61. doi:10.31799/1684-8853-2020-5-50-61
For citation: Kurkin S. A., Grubov V. V., Maksimenko V. A., Pitsik E. N., Khramova M. V., Hramov A. E. System for monitoring and adjusting the learning process of primary schoolchildren based on the EEG data analysis. Informatsionno-upravliaiushchie sistemy [Information and Control Systems], 2020, no. 5, pp. 50-61. doi:10.31799/1684-8853-2020-5-50-61
УВАЖАЕМЫЕ АВТОРЫ!
Научная электронная библиотека (НЭБ) продолжает работу по реализации проекта SCIENCE INDEX. После того как Вы зарегистрируетесь на сайте НЭБ (http://elibrary.ru/ defaultx.asp), будет создана Ваша личная страничка, содержание которой составят не только Ваши персональные данные, но и перечень всех Ваших печатных трудов, имеющихся в базе данных НЭБ, включая диссертации, патенты и тезисы к конференциям, а также сравнительные индексы цитирования: РИНЦ (Российский индекс научного цитирования), h (индекс Хирша) от Web of Science и h от Scopus. После создания базового варианта Вашей персональной страницы Вы получите код доступа, который позволит Вам редактировать информацию, помогая создавать максимально объективную картину Вашей научной активности и цитирования Ваших трудов.
