CC BY
8
SOURCES AND IMPACT OF HUMAN BRAIN POTENTIAL VARIABILITY IN THE BRAIN-COMPUTER INTERFACE
Ganin IP1^, Vasilyev AN1'2, Glazova TD1, Kaplan AYa1
1 Lomonosov Moscow State University, Moscow, Russia
2 Neurocognitive Research Center (MEG Center), Moscow State University of Psychology and Education, Moscow, Russia
In the brain-computer interface based on the P300 wave (P300 BCI), the selection of the command by the user becomes possible due to focusing the user's attention on the external stimulus/command and extraction of the response to this stimulus in the form of the event-related potential (ERP) components from EEG. To obtain the ERP signal, stimuli should be repeated many times, however, in view of the existing variability in latency of the response to certain stimuli, the averaged ERPs may give a distorted view of the nature of such responses and reduce accuracy of the interface. The study was aimed to develop an effective method for identification of the effects of the ERP components' latency variability and for accounting these effects in the P300 BCI, as well as to identify the possible impact of psychophysiological factors on the nature of ERP variability. We have conducted a BCI-based study of 19 healthy subjects involving extraction and adjustment of latency in the N1 and P300 spatial components, which play a key role in the command classification in the P300 BCI, to explore the mechanisms underlying variability. Such an approach ensured higher accuracy compared to the use of conventional EEG leads, and the highest increase of 10% was observed when using the minimum number of the stimulus repetitions. Furthermore, modifications of the interface allowing one to ensure a higher level of the user's focus on the task and a more accurate visual fixation on the target objects contributed to the increase in the amplitude of the ERP components by reducing variability of the responses to single stimuli. The findings emphasize the important role of the processes underlying the ERP components' variability and provide an effective tool for scientific exploration of such processes and the development of advanced BCI systems.
Keywords: brain-computer interface, BCI, electroencephalogram, EEG, event-related potentials, ERP, N1, P300, ERP variability Funding: the study was supported by the Russian Science Foundation Grant № 21-75-00021, https://rscf.ru/project/21-75-00021/
Acknowledgements: the authors would like to thank Yu. Nuzhdin (Kurchatov Institute) for developing and supporting software for EEG recording used to perform the study
Author contribution: Ganin IP — conducting research, data analysis and interpretation, literature review, manuscript writing; Vasilyev AN — data analysis and interpretation, literature review, manuscript writing; Glazova TD — conducting research, literature review; Kaplan AYa — data interpretation.
Compliance with ethical standards: the study was approved by the Ethics Committee of the Lomonosov Moscow State University (protocol № 113-d of 19 June 2020); the informed consent was submitted by all study participants.
[23 Correspondence should be addressed: Ilya P. Ganin
Leninskiye Gory, 1, str. 12, k. 246, Moscow, 119234, Russia; ipganin@mail.ru
Received: 14.04.2023 Accepted: 27.04.2023 Published online: 28.04.2023 DOI: 10.24075/brsmu.2023.013
ИСТОЧНИКИ И ЗНАЧИМОСТЬ ВАРИАТИВНОСТИ ПОТЕНЦИАЛОВ МОЗГА ЧЕЛОВЕКА В ИНТЕРФЕЙСЕ МОЗГ-КОМПЬЮТЕР
И. П. Ганин1 А. Н. Васильев1,2, Т. Д. Глазова1, А. Я. Каплан1
1 Московский государственный университет имени М. В. Ломоносова, Москва, Россия
2 Центр нейрокогнитивных исследований (МЭГ-центр), Московский государственный психолого-педагогический университет, Москва, Россия
В интерфейсе мозг-компьютер на волне P300 (ИМК-P300) выбор команд пользователя возможен за счет фокусирования им внимания на внешнем стимуле-команде и выделении из ЭЭГ реакции к этому стимулу — в виде компонентов потенциалов, связанных с событиями (ПСС). Для получения сигнала ПСС стимулы необходимо многократно повторять, однако ввиду существующей вариативности латентности реакций на отдельные стимулы усредненные ПСС могут давать искаженное представление о характере таких реакций, а также снижать точность работы интерфейса. Целью работы было разработать эффективный способ выявления эффектов вариативности латентности компонентов ПСС и учета этих эффектов в ИМК^300, и выявить возможное влияние психофизиологических факторов на характер вариативности ПСС. Для изучения механизмов вариативности мы провели ИМК-исследование на 19 здоровых испытуемых, где использовали выделение и коррекцию латентности в пространственных компонентах N1 и P300, играющих ключевую роль в классификации команд в ИМК^300. Этот подход обеспечил более высокую точность по сравнению с использованием обычных отведений ЭЭГ, при этом наибольший рост в 10% наблюдался при минимальном числе повторов стимулов. Также модификации интерфейса, позволяющие обеспечить более высокий уровень внимания пользователя к задаче и более четкую фиксацию взгляда на целевых объектах, способствовали повышению амплитуд компонентов ПСС посредством снижения вариативности реакций на единичные стимулы. Полученные результаты подчеркивают важную роль процессов вариативности компонентов ПСС и дают эффективный инструмент для их научного изучения, а также для разработки перспективных систем ИМК.
Ключевые слова: интерфейс мозг-компьютер (ИМК), электроэнцефалограмма (ЭЭГ), потенциалы, связанные с событиями, ПСС, N1, P300, вариативность ПСС
Финансирование: исследование выполнено за счет гранта Российского научного фонда № 21-75-00021, https://rscf.ru/project/21-75-00021/
Благодарности: авторы благодарят Ю. Нуждина из НИЦ «Курчатовский институт» за разработку и поддержку программного обеспечения для регистрации ЭЭГ, при помощи которого проведено исследование.
Вклад авторов: И. П. Ганин — проведение исследования, анализ и интерпретация данных, анализ литературы, подготовка текста рукописи; А. Н. Васильев — анализ и интерпретация данных, анализ литературы, подготовка текста рукописи; Т. Д. Глазова — проведение исследования, анализ литературы; А. Я. Каплан — интерпретация данных.
Соблюдение этических стандартов: исследование одобрено этическим комитетом МГУ имени М. В. Ломоносова (протокол № 113-д от 19 июня 2020 г.); все участники подписали добровольное информированное согласие на участие в исследовании.
[>3 Для корреспонденции: Илья Петрович Ганин
Ленинские горы, д. 1, стр. 12, к. 246, Москва, 119234, Россия; ipganin@mail.ru
Статья получена: 14.04.2023 Статья принята к печати: 27.04.2023 Опубликована онлайн: 28.04.2023 DOI: 10.24075/vrgmu.2023.013
Brain-computer interfaces (BCI) make it possible to directly translate brain activity into commands to control computer or any other device without involving muscles and nerves, only via analysis of the user's electroencephalogram (EEG) [1]. The concept of BCI, proposed and developed many years ago, has become an interdisciplinary technology, the primary purpose of which is supporting people with severe speech and movement disorders [2], along with the use as a tool for instrumental diagnosis or cognitive training [3-5].
BCI technologies ofter involve the use of event-related potentials (ERPs) [6]. One is the most widely used and well-proven systems is referred to as P300 BCI, since it is based on the analysis of the P300 component related to attention [7, 8]. The user of such interface usually mentally counts the number of flashes of the character or other command symbol of interest. The ERPs elicited to flashing of this (target) object are distinguished from ERPs elicited to flashing of all other (non-target) symbols by the presence of the P300 component [9]. The BCI algorithm recognizes the target symbol (command) by this feature and the presence of other components (primarily N1) in the ERP [10, 11].
The P300 BCI systems are in demand for communication: during typewriting or step-by-step control of certain device [12]. However, the main disadvantages of those include the need for repetition of stimuli aimed at accumulating ERP responses with the least error when the BCI user has to focus on the task for a long time. Furthermore, despite the assumption of similarity of the brain responses to the repeated stimuli, there is some temporal variability of certain responses relative to stimuli [13, 14]. This is a well-known neurophysiological phenomenon that generally reflects a number of natural brain processes at different levels, from cellular to the neural network level, and is also determined by fluctuation in the processes underlying perception of external stimuli [15].
It is known that such variability can affect the shape of the resulting averaged ERPs, including reducing the peak amplitude of certain components [16]. Lack of accounting of the variability effects may negatively affect the effectiveness of the P300 BCI based on the ERP extraction method, thereby reducing accuracy of the target command recognition [17, 18].
In general, changes in the ERP variability are considered to be associated with fatigue, increased cognitive load, complication of the user's task [15, 19], as well as conditions characterized by reduced attention, such as ADHD and autism [20, 21]. However, the factors affecting ERP variability in terms of P300 BCI were never systematically studied. Meanwhile, identification of the BCI operation modes having a beneficial or negative effect on ERPs and the command classification accuracy would make it possible to develop more effective systems capable of ensuring more reliable control, especially when it comes to potential users with reduced attention.
It also seems appropriate to consider ERP variability in the P300 BCI by modifying the command classification algorithms. This can be particularly important when a relatively small number of stimuli is accumulated in the interface, and the effects of variability may not be compensated by the number of averaging procedures. Given the ERP components' different contributions to classification along with variation in their topography among various users [22], extracting independent spatial components to analyze and consider their variability separately can be a more effective approach.
The study was aimed to identify possible factors of the stimulus environment and P300 BCI operation modes affecting ERP variability, as well as to develop and test more effective
methods for independent detection of variability of individual ERP components during classification.
METHODS
The study involved 19 healthy subjects (5 males and 14 females) aged 18-23. Inclusion criteria: healthy male and female volunteers aged 18-35. Exclusion criteria: diagnosed neurological/mental disorder, episodes of seizures or diagnosed status epilepticus.
During the experiment the subject sat in a chair in front of the monitor on which a standard P300 BCI matrix sized 6 x 6 with letters of Russian alphabet and numbers was presented. The angular dimensions of the matrix were 18° x 18°, the cell size was 1.7°, and the cell spacing was 1.1°. The background of the screen and cells was black (RGB 0,0,0), while cell frames and characters within the cells were grey (RGB 89,90,97). Stimuli were represented by random flashes (the background color changed from black to grey, and the color of characters changed from grey to black) of rows and columns in the matrix. The duration of stimulus and the interstimulus interval were 97 and 48.5 ms, respectively (16 and 8 frames for the refresh rate of 165 Hz). Stimulation involved using the stimulus sequences, each sequence included presentation of all 12 stimuli available in the matrix (six rows and six columns).
A separate experimental mode included 15 blocks, one cell of the matrix per block was designated as a target cell (it was marked by repeated wink at the start of the block). Five stimulus sequences per block were presented, which corresponded to 60 stimuli (10 target stimuli and 50 non-target ones). Thus, each mode included 150 target stimuli and 750 non-target ones.
Several modes distinguished by parameters of the stimulus environment and the subject's task were used to study the impact of various factors on the ERP variability. In the passive attention mode the subject was not supposed to count flashes of the target stimulus as in the P300 BCI, he/she simply fixed his/her gaze on the target cell. The task was complicated by using the mode involving mixing-up letters: the characters in all cells of the matrix randomly changed their places with each target flash. The subjects were asked to count not only all target flashes, but the number of consonants in the target cell when the character changed. To make it easier to fix gaze on the cell and reduce the effects of distractor in the modes involving the use of half-empty matrix, the characters were not made permanently visible, these appeared only with flashes (Fig. 1).
The modes and brief instructions for the subject were as follows:
1) ordinary matrix, passive attention ("just look at the target cell");
2) ordinary matrix, active attention ("count the number of flashes of the target cell");
3) half-empty matrix, active attention ("count the number of flashes of the target cell");
4) half-empty matrix, mixing up, active attention ("count the number of flashes of the target cell");
5) half-empty matrix, mixing up, cognitive load ("count the number of consonants in the target cell");
6) ordinary matrix, mixing up, active attention: ("count the number of flashes of the target cell");
7) ordinary matrix, mixing up, cognitive load ("count the number of consonants in the target cell").
All modes alternated to generate a pseudo-random sequence, except for the passive attention mode that was always the first due to special instruction.
Fig. 1. The stimulus matrix P300 BCI used in the study. The matrix was located in the center of the screen on the black background. "Ordinary matrix" is on the left, "half-empty matrix" is on the right
EEG was recorded with 30 scalp electrodes (Fp1, Fp2, F7, F3, Fz, F4, F8, FC5, FC1, FC2, FC6, T7, C3, Cz, C4, T8, CP5, CP1, CP2, CP6, P7, P3, Pz, P4, P8, PO7, POz, PO8, O1, O2) and a common reference electrode TP9 + TP10 using the NVX52 amplifier (MCS, Zelenograd; Russia). The sampling frequency was 1000 Hz. A miniature photosensor mounted in the upper left corner of the screen was used to ensure EEG synchronization with the flashes. Signal recording and management of experimental procedure were implemented in the original Resonance programming environment written in C++ (http://resonance.bcilab.net/documentation).
EEG signal processing and classification were performed in MATLAB 9.13 (R2022b) (MathWorks; USA). The EEG signal was band-pass filtered within the 1-10 Hz range using a FIR filter without a phase shift. Then ocular artifacts were removed by independent component analysis (ICA). After that the continuous signal was split into epochs from -400 to 1200 ms relative to the stimulus onset.
The next phase of analysis involved acquisition of spatial filters to extract the components of interest (N1 and P300) from the multichannel EEG signal. For that the epochs in the vicinity of individual ERP peaks were extracted in each subject, after that
0 0.2 0.4 0.6 0.8 0 0.2 0.4 0.6 0.8
Time (s)
Fig. 2. The extracted spatial components N1 and P300. The figure above shows topography of the spatial filter patterns. The figure below shows components N1 and P300 averaged across all subjects. Vertical axis — normalized amplitude in arbitrary units; horizontal axis — time (s). The vertical dotted line (0 s) corresponds to the stimulus onset. N = 19 subjects
Table 1. The average amplitudes of the N1 and P300 components In all modes when using the standard averaging method (no latency correction) and when averaging the epochs adjusted to latency of the appropriate component. The mean and standard error of the mean are provided. N = 19 subjects
Component Mode 1 Mode 2 Mode 3 Mode 4 Mode 5 Mode 6 Mode 7
Amplitude with no latency correction, AU
N1 -1.17 ± 0.11 -1.38 ± 0.08 -1.53 ± 0.09 -1.50 ± 0.08 -1.81 ± 0.09 -1.29 ± 0.09 -1.67 ± 0.11
P300 0.92 ± 0.07 1 ± 0.06 0.94 ± 0.07 0.86 ± 0.06 1.07 ± 0.08 0.89 ± 0.06 1.05 ± 0.05
Amplitude with latency correction, AU
N1 -1.50 ± 0.09 -1.67 ± 0.08 -1.81 ± 0.07 -1.77 ± 0.07 -2.05 ± 0.09 -1.59 ± 0.07 -1.93 ± 0.09
P300 1.47 ± 0.06 1.59 ± 0.04 1.52 ± 0.06 1.46 ± 0.04 1.62 ± 0.07 1.53 ± 0.04 1.67 ± 0.04
optimal spatial projections (spatial filters) were calculated based on the Fisher's criterion [23]. Such method made it possible to reduce the EEG signal dimension, increase the signal-to-noise ratios of the studied components, and largely isolate two components from each other for independent study [23]. The further analysis was performed for these two extracted spatial components (once for N1 and once for P300). Signals of the components were normalized to the standard deviation of all non-target epochs within each subject (hereinafter, AU instead of pV).
A set of target and non-target epochs was formed within each subject, component (N1 and P300), and mode. To acquire ERPs averaged by conventional method, all the epochs of the same subject were averaged individually for each mode, for the class of the target and non-target epochs of the N1 and P300 sets. The amplitude of these components was defined as the minimum/maximum signal value within the 100-350 and 200-500 ms windows, respectively, and the peak latencies were defined as the time after the stimulus onset when the signal reached its maximum or minimum.
Furthermore, to analyze the ERP variability, the N1 and P300 component latencies were calculated within certain non-averaged target epochs as local minima or maxima in the same windows as for ERP. The component's amplitude was determined by the signal values for the latencies found within this epoch. To assess variability of the ERP peak latencies, the mean absolute deviation (MAD) was calculated in each mode for each subject. To estimate the effect of variability on the ERP amplitude, the epochs were shifted along the time axis by the difference between the average latency and the component latency within certain epoch prior to averaging.
To estimate the effects of ERP variability on the effectiveness of command recognition in the BCI, classification accuracy was calculated for ordinary EEG channels (standard approach) and for the extracted spatial components N1 and P300. It is important to note that classification scores of two types were calculated for the latter: without equalization of latency peaks and with equalization (correction for N1 or P300 only and correction for both peaks, N1 and P300). The signal amplitude values within the 0-600 ms window (every 10th point) in 11 channels of EEG leads Cz, CP1, CP2, P3, Pz, P4, PO7, POz, PO8, O1, O2 or two channels obtained for N1 and P300 of
appropriate spatial components were used as the Fisher's linear discriminant features. The classification accuracy was assessed by cross-validation with sequential testing of the data of a single block (all epochs of the same target cell) of the classifier trained using the other 14 blocks. The classification accuracy was determined as a proportion of the correctly recognized letters (out of 15). When performing testing, accuracy was calculated for different number of the stimulus sequences (one to five). The accuracy was calculated for each mode, subject, and signal feature extraction method.
Statistical analysis was performed in MATLAB using the generalized linear mixed effects models. A single constant coefficient was used as a random factor for the "subject" variable, while experimental conditions ("active attention", "cognitive load", "half-empty matrix", "mixing up elements") and latency correction modes were considered as fixed effects. The fixed effect significance was assessed using F-test. The following dependent variables were assessed: amplitude, latency, MAD of the N1 and P300 latencies, and classification accuracy. We used binomial regression to assess classification accuracy and linear regression to assess other parameters.
RESULTS
Fig. 2 shows the extracted spatial components N1 and P300 and the corresponding patterns (topographic distribution of weighting coefficients). The N1 component with the average latency of 187 ms had typical lateral occipital localization, while P300 with the latency of 315 ms had medial parietal localization.
Table 1 provides the group-averaged amplitudes of the N1 and P300 components obtained in each mode, before and after correction of latencies within individual epochs. The N1 and P300 amplitudes of the averaged ERPs increased after applying correction: F(1.258) = 581.24; p = 0.00000. The factor of active attention turned out to be significant for the N1 amplitude that increased relative to passive attention to the stimulus (mode 1): F(1.36) = 17.87; p = 0.00015. The increase in the N1 amplitude was reported for such factors, as "half-empty matrix" (F(1,110) = 16.10; p = 0.00011) and "cognitive load" (F(1.110) = 48.49; p = 0.00000). The increase in the P300 amplitude was reported for the "cognitive load" factor (F(1.110) = 18.01; p = 0.00005), while the decrease
Component Mode 1 Mode 2 Mode 3 Mode 4 Mode 5 Mode 6 Mode 7
Latency, ms
N1 187 ± 3.25 187 ± 2.6 184 ± 2.3 185 ± 2.2 185 ± 2.1 191 ± 2.6 191 ± 2.7
P300 323 ± 10.5 320 ± 9.3 303 ± 10.6 302 ± 10.4 305 ± 11.6 316 ± 10.3 325 ± 10.8
Mean absolute deviation (MAD) of latency, ms
N1 21.1 ± 1.5 18.9 ± 1.4 16.3 ± 1.2 18.0 ± 1.1 15.1 ± 0.8 19.2 ± 1.6 17.5 ± 1.5
P300 42.5 ± 2.2 41.3 ± 2.2 43.2 ± 2.7 44.9 ± 2.6 43.0 ± 2.4 44.0 ± 2.5 44.9 ± 2.7
Table 2. The average absolute latencies and the average indicators of their variability (MAD) for the N1 and P300 components in all modes. The mean and standard error of the mean are provided. N = 19 subjects
Table 3. The average classification accuracy obtained in all modes for one or two stimulus sequences that has been calculated for various signal feature sets used by the classifier — usual 11 EEG electrodes and the extracted spatial components N1 and P300 with or without peak latency correction. The mean and standard error of the mean are provided. N = 19 subjects
Method of the signal feature extraction Mode 1 Mode 2 Mode 3 Mode 4 Mode 5 Mode 6 Mode 7
Accuracy when using one stimulus sequence, %
EEG leads 66.7 ± 5.4 75.3 ± 4.6 77.29 ± 3.7 74.4 ± 4.0 84.2 ± 3.0 68.4 ± 4.6 80.7 ± 3.1
Spatial components 56.4 ± 5.8 63.5 ± 4.2 68.8 ± 3.7 62.8 ± 3.2 76.8 ± 2.8 59.7 ± 3.7 73.0 ± 3.3
Spatial components + correction of N1 + P300 76.8 ± 3.4 81.8 ± 2.9 86.3 ± 3.0 85.3 ± 2.6 86.7 ± 2.9 76.1 ± 2.5 84.9 ± 2.2
Accuracy when using two stimulus sequences, %
EEG leads 83.9 ± 4.1 90.0 ± 2.1 94.7 ± 1.4 94.0 ± 1.7 96.8 ± 1.3 88.8 ± 4.1 92.3 ± 2.1
Spatial components 75.8 ± 4.9 86.6 ± 3.3 86.3 ± 2.8 88.8 ± 2.0 93.3 ± 1.9 82.8 ± 4.0 93.0 ± 1.2
Spatial components + correction of N1 + P300 94.0 ± 1.2 97.1 ± 1.0 97.5 ± 1.6 93.3 ± 1.8 97.9 ± 1.0 95.4 ± 1.5 96.8 ± 1.5
was reported for the factor of "mixing up elements" (F(1.110) = 4.72; p = 0.032).
The average latencies of the N1 and P300 components together with the indicator of the latency variability (MAD) are provided in Table 2. The decrease in the N1 and P300 latencies were reported for the factor of "half-empty matrix": F(1.110) = 45.87, p = 0.00000 and F(1,110) = 24.51, p = 0.00000, respectively. The increase in the N1 latency was also reported for the factor of "mixing up elements": F(1.110) = 5.17; p = 0.025. Active attention resulted in the decrease of the N1 component MAD relative to the passive attention mode: F(1.36) = 1.60; p = 0.0016. The decrease in the N1 MAD was reported for the factors of "half-empty matrix" (F(1.110) = 12.43; p = 0.00061) and "cognitive load" (F(1.110) = 11.56; p = 0.00094). As for P300, the increase in MAD was reported for the factor of "mixing up elements": F(1.110) = 4.80; p = 0.03056.
Table 3 provides assessment of the average classification accuracy in all modes using different signal feature extraction methods: EEG channels and the channels for N1 and P300 of the corresponding spatial components, to which the latency correction was applied or not applied. The table provides data for the minimum number (1 or 2) of the stimulus sequences per letter, when accuracy is still low, and the differences between the modes are larger. The trend towards an increase in accuracy is reported for the "cognitive load" factor: F(1.108) = 3.39; p = 0.068.
Fig. 3 presents the average classification accuracy for different signal feature extraction methods and different number of the stimulus sequences. When using spatial filters (only two data vectors, for N1 and P300) without latency correction, the accuracy was the lowest and was even lower than when using the usual 11 EEG electrodes: F(1.3284) = 5.99, p = 0.014. Applying latency correction to the spatial component N1 only yielded higher accuracy, however, this option did not differ significantly from the option involving the use of usual EEG electrodes: F(1.3284) = 1.1771, p = 0.28. However, applying latency correction to the spatial component P300 only resulted in higher accuracy compared to the use of usual EEG electrodes: F(1.3284) = 24.51, p = 0.00000. The highest classification accuracy values were obtained when applying latency correction to both N1 and P300 (in each of the two appropriate spatial components). In this case, the accuracy was higher compared to the use of usual EEG electrodes (F(1.3284) = 24.29, p = 0.00000) and higher than when applying latency correction to P300 only (F(1.3284) = 4.34, p = 0.037) (as for the latter, the differences were reported for the 2nd and 3rd stimulus sequences: p < 0.05).
DISCUSSION
In our study we proposed an effective approach to assessing the ERP variability in the P300 BCI that allowed us to identify a number of factors affecting the ERP characteristics and explore the contribution of the variability effects to the command recognition accuracy in this interface.
To analyze the effects of the ERP latency variability, it is necessary to detect the components in individual (non-averaged) epochs. This process is very complicated due to both technogenic and physiological noise, that is why it is extremely important to make the most of valuable information contained in the EEG signal. Despite the fact that in some trials the effects of variability were studied in terms of the P300 BCI, the impact of these effects was estimated in usual EEG channels for the P300 component only [17, 24]. In our previous study, we applied latency correction to two components, N1 and P300, however, each component was analyzed in its own channel set [18]. The use of the combined information from all channels with simultaneous analysis of several components
100
95
90
85
80
75
70
65
60
Number of sequences
Fig. 3. The average classification accuracy with different number of the stimulus sequences calculated for various signal feature sets used by the classifier — usual 11 EEG electrodes, extracted spatial components N1 and P300 (no latency correction, latency correction applied to N1 only, to P300 only, or to both components, N1 and P300). The mean and standard error of the mean are provided. N = 19 subjects
-J' EEG leads —J— Spatial components —J— Spatial components + correction of N1 | Spatial components + correction of P30C
+ Spatial components + correction of N1 + P300_
in each of these channels can be a more effective approach. For example, the independent components extracted by ICA have been already used by the authors of papers on assessing variability (not related to BCI), however, these researchers have analyzed only one early component of ERP [21. 25]. Furthermore, the ICA method does not guarantee extraction of the components of interest for analysis. In this study we have proposed the use of spatial filters for extraction of two components, N1 and P300, that are functionally significant for the P300 BCI, with subsequent analysis of variability in these components instead of individual EEG channels. This method was used earlier [23], but in that study it was an additional step of preprocessing and extraction of the signal features for classification in the BCI, it had nothing to do with assessment of the ERP variability effects. The extraction of spatial components aimed at independent correction of these components has never been applied previously. Moreover, the use of the approach involving spatial components reduces the likelihood of erroneous peak detection within individual epochs compared to the use of signal in certain EEG leads, thereby making the variability analysis more objective.
An essential aspect of the work was to reveal the possible factors affecting the ERP characteristics in the P300 BCI. Active attention (the directive to emotionally count flashes) resulted in the increase in the N1 component amplitude, and the mechanism underlying such an increase was likely to include the decrease in the latency variability of responses to individual stimuli, since a simultaneous decrease in MAD was observed. The increase in the ERP components' amplitude relative to passive attention to stimuli in the P300 BCI has been earlier reported for such construct in this group [26]. Presumably, the directive to actively count the stimuli improves fixation of gaze on the target position within the matrix, which is important for the N1 component [27]. Lack of characters in all cells of the matrix is also likely to improve fixation of gaze on the target cell, since the N1 amplitude increase in the "half-empty matrix" mode has been reported along with the decrease in its variability. This is consistent with opposite effects on the N1 component in the environment, where tracing the target objects is complicated by their mobility [18], and supports the relationship between the features of oculomotor system function and the ERP components' variability [28].
The constant changing of characters in the matrix cells is likely to adversely affect attention to the target stimulus, as evidenced by the decrease in the P300 amplitude and the increase in its variability, along with the increase in the N1 latency. The negative impact of such manipulations with the stimulus environment on the P300 BCI is also confirmed by the fact that the subjects have reported trouble following instructions in the modes involving mixing-up characters. At the same time, an interesting and not entirely obvious result is that additional cognitive load applied in the modes involving mixing-up characters (counting consonants with the change of letters), in contrast, resulted in the increase in the N1 and P300 amplitudes. Furthermore, the effect reported for N1 at least
partially resulted from the reduced variability. It is well-known that the effects of individual responses' variability are enhanced when the subject's attention flits between two competing tasks [29]. Perhaps, the cognitive load integrated into a task of tracing the target events, that was used in our study, on the contrary, caused the increase in attention, that is why such modification of the stimulus environment may be prospective for the P300 BCI.
The potential effectiveness of using the factors that have a beneficial effect on attention in the BCI is also confirmed by the trend towards the increased target stimuli classification accuracy in the modes with cognitive load (Table 3). The method of applying variability correction not to usual EEG leads, but to the extracted spatial components N1 and P300, that has been proposed in our study, has ensured the best classification accuracy (Fig. 2). Furthermore, the largest increase in accuracy is observed when using the least number of the stimulus sequences (94% vs. 84%). This emphasizes the value of this method for the P300 BCI operation modes and provides superior results compared to that yielded by the studies also involving extraction of spatial components, but not taking into account the effects of variability [23, 30]. The fact, that the N1 and P300 components' contributions to the effectiveness of classification are unequal, attracts attention: the contribution of uncorrected N1 is larger than that of uncorrected P300. However, given the higher P300 variability, correction of its latency resulted in the significantly increased accuracy, thereby overperforming both correction of N1 only and the use of standard EEG electrodes.
To date, the fact, that in this study we have not adjusted latency in the non-target epochs, is considered to be a limitation of the approach. In the future, it would be necessary to develop an algorithm, which, for example, would allow us to avoid correction of low-amplitude peaks in the non-target epochs, for implementation of the online BCI.
CONCLUSIONS
The paper proposes an approach to analysis of the ERP latency variability in the extracted EEG spatial components. The use of this method in the P300 BCI has made it possible to achieve better results in terms of the command classification accuracy compared to the existing methods. Furthermore, the use of such an approach has revealed some factors of the stimulus environment and the P300 BCI operation modes having an impact on the ERP variability effects. Specifically, modifications of the interface affecting the user's attention, including the cognitive load applied in addition to the main task, and making it easier to fix gaze on the target objects have a beneficial effect on the ERP amplitude and the decrease in variability of individual responses to stimuli. The findings complement the existing knowledge of the mechanisms underlying the ERP latency variability and provide new reasons for the development of more effective BCI systems.
References
1. Mridha MF, Das SC, Kabir MM, Lima AA, Islam MR, Watanobe Y. Brain-Computer Interface: Advancement and Challenges. Sensors (Basel). 2021; 21 (17): 5746.
2. Orban M, Elsamanty M, Guo K, Zhang S, Yang H. A Review of Brain Activity and EEG-Based Brain-Computer Interfaces for
Rehabilitation Application. Bioengineering (Basel). 2022; 9 (12): 768.
3. Carelli L, Solca F, Faini A, Meriggi P Sangalli D, Cipresso P, Riva G, Ticozzi N, Ciammola A, Silani V, Poletti B. Brain-Computer Interface for Clinical Purposes: Cognitive Assessment and
Rehabilitation. Biomed Res Int. 2017; 2017: 1695290. 18.
4. Ganin IP, Kosichenko EA, Sokolov AV, Ioannisyanc OM, Arefev IM, Basova AYa, Kaplan AYa. Adaptation of the p300-based brain-computer interface for anorexia nervosa patients state evaluation. 19. Bulletin of RSMU. 2019; 2: 32-38.
5. Eldeeb S, Susam BT, Akcakaya M, Conner CM, White SW, Mazefsky CA. Trial by trial EEG based BCI for distress versus non distress classification in individuals with ASD. Sci Rep. 2021; 11 20. (1): 6000.
6. Abiri R, Borhani S, Sellers EW, Jiang Y, Zhao X. A comprehensive review of EEG-based brain-computer interface paradigms. J Neural Eng. 2019; 16 (1): 011001. 21.
7. Luck SJ. An introduction to the event related potential technique. MIT Press, Cambridge, MA; 2005.
8. Pan J, Chen X, Ban N, He J, Chen J, Huang H. Advances in 22. P300 brain-computer interface spellers: toward paradigm design
and performance evaluation. Front Hum Neurosci. 2022; 16: 1077717.
9. Farwell LA, Donchin E. Talking off the top of your head: toward
a mental prosthesis utilizing event-related brain potentials. 23. Electroencephalography and Clinical Neurophysiology. 1988; 70: 510-523.
10. Krusienski DJ, Sellers EW, McFarland DJ, Vaughan TM, Wolpaw JR. Toward enhanced P300 speller performance. J Neurosci 24. Methods. 2008; 167 (1): 15-21.
11. Rezeika A, Benda M, Stawicki P, Gembler F, Saboor A, Volosyak I. Brain-Computer Interface Spellers: A Review. Brain Sciences. 25. 2018; 8 (4): 57.
12. Allison BZ, Kubler A, Jin J. 30+ years of P300 brain-computer interfaces. Psychophysiology. 2020; 57 (7): e13569. 26.
13. Makeig S, Onton J. ERP features and EEG dynamics: an ICA perspective. Oxford handbook of event-related potential components. Oxford University Press, New York; 2011.
14. Dowdall JR, Luczak A, Tata MS. Temporal variability of the N2pc 27. during efficient and inefficient visual search. Neuropsychologia. 2012; 50 (10): 2442-53.
15. Dinstein I, Heeger DJ, Behrmann M. Neural variability: friend or
foe? Trends Cogn. Sci. 2015; 19 (6): 322-328. 28.
16. Ouyang G. Hildebrandt A, Sommer, W, Zhou C. Exploiting the intra-subject latency variability from single-trial event-related potentials in the P3 time range: A review and comparative 29. evaluation of methods. Neurosci. Biobehav. Rev. 2017; 75: 1-21.
17. Arico P, Aloise F, Schettini F, Salinari S, Mattia D, Cincotti 30. F. Influence of P300 latency jitter on event related potential-based brain-computer interface performance. Journal of neural engineering. 2014; 11 (3): 035008.
Ganin IP, Kaplan AYa. Study of the human brain potentials variability effects in P300 based brain-computer interface. Bulletin of RSMU. 2022; 3: 78-85.
Yagi Y, Coburn KL, Estes KM, Arruda JE. Effects of aerobic exercise and gender on visual and auditory P300, reaction time, and accuracy. Eur J Appl Physiol Occup Physiol. 1999; 80 (5): 402-8.
Kovarski K, Malvy J, Khanna RK, Arsène S, Batty M, Latinus M. Reduced visual evoked potential amplitude in autism spectrum disorder, a variability effect? Translational Psychiatry. 2019; 9 (1): 1-9.
Gonen-Yaacovi G, Arazi A, Shahar N, Karmo A, Haar S, Meiran N, Dinstein I. Increased ongoing neural variability in ADHD. Cortex. 2016; 81: 50-63.
Tou SLJ, Warschausky SA, Karlsson P, Huggins JE. Individualized Electrode Subset Improves the Calibration Accuracy of an EEG P300-design Brain-Computer Interface for People with Severe Cerebral Palsy. bioRxiv. 2023: 533775.
Pires G, Nunes U, Castelo-Branco M. Statistical spatial filtering for a P300-based BCI: tests in able-bodied, and patients with cerebral palsy and amyotrophic lateral sclerosis. Journal of neuroscience methods. 2011; 195 (2): 270-281. Thompson DE, Warschausky S, Huggins JE. Classifier-based latency estimation: a novel way to estimate and predict BCI accuracy. Journal of neural engineering. 2013; 10 (1): 016006. Milne E. Increased intra-participant variability in children with autistic spectrum disorders: evidence from single-trial analysis of evoked EEG. Front Psychol. 2011; 2: 51. Basyul IA, Kaplan AYa. Izmeneniya N200 i P300 komponentov potencialov, svyazannyx s sobytiyami, pri var'irovanii uslovij vnimaniya v sisteme Brain Computer Interface. Zhurnal vysshej nervnoj deyatel'nosti im. IP Pavlova. 2014; 64 (2): 159-65. Russia. Hillyard SA, Vogel EK, Luck SJ. Sensory gain control (amplification) as a mechanism of selective attention: electrophysiological and neuroimaging evidence. Philos Trans R Soc L. B Biol Sci. 1998; 353 (1373): 1257-70.
Zhang B, Stevenson SS, Cheng H, Laron M, Kumar G, Tong J, et al.
Effects of fixation instability on multifocal VEP (mfVEP) responses
in amblyopes. Journal of Vision. 2008; 8 (3): 16.
Polich J. Updating P300: an integrative theory of P3a and P3b.
Clin. Neurophysiol. 2007; 118 (10): 2128-48.
Zhang Y, Zhou G, Zhao Q, Jin J, Wang X, Cichocki A. Spatial-
Temporal Discriminant Analysis for ERP-Based Brain-Computer
Interface. IEEE Transactions on Neural Systems and Rehabilitation
Engineering. 2013; 21 (2): 233-43.
Литература
1. Mridha MF, Das SC, Kabir MM, Lima AA, Islam MR, Watanobe Y. Brain-Computer Interface: Advancement and Challenges. Sensors (Basel). 2021; 21 (17): 5746.
2. Orban M, Elsamanty M, Guo K, Zhang S, Yang H. A Review of Brain Activity and EEG-Based Brain-Computer Interfaces for Rehabilitation Application. Bioengineering (Basel). 2022; 9 (12): 768.
3. Carelli L, Solca F, Faini A, Meriggi P, Sangalli D, Cipresso P, Riva G, Ticozzi N, Ciammola A, Silani V, Poletti B. Brain-Computer Interface for Clinical Purposes: Cognitive Assessment and Rehabilitation. Biomed Res Int. 2017; 2017: 1695290.
4. Ганин И. П., Косиченко Е. А., Соколов А. В., Иоаннисянц О. М., Арефьев И. М., Басова А. Я., и др. Адаптация технологии интерфейсов мозг-компьютер на волне P300 для оценивания состояния больных нервной анорексией. Вестник Российского государственного медицинского университета. 2019; 2: 36-43.
5. Eldeeb S, Susam BT, Akcakaya M, Conner CM, White SW, Mazefsky CA. Trial by trial EEG based BCI for distress versus non distress classification in individuals with ASD. Sci Rep. 2021; 11 (1): 6000.
6. Abiri R, Borhani S, Sellers EW, Jiang Y, Zhao X. A comprehensive review of EEG-based brain-computer interface paradigms. J Neural Eng. 2019; 16 (1): 011001.
7. Luck SJ. An introduction to the event related potential technique. MIT Press, Cambridge, MA; 2005.
8. Pan J, Chen X, Ban N, He J, Chen J, Huang H. Advances in P300 brain-computer interface spellers: toward paradigm design and performance evaluation. Front Hum Neurosci. 2022; 16: 1077717.
9. Farwell LA, Donchin E. Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalography and Clinical Neurophysiology. 1988; 70: 510-523.
10. Krusienski DJ, Sellers EW, McFarland DJ, Vaughan TM, Wolpaw JR. Toward enhanced P300 speller performance. J Neurosci Methods. 2008; 167 (1): 15-21.
11. Rezeika A, Benda M, Stawicki P, Gembler F, Saboor A, Volosyak I. Brain-Computer Interface Spellers: A Review. Brain Sciences. 2018; 8 (4): 57.
12. Allison BZ, Kubler A, Jin J. 30+ years of P300 brain-computer interfaces. Psychophysiology. 2020; 57 (7): e13569.
13. Makeig S, Onton J. ERP features and EEG dynamics: an ICA perspective. Oxford handbook of event-related potential components. Oxford University Press, New York; 2011.
14. Dowdall JR, Luczak A, Tata MS. Temporal variability of the N2pc during efficient and inefficient visual search. Neuropsychologia. 2012; 50 (10): 2442-53.
15. Dinstein I, Heeger DJ, Behrmann M. Neural variability: friend or
foe? Trends Cogn. Sci. 2015; 19 (6): 322-328.
16. Ouyang G. Hildebrandt A, Sommer, W, Zhou C. Exploiting the intra-subject latency variability from single-trial event-related potentials in the P3 time range: A review and comparative evaluation of methods. Neurosci. Biobehav. Rev. 2017; 75: 1-21.
17. Aricô P, Aloise F, Schettini F, Salinari S, Mattia D, Cincotti F. Influence of P300 latency jitter on event related potential-based brain-computer interface performance. Journal of neural engineering. 2014; 11 (3): 035008.
18. Ганин И. П., Каплан А. Я. Изучение эффектов вариативности потенциалов мозга человека в интерфейсе мозг-компьютер на волне P300. Вестник Российского государственного медицинского университета. 2022; 3: 78-85.
19. Yagi Y, Coburn KL, Estes KM, Arruda JE. Effects of aerobic exercise and gender on visual and auditory P300, reaction time, and accuracy. Eur J Appl Physiol Occup Physiol. 1999; 80 (5): 402-8.
20. Kovarski K, Malvy J, Khanna RK, Arsène S, Batty M, Latinus M. Reduced visual evoked potential amplitude in autism spectrum disorder, a variability effect? Translational Psychiatry. 2019; 9 (1): 1-9.
21. Gonen-Yaacovi G, Arazi A, Shahar N, Karmo A, Haar S, Meiran N, Dinstein I. Increased ongoing neural variability in ADHD. Cortex. 2016; 81: 50-63.
22. Tou SLJ, Warschausky SA, Karlsson P, Huggins JE. Individualized Electrode Subset Improves the Calibration Accuracy of an EEG P300-design Brain-Computer Interface for People with Severe Cerebral Palsy. bioRxiv. 2023: 533775.
23. Pires G, Nunes U, Castelo-Branco M. Statistical spatial filtering for a P300-based BCI: tests in able-bodied, and patients with cerebral palsy and amyotrophic lateral sclerosis. Journal of neuroscience methods. 2011; 195 (2): 270-281.
24. Thompson DE, Warschausky S, Huggins JE. Classifier-based latency estimation: a novel way to estimate and predict BCI accuracy. Journal of neural engineering. 2013; 10 (1): 016006.
25. Milne E. Increased intra-participant variability in children with autistic spectrum disorders: evidence from single-trial analysis of evoked EEG. Front Psychol. 2011; 2: 51.
26. Басюл И. А., Каплан А. Я. Изменения N200 и P300 компонентов потенциалов, связанных с событиями, при варьировании условий внимания в системе Brain Computer Interface. Журнал высшей нервной деятельности им. ИП Павлова. 2014; 64 (2): 159-65.
27. Hillyard SA, Vogel EK, Luck SJ. Sensory gain control (amplification) as a mechanism of selective attention: electrophysiological and neuroimaging evidence. Philos Trans R Soc L. B Biol Sci. 1998; 353 (1373): 1257-70.
28. Zhang B, Stevenson SS, Cheng H, Laron M, Kumar G, Tong J, et al. Effects of fixation instability on multifocal VEP (mfVEP) responses in amblyopes. Journal of Vision. 2008; 8 (3): 16.
29. Polich J. Updating P300: an integrative theory of P3a and P3b. Clin. Neurophysiol. 2007; 118 (10): 2128-48.
30. Zhang Y, Zhou G, Zhao Q, Jin J, Wang X, Cichocki A. Spatial-Temporal Discriminant Analysis for ERP-Based Brain-Computer Interface. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2013; 21 (2): 233-43.
