CC BY
57
studying the ability to control human phantom fingers in p300 brain-computer interface
Kaplan AYa1>2 Zhigulskaya DD1, Kirjanov DA1
1 Laboratory for Neurophysiology and Nero-Computer Interfaces, Faculty of Biology, Lomonosov Moscow State University, Moscow, Russia
2 Laboratory for Brain-Machine Interfaces and Applied Neuroengineering,
Lobachevsky National Research State University of Nizhny Novgorod, Nizhny Novgorod, Russia
In this work we have tested the assumption that an individual can control a target finger of a phantom by voluntarily focusing his attention on the luminous marker located on that finger in the complex of a P300 wave-based brain-computer interface (P300 BCI) and an anthropomorphic phantom. Because each correct movement of phantom fingers indicates a sufficient mental effort aimed at this action, creating a new ideomotor training simulator of smaller movements of the hand becomes possible. Our study included 21 volunteer subjects of both sexes aged 18-25. It was shown that with P300 BCI complex the subjects learned to control phantom fingers on the first day of the experiment, the percentage of successful attempts being no less than 69 %. Failures were mainly related to the insufficient attention focus on luminous markers on the target phantom fingers. We hypothesize that P300 BCI — Hand Phantom complex can be a basis for developing a fine motor skills simulator.
Keywords: brain-computer interface, BCI, electroencephalogram, neurorehabilitation, stroke, evoked potentials, P300
Funding: this work was partially supported by the Skolkovo Foundation, grant no. 1110034, and the Russian Science Foundation, grant no. 15-19-20053.
[23 Correspondence should be addressed: Alexandr Kaplan
ul. Leninskie gory, d. 1, str. 12, Moscow, Russia, 117234; [email protected]
Received: 11.04.2016 Accepted: 15.04.2016
изучение возможности управления отдельными пальцами фантома кисти руки человека в контуре интерфейса мозг-компьютер на волне p300
А. Я. Каплан1,2 Д. Д. Жигульская1, Д. А. Кирьянов1
1 Лаборатория нейрофизиологии и нейрокомпьютерных интерфейсов, биологический факультет, Московский государственный университет имени М. В. Ломоносова, Москва
2 Лаборатория разработки мозго-машинных интерфейсов и прикладной нейроинженерии, Национальный исследовательский Нижегородский государственный университет им. Н. И. Лобачевского, Нижний Новгород
В исследовании проверяли предположение, что в контуре предложенного комплекса интерфейса мозг-компьютер на основе волны Р300 (ИМК-Р300) и антропоморфного фантома кисти руки человек сможет управлять сгибанием целевого пальца фантома, произвольно фокусируя свое внимание на расположенном на этом пальце световом маркере. Поскольку каждое правильное срабатывание пальцев фантома будет свидетельствовать о достаточной выраженности направленных на это действие мысленных усилий, открывается перспектива создания на этой основе идеомотор-ного тренажера мелких движений кисти. В качестве испытуемых-добровольцев были задействованы 21 человек обоих полов в возрасте 18-25 лет. Было показано, что испытуемые действительно уже в первый экспериментальный день приобретали навык управления пальцами фантома руки в контуре ИМК-Р300 с надежностью не менее 69 % успешных попыток. При этом основные ошибки управления были связаны с недостаточной концентрацией внимания на сигналах светового маркера целевых пальцев фантома. Сделано предположение, что разработанный комплекс «ИМК-Р300 — Фантом кисти» может послужить основой для создания тренажера мелкой моторики кисти.
Ключевые слова: интерфейс мозг-компьютер, ИМК, электроэнцефалограмма, нейрореабилитация, инсульт, вызванные потенциалы, P300
Финансирование: работа выполнена при частичной поддержке Фонда «Сколково», грант № 1110034, и РНФ, грант № 15-19-20053.
[X] Для корреспонденции: Александр Яковлевич Каплан
117234, г. Москва, ул. Ленинские горы, д. 1, стр. 12; [email protected]
Статья поступила: 11.04.2016 Статья принята к печати: 15.04.2016
Including the elements of mental practice into rehabilitation programs for patients with impaired motor skills after stroke or neurological trauma is becoming a new trend in modern neurorehabilitation [1-4]. This approach is based on Nikolai Bernstein's concept according to which creating a mental plan
for executing the motor act rather than performing the act itself would be a true response to the tasks set by the environment. [5]. Such activity boosts neural plasticity processes that drive restoration of neuronal pools for motor control [6, 7]. Motor imagery that triggers restructuring of the motor act plan in
CTATbfl I HEMPQMHTEPOEMCbl
neuronal networks can be as effective for the restoration of impaired motor coordination as the actual execution of the movement [1, 3, 4]. Indeed, the studies based on transcranial magnetic stimulation used to test cortical excitability demonstrated the activation of cortical structures associated with motor representation during motor imagery [8, 9].
However, despite the seeming simplicity of motor imagery, whether it is effective in triggering cortical restructuring depends on mental effort intensity, stability and direction [10, 11]. Yet, an individual needs a feedback on motor imagery quality, otherwise his or her mental effort will weaken if repeated multiple times, motor images will fade, and he or she will gradually lose interest in the training procedure. The feedback loop can be provided by brain-computer interface (BCI) technologies that make use of mu-rhythm depression recorded by EEG to detect mental representations of movements and thansform those events into commands for controlling virtual reality objects or real objects [2,9,10,12]. Thus, the operator's mental effort shaped as movement representation can be translated into the actions of physical objects or virtual objects on the screen via BCI. Given a person is motivated, his or her motor imagery skills can develop sufficient intensity and sustainability over time. Using such skills during training sessions is an effective trigger for adaptive plasticity processes in the corresponding brain structures [1, 9, 12].
The weakness of this approach is the extremely low level of differentiation of mental movement representations in relation to their subsequent BCI-based identification. Only 2 to 3 motor images can be reliably (with 0.6-0.7 probability) identified by motor imagery based BCIs. Usually, those are left or right arm/ leg movements [10, 12]. It is insufficient for establishing several feedback channels for mental rehearsal of fine motor skills, such as movements of individual fingers, which are the hardest to restore.
At the same time, there is a BCI technology that makes use of reliable EEG-based detection of human focus of attention on external screen characters and provides for a library of no less than 36 commands [13]. Detection of the attention focus is based on EEG responses to short flashes of external objects, such as symbols on the screen; response to a target stimulus is identified on the basis of specific response parameters, a P300 wave in particular [1, 14, 15]. Still, due to the necessity of using a stimulus medium formed by symbols, application of BCI-P300 in rehabilitation could be reduced to communicators, such as for text entry in patients with severe motor and speech impairment, and for activation of remote control buttons [16].
In this work we test the hypothesis that BCI-P300 can be used non-conventionally, namely, as a basis for a training simulator for improving fine motor skills (of fingers) with a multichannel feedback. In such a simulator, the anthropomorphic hand phantom with movable fingers can be used as an actuator. We speculate that using a BCI-P300 system proposed in this work, an individual will be able to control phantom finger flexion by focusing his or her attention on the fingers.
The onset of finger movement will indicate sufficient intensity of mental effort aimed at focusing attention on the process.
METHODS
The study enrolled 12 right-handed volunteers (6 male and 15 female, 18-25 years of age) with normal or corrected-to-normal vision. The study was approved by Bioethics Committee of Lomonosov Moscow State University. All participants gave written informed consent. The subjects were seated in a
comfortable armchair, arms on armrests. Above the right arm of the subject covered with non-transparent fabric, the anthropomorphic hand was placed, its movable fingers were connected to servo-motors by flexible cords. Light markers (light-emitting diodes) were attached to the distal phalanx of each phantom finger, light intensity being 5 cd/m2. Turning them on and off was a visual stimulus for event-related potentials (ERPs) recorded by EEG.
For unipolar EEG recording, 8 electrodes (Cz, Pz, PO3, PO4, PO7, PO8, O1, O2), the referential indifferent ear electrode and the Ground Electrode in Fpz position were used. To record biopotentials, NVX52 electroencephalograph (MKS, Russia) was set up to a sampling frequency of 500 Hz, a passband of 0.1-30 Hz (second order Butterworth filter), a band-reject filter of 50 Hz
To identify ERPs associated with target stimuli, i.e., flashes of the light markers on the phantom finger that the subject's attention was focused on, a classifier based on Fisher's linear discriminant analysis (LDA) was used; its output was transformed into a finger-flexion command for the phantom if the preset threshold was exceeded. The classifier analyzed short intervals of 10 ms long that the 0-800 ms EEG signal was split into (0-800 ms is time from stimulus presentation). Commands for stimuli and servomotors activation were executed by phantom hand components, namely, two programmed microcomputers Freeduino Nano v5ATmega328 (Russia). Both microcomputers were connected to the computer via USB ports. The phantom was connected to NVX52 amplifier for EEG recording, which facilitated synchronized ERP recording, their processing by the classifier and command generation.
During the experimental session, a randomized sequence of ten flashes on each phantom finger was presented to the subject to implement one command, i.e., to flex one phantom finger that the subject's attention was drawn to by flashes. Each command was preceded by instructing the subject on what phantom finger had to be chosen. Light-emitting diodes were on for 50 ms; the interval between the stimuli was 150 ms. Every subject had 20 experimental sessions spaced by small breaks; their results were used to assess how effectively the subject operated the hand phantom.
The classifier was trained right before the experimental session on non-random sampling sets of target and non-target ERPs. The procedure lasted for 4 min.
To estimate which operating mode was the most effective for attention focus control, two types of signals were tested, with subject's attention focused on flash offset and flash onset. In both cases the length of the signal was set to 50 ms.
To assess how effectively the phantom fingers were operated, control accuracy was measured by the number of right, wrong or zero phantom finger flexions resulting from the subject focusing the attention on a certain phantom finger. Statistical analysis of data obtained from all subjects was carried out using Stasoft Statistica 7.0 software. To estimate the difference in the effectiveness of both operating modes Wilcoxon sign ranked test was used.
RESULTS
Fig.1 shows statistical data on subjects' performance accuracy during the procedure of selecting a target phantom finger for its further flexion by shifting the focus of attention to it. The data are presented as a percentage of the total number of trials. The accuracy rates of control of phantom fingers were averaged across the groups for various operating modes (using
90
80
70
60
50
40
30
20
10
Median
I—. Quartiles, 1—1 25-75 %
~T Range, 0-100 %
1st 2nd
Operating modes
Fig. 1. Accuracy of phantom finger control in two BCI operating modes
In the first and second operating modes, the stimuli used were flashing events
onset and offset, respectively.
flash onset and offset as stimuli). The results did not show any statistical difference and were 69 % and 57 % , respectively, with maximal control accuracy being 95 % for some subjects in both modes. The range of accuracy scores for each subject in the second mode was substantially wider than in the first mode. It leads us to conclude that using BCI-P300 technology with light markers placed on phantom fingers controlled via BCI can be seen as a basis for developing a neurosimulator for fine motor skills, with a customized set-up of light signaling for each operating mode.
As shown in fig. 1, the subjects could not issue a command to flex the target finger by shifting their attention to the light marker placed on the phantom hand even in the optimal operating mode. In case of errors, either a non-target finger is flexed or no movement occurs; therefore, analysis of errors of both types was carried out; its results are presented in fig. 2. Scores are presented as absolute values of the number of errors of both types that occurred during 20 attempts to initiate target finger flexion. It should be reminded that each trial could result in one right and five wrong responses (four non-target finger responses and one zero movement response).
As shown in fig. 2, subjects rarely failed to select the target
finger for flexion. In average, there were no more than 1.5 errors in 20 trials. But in a greater number of cases (5-6 depending on the operating mode), subjects failed to initiate flexion of any finger leaving the phantom hand motionless. Wilcoxon test showed significant differences (p <0.05) between the number of type 1 and type 2 errors for each operating mode.
Initiating a non-target finger movement is possibly related to the weak sustained attention; attention is drawn to the nontarget finger, and the latter is wrongly detected as a flexion target. At the same time, the absence of commands for phantom fingers on completing another attempt to activate the target finger indicates insufficient attention focus on flashing events. As a result, no distinct ERPs are generated in response to target stimuli, and classifier output does not reach the threshold value for command issue.
DISCUSSION
The obtained data confirmed that it is possible to design a BCI-P300 - Phantom Hand system, in which an individual can control flexion of phantom fingers by voluntarily focusing his or her attention on them. Because flexion of a finger that an individual selects in his mind indicates the sufficient intensity of his mental effort to focus his attention on triggering the movement, development of the effective training simulator for fine motor skills that makes use of this technology is highly possible.
However, up to now no attempts to use a BCI-P300 technology for feedback in motor function training have been reported. It is probably due to the fact that the operator does not need to perform motor imagery or his attention is shifted to external objects
However, since pivotal works of Botvinnik and Cohen [17] were published, their findings confirmed multiple times thereafter [18,19], it has been known that under certain conditions the external object, such as a rubber hand, can be easily and reliably identified with the internal representation of one's own hand. Moreover, there is no need to imagine the movement of one's own hand to activate the neurons in the motor cortex, it is enough to watch the hand of the other person or its artificial replica move [20, 21]. Coupled with the data obtained in this work confirming the possibility of using the BCI-P300 technology for controlling individual fingers of the hand phantom, the facts mentioned above hold promise for creating a fine motor skills training neurosimulator based on the BCI-P300 — Phantom Hand system.
18 16 rs 14
e
1st 2nd
Wrong action
■
t ■
■ •
1st 2nd
Zero action
■ Median
Quartiles, 25-75 %
Range, -1- 0-100 %
Operating modes:
1st — flashing event onset;
2nd — flashing event offset
Fig. 2. Number of type 1 and type 2 errors in both operating modes with subjects attempting to issue a command to flex the target finger by focusing their attention on the light marker
Statistically significant difference ( Wilcoxon test, p<0.05) was observed when comparing averaged type 1 and type 2 errors for each operating mode.
СТАТЬЯ I НЕЙРОИНТЕРФЕЙСЫ
CONCLUSIONS
The BCI-P300 technology can be used to generate commands for mental control of fingers of the human hand phantom with reliability of no less than 69 %, which is sufficient to develop a fine motor skills neurosimulator.
The majority of the BCI-P300 operator's errors in controlling the fingers of the human hand phantom are associated with insufficient focus on the signals of the light markers placed on phantom fingers, which necessitates the improvement of the stimulus medium.
References
1. Kaplan AYa. Nerofiziologicheskie osnovaniya i prakticheskie realizatsii tehnologii mozg-mashinnykh interfeisov v nevrologicheskoi reabilitatsii. Hum Physiol. 2016; 42 (1): 118-27. Russian.
2. Mokienko OA, Chernikova LA, Frolov AA, Bobrov PD. [Motor imagery and its practical application]. Zh Vyssh Nerv Deiat Im I P Pavlova. 2013 Mar-Apr; 63 (2): 195-204. Russian.
3. Park J, Lee N, Cho M, Kim D, Yang Y. Effects of mental practice on stroke patients' upper extremity function and daily activity performance. J Phys Ther Sci. 2015 Apr; 27 (4): 1075-7.
4. Kho AY, Liu KP, Chung RC. Meta-analysis on the effect of mental imagery on motor recovery of the hemiplegic upper extremity function. Aust Occup Ther J. 2014 Apr; 61 (2): 38-48.
5. Bernshtein NA. Fiziologiya dvizhenii i aktivnost'. Gazenko OG, editor. Moscow: Nauka; 1990. 494 p. Russian.
6. Skvortsova VI, Krylov VV. Gemorragicheskii insul't: Prakticheskoe rukovodstvo. Moscow: GEOTAR-Media; 2005. 160 p. Russian.
7. Hara Y. Brain plasticity and rehabilitation in stroke patients. J Nippon Med Sch. 2015; 82 (1): 4-13.
8. Mokienko OA, Chervyakov AV, Kulikova SN, Bobrov PD, Chernikova LA, Frolov AA, et al. Increased motor cortex excitability during motor imagery in brain-computer interface trained subjects. Front Comput Neurosci. 2013 Nov 22; 7: 168.
9. Yuan H, He B. Brain-computer interfaces using sensorimotor rhythms: current state and future perspectives. IEEE Trans Biomed Eng. 2014 May; 61 (5): 1425-35.
10. Vasil'ev AN, Liburkina SP, Kaplan AYa. Lateralizatsiya patternov EEG u cheloveka pri predstavlenii dvizhenii rukami v interfeise mozg-komp'yuter. Zh Vyssh Nerv Deiat Im I P Pavlova. 2016; 66 (3): 1-11. Russian.
11. Machado S, Lattari E, de Sa AS, Rocha NB, Yuan TF, Paes F, et al. Is mental practice an effective adjunct therapeutic strategy for upper limb motor restoration after stroke? A systematic review and meta-analysis. CNS Neurol Disord Drug Targets. 2015; 14 (5): 567-75.
12. Frolov AA, Biryukova EV, Bobrov PD, Mokienko OA, Platonov AK, Pryanichnikov VE, et al. Printsipy neiroreabilitatsii, osnovannye na ispol'zovanii interfeisa "mozg-komp'yuter" i biologicheski adekvatnogo upravleniya ekzoskeletonom. Hum Physiol. 2013; 39 (2): 99-105. Russian.
13. Powers JC, Bieliaieva K, Wu S, Nam CS. The Human Factors and Ergonomics of P300-Based Brain-Computer Interfaces. Brain Sci. 2015 Aug 10; 5 (3): 318-56.
14. Bamdad M, Zarshenas H, Auais MA. Application of BCI systems in neurorehabilitation: a scoping review. Disabil Rehabil Assist Technol. 2015; 10 (5): 355-64.
15. Kaplan AY, Shishkin SL, Ganin IP, Basyul IA, Zhigalov AY. Adapting the P300-based brain-computer interface for gaming: a review. IEEE Trans Comput Intell AI Games (Special Issue on Brain/ Neuronal-Computer Games Interfaces and Interaction). 2013; 5 (2): 141-9.
16. Cipresso P, Carelli L, Solca F, Meazzi D, Meriggi P, Poletti B, et al. The use of P300-based BCIs in amyotrophic lateral sclerosis: from augmentative and alternative communication to cognitive assessment. Brain Behav. 2012 Jul; 2 (4): 479-98.
17. Botvinick M, Cohen J. Rubber hands 'feel' touch that eyes see. Nature. 1998 Feb 19; 391 (6669): 756.
18. Kammers MP, de Vignemont F, Verhagen L, Dijkerman HC. The rubber hand illusion in action. Neuropsychologia. 2009 Jan; 47 (1): 204-11.
19. Kalckert A, Ehrsson HH. The moving rubber hand illusion revisited: comparing movements and visuotactile stimulation to induce illusory ownership. Conscious Cogn. 2014 May; 26: 117-32.
20. Rizzolatti G, Fogassi L, Gallese V. Neurophysiological mechanisms underlying the understanding and imitation of action. Nat Rev Neurosci. 2001 Sep; 2 (9): 661-70.
21. Gatti R, Tettamanti A, Gough PM, Riboldi E, Marinoni L, Buccino G. Action observation versus motor imagery in learning a complex motor task: a short review of literature and a kinematics study. Neurosci Lett. 2013 Apr 12; 540: 37-42.
Литература
1. Каплан А. Я. Нерофизиологические основания и практические реализации технологии мозг-машинных интерфейсов в неврологической реабилитации. Физиол. чел. 2016; 42 (1): 118-27.
2. Мокиенко О. А., Черникова Л. А., Фролов А. А., Бобров П. Д. Воображение движения и его практическое применение. Журн. высш. нервн. деят. 2013. 63 (2): 195-204.
3. Park J, Lee N, Cho M, Kim D, Yang Y. Effects of mental practice on stroke patients' upper extremity function and daily activity performance. J Phys Ther Sci. 2015 Apr; 27 (4): 1075-7.
4. Kho AY, Liu KP, Chung RC. Meta-analysis on the effect of mental imagery on motor recovery of the hemiplegic upper extremity function. Aust Occup Ther J. 2014 Apr; 61 (2): 38-48.
5. Бернштейн Н. А. Физиология движений и активность. Газен-ко О. Г., редактор. М.: Наука; 1990. 494 с.
6. Скворцова В. И., Крылов В. В. Геморрагический инсульт: Практическое руководство. М: ГЭОТАР-Медиа; 2005. 160 с.
7. Hara Y. Brain plasticity and rehabilitation in stroke patients J Nippon Med Sch. 2015; 82 (1): 4-13.
8. Mokienko OA, Chervyakov AV, Kulikova SN, Bobrov PD, Chernikova LA, Frolov AA, et al. Increased motor cortex excitability during motor imagery in brain-computer interface
trained subjects. Front Comput Neurosci. 2013 Nov 22; 7: 168.
9. Yuan H, He B. Brain-computer interfaces using sensorimotor rhythms: current state and future perspectives. IEEE Trans Biomed Eng. 2014 May; 61 (5): 1425-35.
10. Васильев А. Н., Либуркина С. П., Каплан А. Я. Латерализа-ция паттернов ЭЭГ у человека при представлении движений руками в интерфейсе мозг-компьютер. Журн. высш. нервн. деят. 2016; 66 (3): 1-11.
11. Machado S, Lattari E, de Sa AS, Rocha NB, Yuan TF, Paes F, et al. Is mental practice an effective adjunct therapeutic strategy for upper limb motor restoration after stroke? A systematic review and meta-analysis. CNS Neurol Disord Drug Targets. 2015; 14 (5): 567-75.
12. Фролов А. А., Бирюкова Е. В., Бобров П. Д., Мокиенко О. А., Платонов А. К., Пряничников В. Е. и др. Принципы нейро-реабилитации, основанные на использовании интерфейса «мозг-компьютер» и биологически адекватного управления экзоскелетоном. Физиол. чел. 2013; 39 (2): 99-105.
13. Powers JC, Bieliaieva K, Wu S, Nam CS. The Human Factors and Ergonomics of P300-Based Brain-Computer Interfaces. Brain Sci. 2015 Aug 10; 5 (3): 318-56.
14. Bamdad M, Zarshenas H, Auais MA. Application of BCI systems
in neurorehabilitation: a scoping review. Disabil Rehabil Assist Technol. 2015; 10 (5): 355-64.
15. Kaplan AY, Shishkin SL, Ganin IP, Basyul IA, Zhigalov AY. Adapting the P300-based brain-computer interface for gaming: a review. IEEE Trans Comput Intell AI Games (Special Issue on Brain/ Neuronal-Computer Games Interfaces and Interaction). 2013; 5 (2): 141-9.
16. Cipresso P, Carelli L, Solca F, Meazzi D, Meriggi P, Poletti B, et al. The use of P300-based BCIs in amyotrophic lateral sclerosis: from augmentative and alternative communication to cognitive assessment. Brain Behav. 2012 Jul; 2 (4): 479-98.
17. Botvinick M, Cohen J. Rubber hands 'feel' touch that eyes see. Nature. 1998 Feb 19; 391 (6669): 756.
18. Kammers MP, de Vignemont F, Verhagen L, Dijkerman HC. The rubber hand illusion in action. Neuropsychologia. 2009 Jan; 47 (1): 204-11.
19. Kalckert A, Ehrsson HH. The moving rubber hand illusion revisited: comparing movements and visuotactile stimulation to induce illusory ownership. Conscious Cogn. 2014 May; 26: 117-32.
20. Rizzolatti G, Fogassi L, Gallese V. Neurophysiological mechanisms underlying the understanding and imitation of action. Nat Rev Neurosci. 2001 Sep; 2 (9): 661-70.
21. Gatti R, Tettamanti A, Gough PM, Riboldi E, Marinoni L, Buccino G. Action observation versus motor imagery in learning a complex motor task: a short review of literature and a kinematics study. Neurosci Lett. 2013 Apr 12; 540: 37-42.
