CC BY
23Управление курсором мыши на основе комбинированного использования нейроинтерфейса и компьютерного зрения
fO CS
о
CS cd
Колегаев Борис Ярославич
аспирант, кафедра «Вычислительные системы и информационная безопасность», Донской государственный технический университет, kolegaevboris@gmail.com
В данной статье представлен подход к реализации метода управления курсором мыши, посредством интеграции нейроинтерфейса и методов компьютерного зрения. Этот подход предоставляет пользователю возможность манипулировать курсором мыши с помощью мозговой активности и ориентиров на лице. Нейроинтерфейс перехватывает сигналы мозга, связанные с определенными командами, тогда как алгоритмы компьютерного зрения следят за направлением взгляда для точного позиционирования курсора. Особенно важно, что такой подход имеет потенциал значительно облегчить взаимодействие людей с ограниченными возможностями. Люди, сталкивающиеся с физическими или двигательными ограничениями, смогут эффективно управлять компьютером, основываясь на своих мыслях и намерениях. Это направление исследований имеет важное практическое значение для создания более инклюзивных технологических решений и повышения качества жизни всех пользователей.
Ключевые слова: нейронный интерфейс, компьютерное зрение, ме-диапайп, Emotiv EPOC+.
Introduction
The constant evolution of technology is shaping the way we interact with computers and computing systems. Among the many innovations in human-computer interaction, non-contact control methods have emerged as a promising trend, offering users new ways to interact with computers without traditional physical input devices.
A non-contact cursor control system is important for people with disabilities, offering them an inclusive means of interacting with a computer or other systems whose interface involves tactile contact with an input device. In the face of global pandemic challenges, including the recent COVID-19, a non-contact control system is even more important. During such crises, reducing direct physical contact with public surfaces becomes paramount to contain the spread of infection.
Two major technologies in this area, computer vision and neural interfaces, have separately shown great promise in enabling touchless interaction. Computer vision has demonstrated a wide range of applications including face recognition, gesture tracking and object detection, which has revolutionised the user experience in various domains [1]. On the other hand, neural interfaces have attracted considerable attention because of their ability to directly interpret brain signals to perform certain actions [2]. This paper presents an approach that combines the capabilities of computer vision and a neural interface, specifically Emotiv EPOC+ [3], for non-contact cursor control of a computer mouse.
The aim of the work is to realise contactless control of a computer mouse through the use of computer vision methods and neural interface technology. To achieve this goal the following objectives need to be addressed:
1) Controls the position of the computer mouse cursor by directing the user's gaze.
2) Implementation of a click using a neural interface.
Main part. Python was chosen as the development language due to the availability of a large number of libraries required for the project implementation.
Cursor position control. Mediapipe library [4] was used to detect the gaze direction in this project. This library contains pre-trained and processor optimised neural networks that allow the detection of human face, body and hands in an image. In face detection, Mediapipe provides access to the coordinates of 468 facial points that cover various anatomical features of the face including eyes, eyebrows, nose, mouth and facial contours. An example of key point detection on a person's face is shown in Figure 1.
О Ш
m x
<
m о x
X
Figure 1. Points detected with Mediapipe
Since it is necessary to use the user's gaze direction to control the mouse cursor, it was decided to read information about the nose bridge coordinates from the webcam and bind it to the coordinates of the computer mouse cursor. The cursor movement itself is implemented using the moveTo() method of the pyautogui library [5], which allows to automate actions that are usually performed manually by the user, such as moving the cursor, mouse clicks, keystrokes and other actions.
To ensure convenience in using this approach, the nose bridge coordinates are read only in a defined rectangular area just above the center of the screen. Within this area, the nose bridge coordinates are converted relative to the screen resolution. Thus, if the user is not looking in the correct area, gaze control does not work. When testing this approach, the following problems were identified:
1) In order for the program to work correctly, it is necessary to help the user to correctly position their face relative to a given area on the screen.
2) The area where the nose bridge coordinates were being read took up some space on the screen, making it difficult to see anything behind it.
To solve these problems, it was decided to use PyQT library [6] to create a complete interface for the program. As a result, the video stream is read from the camera and displayed in a separate window. If the user is incorrectly positioned relative to the recognition area, he can see himself in the video and orient his position due to this. At the same time, the window with the video stream is semi-transparent, which allows the user to see the contents of the computer desktop behind it. Once the user has correctly positioned himself in relation to the recognition area, only the recognition area itself becomes visible and remains semitransparent.
An example of the program operation at incorrect and correct positioning of the user is shown in Fig. 2 and Fig. 3.
Mouse click control. Emotiv EPOC+ neurointerface was chosen as a device for reading brain activity. This model is quite popular among researchers, it is represented by a portable device Fig. 4, which is placed on the user's head and has 14 sensors to record brain activity. These sensors cover different areas of the head, which allows to obtain data from different parts of the brain. Along with the device, special software EmotivBCI [7] Fig. 5.
Figure 4. Emotiv Epoc+ device, side and top view
□ EmotivBCI: iiJW
EMOTIVBCI
i
©
& X
IHM
□ x
O n o
G» | Push 9 | Right
I
Figure 2. Semitransparent video stream at incorrect positioning relative to the recognition area
Figure 3. Video stream trimming at correct positioning relative to the recognition area
Figure 5. EmotivBCI programme interface
After coating the sensors with a special gel and placing the device on the surface of the head, the calibration process must be performed. Calibration allows the EmotivBCI software to understand the individual characteristics of the user's brain activity and adjust the signal recognition algorithms to the user's brain. This is an important step to improve the accuracy and reliability of mental command recognition. Once the calibration is complete, you can move on to the training process. The EmotivBCI software allows you to train the neural interface to recognise certain patterns of brain activity. These patterns can be triggered by a particular thought, imagined movement or facial expression. The training consists of teaching the user to reproduce patterns of brain activity that will be classified by the software as certain commands.
The pyautogui library, namely the click() method, can be used again to make a mouse click. However, first of all, it is necessary to transfer information about successful recognition of the brain activity pattern to the program. To solve this problem, the Emotiv Cortex API (Application Programming Interface) [8] was used. This API provided by Emotiv allows developers to interface with their neurointerfaces such as Emotiv EPOC+, Emotiv Insight and others. Cortex API provides various functions and methods that allow to acquire brain activity data, control devices, process the data and implement various applications and interfaces based on the collected data.
Conclusion. The combination of the Emotiv EPOC+ neural interface with computer vision presents an interesting new approach to non-contact mouse cursor control. However, it should
X X
o
OD A C.
X
OD m
o
2 O
ho CJ
be noted that some users may find Emotiv EPOC+ uncomfortable to use due to the need to regularly apply a special gel to its sensors. This can lead to interruptions for the sensor moisturising procedure every 30-40 minutes, making it difficult to interact with the computer continuously.
To overcome this problem, it may be worth considering a switch to using a neural interface with dry electrodes. This approach would eliminate the need for gel and make the use of the interface more convenient and efficient.
Further development of the project may include research and integration of simpler neural interfaces such as NeuroPlay [9], which have fewer sensors but may be accurate and functional enough to implement basic cursor control commands.
Also, a possible direction for the development of the project is to expand the functionality of the mouse. For example, adding the ability to click the left mouse button or scroll the wheel can significantly increase usability and expand the possibilities for the user.
Non-contact mouse cursor control with simultaneous use of neural interface and computer vision represents an exciting field for research and development. Further improvements in technology and the integration of new features could significantly improve the user experience and expand the scope of such systems in the future.
Mouse cursor control based on the combined use of neural interface and
computer vision Kolegaev B.Ya.
Don State Technical University
JEL classification: C10, C50, C60, C61, C80, C87, C90
This paper presents an approach to realise mouse cursor control by integrating neural interface and computer vision techniques. The proposed technique allows users to control a mouse cursor using a combination of brain activity and landmarks on the human face. The neural interface picks up brain signals associated with specific commands, and computer vision algorithms track the direction of human gaze to position the cursor.
Keywords: neural interface, computer vision, mediapipe, Emotiv EPOC+
References
1. Machine Vision in Everyday Life: Playful Interactions with Visual Technologies in Digital Art, Games, Narratives and Social Media: [website]URL:https://www.researchgate.net/publication/321335312_Machine_Vi sion_in_Everyday_Life_Playful_Interactions_with_Visual_Technologies_in_Digit al_Art_Games_Narratives_and_Social_Media / (accessed 20.06.2023).
2. Brain-computer interfaces for communication and control / Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM // Clinical Neurophysiology 113 (2002) 767-791.
3. emotiv epoc + / emotiv.com [website]. -URL: https://www.emotiv.com/epoc/ (date of access: 20.06.2023)
4. MediaPipe Hands / google.github.io:[website]. -URL: https://google.github.io/mediapipe/ (accessed 20.06.2023)
5. PyAutoGUI / pyautogui.readthedocs.io: [website]. -URL: https://pyautogui.readthedocs.io/en/latest/ (date of reference: 20.06.2023)
6. Qt for Python / https://doc.qt.io/: [website]. -URL: https://doc.qt.io/qtforpython-6/ (accessed 20.06.2023)
7. EmotivBCI / emotiv.com/: [website]. -URL: https://www.emotiv.com/emotiv-bci/ (date of access: 20.06.2023)
8. cortex-v2-example / github.com/[site]. -URL: https://github.com/Emotiv/cortex-v2-example (date of access: 20.06.2023)
9. NeuroPlay / neuroplay.ru /[website]. -URL: https://neuroplay.ru/ (date of access: 20.06.2023)
fO CS
о
CS
со
о ш m
X
<
m о х
X
