CC BY
21
UDC 519.683.8:519.6
About one Model of Computer Control on the Basis of
Gaze Tracking
I. M. Gostev, E. A. Sibirtseva
National Research University Higher School of Economics 20, Myasnitskaya str., Moscow, Russia, 101000
The methodology of human-computer interaction based on the determination of the eyeball position and its movement direction. For interference and noise elimination from extraneous lighting devices infra-red illumination which allows receiving only one highlight on a cornea from an infra-red light-emitting diode is used. Thus the entrance video-stream in an infra-red range is processed in real time in two parallel processes. One process is used for definition of position of a highlight from a light-emitting diode on corneas of eyes, another process — for definition of position of a pupil and a direction of its moving. Aggregate of co-ordinates of a highlight of a pupil, position of a cornea and a direction of its moving allow remote to control of the computer. For prevention of loss of "object" the original technique of accumulation of the previous positions of an eyeball and forecasting of a direction of its moving is used. The given methodology of contactless control of the computer allows simulating pressing keys of the keyboard and/or mouse movement. The developed hardware and software system for gaze direction tracking be used as an alternative method of input medium, which is closer to the natural way of interaction with the environment, as well as the only way to work with a computer for the users with reduced mobility.
Key words and phrases: gaze tracking, non-command interface, human-computer interaction, image processing, pattern recognition.
1. Introduction
At the current moment the non-contact interfaces development is a very urgent task. It is associated with variety areas of science and technology. This also includes non-contact data entry for disabled users; behavior analysis of the human operator for the complex production according to his eye movements; psychological testing and research of mental faculties, and many others, where information can be obtained from the observation of the human pupils and then used [1].
Not by chance, in recent years this trend was paid greater attention. There are studies of such interfaces based on different principles.
For example, a well-known non-contact tracking systems "EC8™" [2], "Tobii T60XL" [3] and "EyeTech VT2" [4]. There are also systems such as "EagleEyes B" [5], where the principle of electrooculography is used, in which electrodes, attached to the user's head, analyze electromagnetic signals that are sent by the eye moves.
However, at present time the issue of building a non-contact interface remains open for several reasons. First of all, the existing systems are expensive and limited. Second, they use special hardware devices, which are produced in single units and cannot be recommended for mass use.
Based on the above, the purpose of this work is the development of hardware and software for the analysis of eye tracking without the use of special hardware devices.
2. Theoretical Principles of Gaze Tracking
In the current paper it is necessary to allocate some directions.
First, it is a direction associated with obtaining eye images. In the study, it was found that the best way of eyes video shooting is in infrared light. In this light practically all noise and reflections are eliminated, and also the sensitivity to illumination changes is significantly reduced.
Received 30th December, 2013.
The second direction concerns the analysis of the user's pupil's state. This work required the investigation of different methods of video flow pre-processing in order to obtain stable contours of the pupil and the brightest reflection on it. On the basis of the pupil movements and moves of the reflection some desirable human can be concluded. In this paper, the essential point is an unambiguous definition of the only reflection on the cornea. An infrared light diode was used IR illumination, which gave the only highlight on the eye cornea. Since the diode is stationary in space, changes of its relative position with the the pupil allows to calculate the gaze direction vector. Fig. 1 shows the relative position of the pupil and reflection according to different eye movements.
Figure 1. Relative position of the pupil and reflection during eye movements
left, right, up, down
In the process of image pre-processing the original video frame is smoothed with convolution with Gaussian kernel, then on the basis of morphological operations of erosion and dilation edge detection is performed, as shown in Fig. 2 [6-8]. To improve the accuracy of the pupil contour recognition the Hough transform is used, which allows to find circles and retrieved coordinates of reflections from the infrared diode. To reduce the error in the tracking results the coordinates from previous frames are stored in a queue, and then interpolated with coefficients according to normal distribution. Also the coordinates are normalized by the relative position of the head. As a result, the relative position of pupil and reflection is converted to the gaze direction vector, which is based on previously received data after calibration.
Figure 2. Video stream processing is parallelized in two directions. Pupils contour and reflection coordinates highlighting
The third stage of the research was an application development to control a computer on the basis of the derived coordinates of the reflection and the pupils' circles.
At this stage the input information is the coordinates of the circles centers and their radiuses (regardless to the area of interest - pupil or reflection), which are interpreted as the state of the eye: their position, blinking left or right eye and closed eyes, or the lack in the field of the camera point of view, as shown in Fig. 3.
Set of coordinates and radiuses
±
Approximation in accordance with the queue
Circles coordinates
Circles coordinates
Circles coordinates Circles coordinates
Computer control
Figure 3. The scheme of information processing in the third stage
computer control
Two options of the computer control in accordance with the developed semantics are possible:
— Keyboard control. The event of the lack of a circle within a certain time is detected and associated with a blink. When a blink event occurs operating system signal is sent to the commission of pressing the left or right arrow on the keyboard. With long-term absence of circles on the image first program in focus is closed. The program can run in the background, so you can control the focus of the third-party applications.
— Mouse control. An original interface was developed to control the mouse, which detects eyes movements. Eyes movements are interpreted as the cursor moves in a certain direction by a few pixels. Blink is detected in the same way as in
the previous version of interaction, but here it emulates clicking the left or right mouse button. Closed eyes are determined by the first method.
3. Features of the Application Implementation
The fundamental objective in creating this application was the realization of eye tracking algorithm at a speed of 30 frames per second. To speed up the work of the various methods special techniques based on the properties of convolution, morphological operations, etc. were used. With the implementation of the "greedy" algorithm processing speed was about 1.5 frames per second.
The program is implemented in Java 1.6, using standard Java-libraries for creating graphical user interface, video conversion into separate frames, as well as emulation of the keyboard and mouse control.
4. Results
As results of the investigation the following tasks were carried out:
1. The conditions for information input about the position and movement of the eyes in the infrared illumination were examined.
2. The algorithm for tracking pupils in multispectral illumination was implemented.
3. An application for computer control using eye movements was developed, and has two different control options: keystrokes, mouse cursor, and the left / right click emulation.
4. An analyze of an application behavior under certain conditions was conducted, as follows:
— lighting change;
— video camera relative location change;
— identification of users with different shapes and eye colors.
5. The possible directions for further research on eye tracking are proposed.
As a result of the study it is possible to consider the research on human-computer interaction with 3d objects — hand movements, and to turn off the computer — the disappearance of user in front of the camera. In addition, tracking can be used not only to control a computer, but also to interact with other technical means, for example, to connect the eye tracking with a wheelchair, so the people with limited motor function can move around controlling their chair with their eyes.
References
1. van Gompel R. P. G. Eye Movements: A Window on Mind and Brain. — NY: Elsevier Science, 2007.
2. Eye-Com Corporation. — http://eyecomcorp.com.
3. Tobii Eye Tracking Research. — 2013. — http://www.tobii.com/en/ eye-tracking-research/global/.
4. EyeTech Digital Systems. — 2013. — https://www.eyetechds.com.
5. EagleEyes The Opportunity Foundation of America. — 2013. — http://www. opportunityfoundationofamerica.org.
6. Pavlidis T. Algorithms for Graphics and Image Processing. — Berlin: Springer, 1982.
7. Ballard D. H., Brown C. M. Computer Vision. — New Jersey: Prentice-Hall, Inc., 1982.
8. Gonzalez R. C, Woods R. E. Digital Image Processing. — New Jersey: Prentice-Hall, Inc., 1992.
УДК 519.683.8:519.6
Система управления компьютером при помощи движения глаз
И. М. Гостев, Е. А. Сибирцева
Национальный исследовательский университет «Высшая школа экономики» ул. Мясницкая, д. 20, Москва, Россия, 101000
Изложена методология взаимодействия человека с ЭВМ, основанная на определении положения и направления движения глазного яблока. Для устранения помех и шумов от посторонних осветительных приборов используется инфракрасная подсветка, которая позволяет получить только один блик на роговице от инфракрасного светодиода. При этом входной видео-поток в инфракрасном диапазоне обрабатывается в реальном масштабе времени в двух параллельных процессах. Один используется для определения положения блика от светодиода на роговицах глаз, другой — для определения положения зрачка и направления его перемещения. Совокупность координат блика зрачка, положения роговицы и направления её перемещения позволяют дистанционно управлять компьютером. Для предотвращения потери «объекта» используется оригинальная методика накопления предыдущих положений глазного яблока и прогнозирование направления его перемещения. Данная методология бесконтактного управления компьютером позволяет имитировать нажатие на клавиши клавиатуры и движения мыши. Разработанная аппаратно-программная система слежения за направлением взгляда может быть использована как альтернативный способ ввода, наиболее приближённый к естественному взаимодействию человека с окружающей средой или как способ работы с компьютером для людей с ограниченной двигательной способностью.
Ключевые слова: трекинг движения глаз, некомандный интерфейс, человеко-машинное взаимодействие, обработка изображений, распознавание образов.
