CC BY
13
UDC 004.942 DOI: 10.20998/2411-0558.2016.21.08
V.D. PAVLENKO, dr. of techn. sciences, professor, ONPU, Odessa,
O.O. FOMIN, Cand. Tech. Sci., ONPU, Odessa,
A.N. FEDOROVA, MSC, ONPU, Odessa,
M.M. DOMBROVSKYI, PhD student, ONPU, Odessa
IDENTIFICATION OF HUMAN EYE-MOTOR SYSTEM BASE
ON VOLTERRA MODEL
A new method of constructing nonparametric dynamic model of the human oculomotor system on the basis of experimental data "input-output". This takes into account nonlinear and inertial properties of the eye of the rectus muscles. A technology for tracking eye movement based on the videos. It is possible to determine the dynamic characteristics of the oculomotor system functions as a transition of the first and second order - integral transforms Volterra kernels. Refs: 14 titles.
Keywords: oculomotor apparatus, modeling, nonparametric dynamic models, Volterra kernels, multidimensional transient characteristics, eye-tracking technology.
Statement of the problem. Control technology based on tracing eye movements (eye-tracking) are starting to get more and more widespread [1, 2]. Traditional areas of implementation such decisions - is a medical (ophthalmology) diagnostics and vision correction [3, 4], building interfaces in information systems, control of complex technical objects, the process of physical training in the sport, etc. Increased interest in such innovation technologies experiencing today and commercial sector.
However, most systems based on technology eye-tracking, for the successful operation require new methods of mathematical description of human eye-motor system (EMS) and special equipment for experimental research [5-7].
For successful solution problems of management, control and diagnostics of technical and medical applications need to have effective methods for identifying human EMS. Without having adequate mathematical model of EMS, taking into account the individual human properties, it is impossible to create modern applications with an expanded set of personalized features, such as medical and athletic trainers, authorized access to data, testing of human-machine systems, and more. Increase of the control objects complexity while maintaining the dynamic properties of systems, increased requirements for accuracy and objectivity of decisions leads to the problem of the development of new intelligent computing systems. These systems will ensure required characteristics and automate the monitoring process for objects of different physical nature. Modern diagnostic systems include both new mathematical techniques and modern resources of intelligent computing
© V.D. Pavlenko, O.O. Fomin, A.N. Fedorova, M.M. Dombrovskyi, 2016
[1, 2].
The paper considers the traditional approach of tracking the rotation angles of the pupil's eye (horizontal and vertical) using video registration. This involves using video cameras to obtain images of the pupil in the dynamics at regular intervals, which would be clearly fixed position pupil's eye when it moves [1].
By means of digital processing and analysis of the sequence of frames static im-ages and recognition the pupil position are restored to the coordinate's position pupil on the plane, namely, the values of the horizontal and vertical rotation angles of eye relative to the start position [2].
Significant disadvantage existing hardware, implementing this technology is fundamental impossibility of measuring dynamic and nonlinear characteristics of EMS, without that knowledge it's impossible to build an effective management system.
For elimination this disadvantage, the traditional structure of tracking system pupil's behavior using video recording in this work was further developed, which allowed not only perform static measuring the pupil's eye position, but also determining the dynamic characteristics of human visual system by the experimental observations "input-output" (identification problem) [8, 9].
Effectiveness of using modern methods of identification largely depends on the adequacy of mathematical models of real objects. As an information model of and natural objects considered as a "black box" used integral power Volterra series [10, 11] which in a compact form is characterized by nonlinear and inertial properties of the investigated object.
The purpose and research problems. The purpose of work is development method for constructing nonparametric dynamic model of eye-motor system, taking into account its inertial and nonlinear properties, based on experimental studies of "input-output" and also computational tools and software for the information technology processing experimental data.
To achieve this goal were set this following tasks:
- development methods for constructing nonlinear dynamic model of EMS as a Volterra kernels which characterizing both nonlinear and inertial properties of the nature objects;
- development information technology of obtaining experimental data for identification EMS based on pupil's movement tracking using video registration;
- development computational methods of identification multidimensional dynamic (transient) characteristics EMS using test inputs as a Heaviside functions of different amplitudes;
- verification constructed model EMS.
The Volterra model. Basis for creation of mathematical (informational) model of investigated object are the results of measurements of its input and output variables, and the solution of the problem associated with the identification of the experimental data and process them with the noise measurements.
To describe the objects of unknown structure appropriate to use the most universal nonlinear nonparametric dynamic models - Volterra model [10, 11]. The nonlinear and dynamic properties investigated object is uniquely described by a sequence of invariant with respect to the type of input signal is of multidimensional weight functions - Volterra kernels.
For continuous nonlinear dynamical system connection between the input x(t) and outputy(t) signals with zero initial conditions can be represented by a series of Volterra
X> t t t
У (t) = Z Уп (t) = j w (x)x(t - x)dx + J J W2 (ti, t2 )x(t - tj)x(t - t2 )d^ d%2 +
n=1 0 0 0
t t t (1)
+ j j j w3(tj, t2, t3) x(t — t) x(t - x2)x(t-t3)dx1 dx2 dx3 +...,
000
where Wn(T1,...,Tn) - Volterra kernel n-th order, function is symmetric with respect to real variables T1,...,Tn; yn(t) - the n-th partial component of response system (n-dimensional convolution integral); t - current time.
For nonlinear dynamical system multiple-input and multiple-output used multivariate Volterra series, which has the form:
V t V V t t
У J (t) = Z j WJ (t) Xh (t - x)dx + Z Z j j wjh (T1' T2) xi1 (t - T1) xi2 (t - T2)dx1 dx2 +
И 0 i1=1i2=1 0 0 ^
V V V t t t
+ Z Z Z j j j j (Tb T2'T3)Xi1 (t - T1)xi2 (t - T2)X3 (t - T3)dT1dT2dT3.
¡1=1 ¡2=1i3 =1 00 0
where y(t) - system response for the j-th output at the current time t for zero initial conditions; wJ . (xlv..,тп) - Volterra kernel n-th order in /1,/2, ...,in
inputs and j-th output (j = 1, 2, ..., д), the functions symmetric with respect to real variables T1, ..., Tn ; X1(t), ..., xv(t) - input signals; v, д - quantity of inputs and outputs, respectively.
In the context the problem stated above - identification EMS - need to use the model (2) for the mathematical description of the object [8]: two pair rectus muscles (input object) provide eye movement up and down, left and right, and various combinations (fig. 1); measured responses - the coordinates u(t) and v(t) current position the pupil relative to the initial position U0 and V0 (the outputs of the object). In this case in model (2) adopting v = 2 and д = 2.
Upper
rectus muscle Figure 1. Direct eye muscles
In this paper to simplify the experiment and data identification, problem solved for the case horizontal pupil's movement (v=1 and m=1), i.e. based on the model (1).
Problem identification (model constructing) as (1) or (2) consist to determine the Volterra kernels based on experimental data "input-output" EMS. Construction of the model is the selection of test actions x(t) and development of algorithm, which enables for the measured response y(t) allocate partial components yn(t) and determined on the basis of their Volterra kernels w«(n, ...,Tn), n = 1, 2, ... [12].
Computing method of multidimensional transient functions for identification EMS. Taking into account specificity investigated object to identification used test multistage signals. If test signal x(t) represents an identity function (Heaviside function) - 9(t), the result of identification the
transition function of the first order h1(t) and the diagonal section n-th order hn(t,...,t).
To determine the sections subdiagonal transition functions n-th order (n > 2) EMS tested using the n step test signal with given amplitude and different intervals between signals. With appropriate processing responses get subdiagonal section n-dimensional transition functions hn(t-T1,..., t-Tn), which
represent n-dimensional integral of Volterra kernel n-order w«(n,..tn):
& &
hn (t - t - t n ) = f ■■■ f wn (t - t1 -^1,-,t - t n ~K )d^\ d^n ■ (3)
0 0
Method for determination sections of n-dimensional transition functions based on the statement, proof of which is similar to that given in [15].
Statement. Let the test effect represents the sum of k (k = 1, 2, ..., n) step signals xf(t) = aQ(t-tf) (i = 1, 2, ..., k), with a time shift t on n, ..., ik,
then, for EMS with a single-input and single-output assessment section of the transient response of n-th order:
n
hn (t - ti,..., t - t n) = ^ Z (-1) i=1 y(t, 8i,-,5 n), (4)
n!a 51,..., Sn =0
where yn (t, 51,..., 5 n) - response EMS, measured in time t, under the action at her multistage signal with amplitude a, and if 5, = 1 (i = 1 ,2, ..., n), then test action comprises a stepped shift signal for ii, otherwise, with 5t = 0 - it does not contain.
For example, to determine transient function second order, first EMS tested stepped signals with a time shift for n and i2
X1 (t) = a0(t-T1) h X2(t) = a9(t-12), (5)
measured appropriate responses y1o(t) = y(t,1,0) and yo1(t) = y(t,0,1), respectively. Then, applied to the input EMS two-stage signal
x(t) = a0(t -i1) + a0(t -i2), (6)
and from the resulting response yn(t) = y(t,1,1) deducted responses to single step signals
y11(t) - y10(t) - y01(t) = - T1, t - T2(7)
From (7), after normalization should be
h2(t - T1, t - t 2) = [ yn(t) - y10 (t) - y01 (t)]. (8)
2a 2
With fixed values 11 and 12 assessment of the transient response second order h2(t-ij,t-i2) is a function of the variable t - section of the surface h2 (t1,t2 ) plane, passing at an angle of 45 degrees to the axes t1 and t2 and were shifted in axis t1 by the amount i0 =i1 -i2. Changing value io, get
different section h2(t, t -i0), on which you can restore all surface h2(t1,12). With i1 = i2 = o, get the diagonal section h2(t,t) .
Information technology of obtaining experimental data for the EMS identification. The experiment was implemented with the help of the proposed system for tracking the behaviour of the pupil based on video recording is performed in the following sequence.
1. Head of the observed person is located in front of the recording device (camera) at the known distance.
2. On fixed intervals, the display shows a graphic test signal in the form of a bright spot (light spot). At the same time turn on video camera to record eye movement from the initial position to a position determined by the light spot (test signal).
3. After passing a series test signals an experiment is terminated. File with video recording of pupil's movement is stored .in memory of the measuring system.
4. After finishing experiment, start the application, that realizing intelligent object detection algorithm (pupil) in the captured video. Plotted graph of changing position pupil of an input in the form of the test signal from the display (experiment "input-output").
5. Obtained data stored in the database and displays.
Software tools identification EMS. Developed software tools that perform automatic image recognition sites (pupil) in the sequence of frames of video recording and calculating coordinates. An important feature of this information technology is demanding of hardware. The experiment can be performed using a mobile phone equipped with a front-facing camera with a resolution of 5 Mpx, frequency of at least 30 frames per second and a processor with a clock speed of 1800 MHz for data processing (any modern smart phones).
In developing the software used the following information technologies: operating system for smartphones and tablet computers - Android 4; Library algorithms, computer vision, image processing and numerical algorithms for general purpose Open Source - library OpenCV (Open Source Computer Vision Library), which is ready to implementation under most existing operating systems (Android, Windows, Linux, iOS); graphics library Android open source - MPAndroidChart; Haar cascade and artificial neural network; programming language Java.
Training Haar detector. The main advantage of the detector of the Haar - speed. Thanks to the fast image processing, it becomes possible to process streaming video. Detector Haar implemented in the library OpenCV. The classifier is formed on the primitives of Haar by computing the characteristic values. For learning to the input of the first classifier is a set of "correct" images with pre-selected region of the image. Next, you iterate through entities and the calculation of the characteristic value. The computed values are saved in a file in format XML.
To search for the object in the image method of Viola-Jones. The method of Viola-Jones put: the integral image representation for the Haar features, building a classifier based on the adaptive busting algorithm and method for combining classifiers in a cascade structure. This allows you to search for the object in the image in real-time.
Results of identification - EMS transient function. Approbation tracking technology of the pupil's behavior based on video registration is performed on the task of analysis of work the oculomotor apparatus along the horizontal axis.
Where in the input (test) signal - distance from the base of the perpendicular, dropped from the center of pupil eye to the plane, in which is formed the perturbation - the light source to the point source (light spots) in the horizontal plane. Measure the response (the output) is a function of the current deviation of pupil in the frame image of the EMS from the starting point, depending on the time.
To determine the diagonal section of the transient response second order object is tested at first step signal with an amplitude of the a (horizontal distance to light spot from the starting point, represents the original position the pupil)
x1(t) = a0(t) (9)
and measured an appropriate response y1(t) . Then, with zero initial conditions applied to the input signal from the object with doubled amplitude a
X (t) = 2a0(t), (10)
and from thus obtained response y2 (t) deducted doubled response y1(t) . After normalization of this difference, we obtain
H2m. y2(t) - 2 ^) . (11)
2a 2
To determine the of the transient response of the first order, we obtain the expression
h1(t) ^ Ml . (12)
a
Measured response of the eye y1(t) and y2(t) to the input test signals a0(t) and 2a0(t) shown in Fig.2, respectively.
Obtained graphs of EMS transient functions first h (t) and second order h2 (t, t) shown in Fig. 3 and 4, respectively.
Comparing response of the constructed model
y (t) = ah1(t) + 2a 2 h2(t, t) (13)
with experimental data - response of object identification y(t) - shown in fig. 5.
o "S
/ /
//
//
y ♦ \ 111) -*-\2(t)
—» •
0.1 0.2 1X3 04 0.5 0.« 0.7 0.1 0.»
Time /, s
Figure 2. Object responses yi(t), J2(0
E E
c
O 2
u Q
/
/
/
/
r
0 0,1 0.2 0.» 0.4 0,5 0« 0,7 0.« 0.«
Time /. s
Figure 3. Transient functions first order k\ (t )
Figure 4. Transient functions second order h2 (t, t)
. J
ij
¡1
1
/ hit)
- 4v "•"V (t)
0 0.1 0,2 0,j 0.1 0.5 ojb 0.7 0.« 0.9
Time I, s
Figure 5. Comparison responses of the objecty(t) and model y(t)
Provided graphs are practically the same (standard deviation o = 0,31) which confirms effectiveness computational algorithm of identification and adequacy of the constructed model based on experimental data "input-output".
Conclusions. Proposed the method for constructing nonparametric dynamic models of human EMS taking into account its inertia and nonlinear properties based on the experimental data "input-output".
Has been further developed technology tracking the pupil's behavior with help of video recording that has allowed to determine eye's dynamic characteristics.
Proposed technology of tracking pupil's behavior does not need special equipment and laboratory experimental conditions and accessible for widespread use. Important feature of the technology is undemanding to the hardware, which opens up the possibility of its use in the modern mobile devices.
Verification of the developed model showed adequacy her under the investigation object - virtually identical (within acceptable error) responses of the object and model at the same impact test.
Obtained results identification of human's oculomotor system will be used in diagnostic tests as a source of primary data based on information technology, presented in [13, 14].
References:
1. Kepler, J., Linz, U. (2004), Biomechanical Modelling of the Human Eye. Netz Werk fur Forschung, Lehre und Praxis, Linz, 231 p.
2. Guestrin, E.D., Eizenman, M. (2006), "General Theory of Remote Gaze Estimation Using the Pupil Center and Corneal Reflections", IEEE Trans. Biomed. Eng., Vol. 53 (6), pp. 11241133.
3. Kopaeva, V.G. (ed.) (2012), Eye Diseases. Fundamentals of Ophthalmology, Medicine, Moscow, 552 p.
4. Shamshinova, A.M., Wolkov, V.V. (1999), Functional Methods of Research in Ophthalmology. GEOTAR-Media, 416 p.
5. Baziyan, B.H., Chigaleychik, L.A., Teslenko, E.L., Lachinova, D.R. (2007), "Using the Analysis of the Trajectories of Eye Movements, Head and Hands for Early Functional Diagnosis of Parkinson's Disease". Bulletin of Experimental Biology and Medicine, Vol. 143 (5), pp. 484-486.
6. Jansson, D., Medvedev, A. (2014), "Volterra Modeling of the Smooth Pursuit System with Application to Motor Symptoms Characterization in Parkinson's Disease". European Control Conference (ECC), pp. 1856-1861.
7. Westwick, D.T. (1995), Methods for the Identification of Multiple-Input Nonlinear Systems. Departments of Electrical Engineering and Biomedical Engineering. McGill University, Montreal, Quebec, Canada, pp. 192-232.
8. Doyle, F.J., Pearson, R.K., Ogunnaike, B.A. (2001), Identification and Control using Volterra Models. Published Springer Technology & Industrial Arts, 314 p.
9. Sidorov. D.N. (2013). Methods of Analysis of Integrated Dynamic Models: Theory and Applications. ISU, Irkutsk, 293 p.
10. Boyd, S., Jang, Y.S., Chua, L.O. (1983), "Measuring Volterra Kernels". IEEE Trans. on Circuits and Systems, Vol. 30 (8), pp. 571-578.
11. Pavlenko, S.V., Pavlenko, V.D., Polozhaenko, S.A. (2012), "Identification Methods Nonlinear Systems on Base Volterra Models using Testing by Poly Pulses". Herald of the National Technical University "KhPI". Subject issue: Information Science and Modelling, NTU "KhPI", Kharkov, No. 62 (968), pp. 155-161.
12. Pavlenko, V. (2009), "The Compensation Method for Identifying the Nonlinear Dynamic Systems in the Form of Volterra Kernels". Proc. of the Odessa Polytehnical University Press, No. 2 (32), pp. 121-129.
13. Pavlenko, V., Fomin, A. (2008), "Methods for Black-Box Diagnostics using Volterra Kernels". Proc. of the 2nd Int. Conf. on Inductive Modelling (ICIM'2008), Kyiv, Ukraine, pp. 104-107.
14. Pavlenko, V., Fomin, O., Ilyin, V. (2009), "Technology for Data Acquisition in Diagnosis Processes by Means of the Identification using Models Volterra". Proc. of the 5th IEEE Int. Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS'2009), Rende (Cosenza), Italy, pp. 327-332.
Поступила в редакцию 13.05.2016
Статью представил д-р техн. наук, проф. НТУ "ХПИ" Леонов С.Ю.
Pavlenko Vitaliy, Dr. Sci. Tech, Professor
Odessa National Polytechnical University
Ave. Shevchenko, 1, Odessa, Ukraine, 65044
Tel.: (048) 771-25-64, e-mail: pavlenko_vitalij@mail.ru
ORCID ID: 0000-0002-5655-4171
Fomin Alexander, PhD, Associate Professor
Odessa National Polytechnical University
Ave. Shevchenko, 1, Odessa, Ukraine, 65044
Tel.: (048) 705-83-79, e-mail: aleksandr.fomin@gmail.com
ORCID ID: 0000-0002-8816-0652
Fedorova Anna, Master, Engineer-Programmer Odessa National Polytechnical University Ave. Shevchenko, 1, Odessa, Ukraine, 65044 Tel.: (048) 705-83-79, e-mail: camomile763@gmail.com
Dombrovskyi Mikola, Postgraduate Odessa National Polytechnical University Ave. Shevchenko, 1, Odessa, Ukraine, 65044 Tel.: (048) 705-85-79, e-mail: dombrovskiin@mail.ru
УДК 004.942
1дентифжащя окоруховоТ системи людини на 0CH0Bi моделi Вольтерра / В.Д. Павленко, О.О. Фомш, Г.М. Федорова, М.М. Домбровський // Вюник НТУ "ХПТ. CepiH: 1нформатика та моделювання. - Харюв: НТУ "ХП1". - 2016. - № 21 (1193). - С. 74 - 85.
Пропонусться новий метод побудови непараметрично! динамiчноl модел1 окорухово! системи людини на основi даних експерименту "вхiд-вихiд". При цьому враховуються нелiнiйнi i iнерцiйнi властивосп прямих м'язiв ока. Розроблено технолопю вiдстеження руху ока на основi вiдеозапису. Це дозволило визначити динамiчнi характеристики окорухово! системи у вигляд1 перехiдних функцiй першого i другого порядюв - iнтегральних перетворень ядер Вольтерра. Бiблiогр.: 13 назв.
Ключевые слова: окорухова система, iдентифiкацiя, моделювання, непараметричш динамiчнi моделi, ядра Вольтерра, багатовимiрнi перехiднi функцп, технологiя вщстеження руху ока.
УДК 004.942
Идентификация глазодвигательной системы человека на основе модели Вольтерра / В.Д. Павленко, А.А. Фомин, А.Н. Федорова, Н.Н. Домбровский // Весник НТУ "ХПИ". Серия: Информатика и моделирование. - Харьков: НТУ "ХПИ". - 2016. - № 21 (1193). - С. 74 - 85.
Предлагается новый метод построения непараметрической динамической модели глазодвигательной системы человека на основе данных эксперимента "вход-выход". При этом учитываются нелинейные и инерционные свойства прямых мышц глаза. Разработана технология отслеживания движения глаза на основе видеозаписи. Это позволило определить динамические характеристики глазодвигательной системы в виде переходных функций первого и второго порядков - интегральных преобразований ядер Вольтерра. Библиогр.: 13 назв.
Ключевые слова: глазодвигательная система, идентификация, моделирование, непараметрические динамические модели, ядра Вольтерра, многомерные переходные функции, технология отслеживания движения глаза.
UDC 004.942
Identification of human eye-motor system base on Volterra model / V.D. Pavlenko, O.O. Fomin, A.N. Fedorova, M.M. Dombrovskyi // Herald of the National Technical University "KhPI". Subject issue: Information Science and Modelling. - Kharkov: NTU "KhPI". - 2016. - № 21 (1193). - P. 74 - 85.
A new method of constructing nonparametric dynamic model of the human oculomotor system on the basis of experimental data "input-output". This takes into account nonlinear and inertial properties of the eye of the rectus muscles. A technology for tracking eye movement based on the videos. It is possible to determine the dynamic characteristics of the oculomotor system functions as a transition of the first and second order - integral transforms Volterra kernels. Refs: 14 titles.
Keywords: oculomotor apparatus, modeling, nonparametric dynamic models, Volterra kernels, multidimensional transient characteristics, eye-tracking technology.
