CC BY
40
УДК 528.88
ИССЛЕДОВАНИЯ В ОБЛАСТИ ИНТЕРАКЦИИ ЧЕЛОВЕКА И КОМПЬЮТЕРА ДЛЯ СИСТЕМ ПОДДЕРЖКИ РЕШЕНИЙ В ЧРЕЗВЫЧАЙНЫХ СИТУАЦИЯХ
Евгений Левин
Мичиганский технологический университет, Институт технологии, 1400 Townsend drive, Хо-утон MI 49931, США, доктор наук, зав. кафедрой прикладной геодезии, сертифицированный фотограмметрист, тел. +1(906)487-24-46, e-mail: [email protected]
Анна Банашек
Варминско-Мазурский Университет, ul. Prowochenskiego 15, pok 204, 10-720 Ольштын
Польша, доктор наук, доцент, кафедра ресурсов недвижимости, тел. +48(89)523-43-96, e-mail: [email protected]
Джэссика МакКарти
НИИ Мичиган Тек, 3600 Green Court, Suite 100, Энн Арбор Мичиган 48105, США, доктор наук, старший научный сотрудник, тел. +1(734)994-72-36, e-mail: [email protected]
Александр Жарновский
Кошалинский университет Технологий, Польша, 75-453 Кошалин, ul. Sniadeckich 2,
доктор наук, профессор, кафедра геоинформатики, тел. +48(94)348-67-19, e-mail: [email protected]
В статье описываются текущие исследования по оптимизации взаимодействий человека с компьютером для применения в сценариях планирования и поддержки решений при чрезвычайных ситуациях. Целью исследований является проверка возможности использования недорогой системы слежения за движениями глаз с открытым программным обеспечением в процессах интерпретации геопространственных изображений. Описываемый в статье проект имеет также образовательный компонент.
Ключевые слова: геопространственные изображения, чрезвычайные ситуации, слежение за движениями глаз.
RESEARCH IN COGNITIVE HUMAN-COMPUTER INTERACTIONS FOR DECISION SUPPORT IN EMERGENCY SITUATIONS RESPONSE
Eugene Levin
Michigan Technological University, School of Technology, Program Chair Surveying Engineering, 1400 USA, Townsend Dr., Houghton, MI 49931, Doctor, Certificated Photogrammetrist, tel. +1 (906)487-24-46, e-mail: [email protected]
Anna Banaszek
University of Warmia and Mazury, Department of Real Estate Resources, 10-720 Olsztyn, Poland, ul. Prowochenskiego 15, pok 204, Associate Professor, tel. +48(89)523-43-96, e-mail: [email protected]
Jessica McCarty
Michigan Tech Research Institute (MTRI), USA, 3600 Green Court, Suite 100, Ann Arbor, MI 48105, Doctor, Research Scientist II, tel. +1-734-994-72-36, e-mail: [email protected]
Aleksander Zarnowski
Koszalin University of Technology, Department of Geoinformatics, ul. Sniadeckich 2, 75-453 Koszalin, Poland, Professor, Doctor, tel. +48(94)348-67-19, e-mail: [email protected]
This paper describes ongoing research efforts in optimization of human-computer interactions in geospatial imaging analysis for emergency situation awareness and planning support. Research is based on using low-cost open-source eye-tracking system in geospatial imagery interpretation. Described research activity has also educational component.
Key words: geospatial imagery, emergency situations, eye-tracking.
Global natural and anthropogenic disasters cause billions of dollars in infrastructure damages, unexpected disruption to socioeconomic activities and the tragic loss of human lives each year (Fritz et al. 2008). Remote sensing techniques and GIS and GNSS tools are frequently used in applications for disaster management in pre- and post-disaster activities. Pre-disaster applications are associated with mitigation and preparedness efforts. Mitigation refers to activities that reduce the vulnerability of societies to the impacts of a disaster, while preparedness refers to activities that facilitate preparation for responding to a disaster when it occurs (Mansourian et al. 2005). Post-disaster applications are associated with response and recovery efforts. Response is related to the immediate and short-term effects of a disaster, while recovery refers to activities that restore communities to pre-disaster conditions, such as reconstruction (Manfre et. al.2012, Mansourian et al. 2005). In spite of multiple efforts in automation of geospatial imagery interpretation and understanding there are still significant amounts of time and effort spent by human analysts on imagery interpretation. Thus nowadays operational workflow of geospatial imagery processing and interpretation can be identified as a "human-in-the-loop". Moreover, there are multiple efforts in deploying voluntary interpreted data (Voigt et al 2011) in emergency situations response. This novel direction is termed as a "crowdsourcing" and involves many voluntary analysts for whom fundamental geospatial education is replaced by quick on-site image interpretation training. Our research is challenged to explore potential use of novel HCI technology such as eye-tracking for the optimization of discussed above geospatial imagery processing in terms of productivity and accuracy.
While the human brain performs searches and analysis of visual data, the operator's eyes subconsciously scan the visual scene. Such eye movements are driven by and indirectly represent results of internal processes of visual searching and matching, performed by the whole human visual system. Tracking and analyzing eye movements potentially allows us to arrange a 'sight-speed' loop with the computer which should perform the rest of the tasks where computations and data storage are predominant. The task-specific use of gaze is best understood for reading text (O'Regan 1990) where the eyes fixate on almost every word, sometimes skipping over small function words. In addition, it is known that saccade size during reading is modulated according to the specific nature of the pattern recognition task at hand (Kowler and Anton 1987). Tasks requiring same/different judgments of complex patterns also elicit characteristic saccades (Dupont et al. 2014). The role of gaze has
been studied by Land and Furneaux (1997) in a variety of other vision-motor tasks such as driving, music reading, and playing ping pong. In each case, gaze was found to play a central functional role, closely linked to the ongoing task demands. In summary, these studies strongly suggest that gaze control and saccadic eye movements play a crucial role in mediating visual cognition, in addition to compensating for peripheral acuity limitations. It is well known from visual attention theory that the correlation between perception and eye-movement is eye-fixation (Yarbus 1967).
This paper is devoted to the research of developing a potential eye-driven image interpretation human-computer system through the performance of a simple task that an image analyst performs every day: manipulating a cursor towards target objects. Baseline control for comparison of eye-tracking-based cursor movement was regular mouse control of the cursor to the same set of targets.
Paul Fitts (Fitts 1954) proposed a metric to quantify the difficulty of a target selection task which nowadays is used by cognitive scientists as a law which is modeling a human psychomotor behavior. The metric was based on an information analogy, where the distance to the target (D) is like a signal and the tolerance or width of the target (W) is like noise. The metric is Fitts's index of difficulty (ID, in bits):
Practically applying Fitts's Law for the mouse or eye-driven targeting, the time to move and point to a target of width W at a distance A is a logarithmic function of the spatial relative error (A/W), according to MacKenzie and Buxton (1992):
where
• MT is the movement time
• a and b are empirically determined constants, that are device dependent.
• c is a constant of 0, 0.5 or 1
• A is the distance (or amplitude) of movement from start to target center
• W is the width of the target, which corresponds to accuracy
The term log2(2A/W + c) is called the index of difficulty (ID). It describes the difficulty of the motor tasks. 1/b is also called the index of performance (IP), and measures the information capacity of the human motor system. Thus comparative verification of numerical performance of the mouse versus eye targeting may give us an initial idea on estimating of eye-driven man machine interfaces efficiency.
One if the most interesting trends in eye-tracking technology is a fact that this technology made an evolution from the exceptionally expensive systems deployed in medical field to the inexpensive ubiquitous systems that are widely applied, for example in controlling a computer as/or communication aids by people with disabilities (COGAIN 2015). Specifically we deployed for our research an open source eye tracker, The Eye Tribe, which is available for under USD $100 (The Eye Tribe 2015). The Eye Tribe Tracker is an eye tracking system that can calculate the location of a
(1)
(2)
person's gaze by means of information extracted from the face and eyes. The eye gaze coordinates are calculated with respect to a screen the person is looking at, and are represented by a pair of (x, y) coordinates given on the screen coordinate system. A typical scenario is represented in Fig. 1(a).
Figure 1. a) The Eye Tribe System b) Calibration process screen (TheEyeTribe)
To compute (x,y) coordinates on the screen and transform from those coordinates to displayed image coordinates is typically performed calibration process as it is depicted on Figure 1(b). Any computer equipped with an eye tracker enables users to use their eye gaze as an input modality that can be combined with other input devices like mouse, keyboard, touch and gestures, which is referred to as active applications. We used the eye tracker as a mouse manipulator in the frame of the Fitts's law research described in section 3.1. The details of our research are listed in the following sections.
An experimental study was performed at both Michigan Technological University (USA; www.mtu.edu) and Koszalin University of Technology (Poland; www.tu.koszalin.pl/). Total of 10 participants included:
> 5 students majoring in Surveying Engineering at Michigan Tech
> 5 students majoring in Geodesy and Cartography at Koszalin U
For the experiments were used:
- 21" Displays with 1600x2000 c resolution;
- PCs with USB 3.0 port;
- The Eye Tribe Tracker.
Figure 2(a) depicts experimental setup in the US and Figure 2(b) in Poland, respectively. The US experimental setup was also equipped with chest holder to stabilize results of the eye-tracker calibration, as shown in Fig. 2(a).
i
Figure 2. Experimental setup: a) at Michigan Tech University and b) at Koszalin University of
Technology
For experimental software we used:
> TheEyeTribeSDK-0.9.41-x86 to calibrate the system tracking the eye movements for all experiment participants;
> FittsStudy research software developed by Central Washington University (Wobbrock et al 2011) for screen test-objects generation and analysis of the results with ISO 9241-9 standard compliance.
Each participant in the experiment carried out three successive operations:
> Calibration with post-calibration tests;
> Measurements of test-objects with cursor control by standard mouse;
> Measurements of test-objects with cursor controlled by eye-tracker.
The test objects were generated by FittsStudy accordingly to Test Options as is shown in Figure 3(a). Test parameters were consistently the same for all experiment participants. Figure 3(b) depicts a sample of circular test objects with 128 pixels diameter demonstrated randomly within 512 pixels radius circular test-field.
Figure 3. a) the experimental parameters setup and b) the test-object sample
Measurements results along with a timeline were recorded in XML format where screen coordinates are in display system units and time is in milliseconds. Sample of the raw experimental data are given in Table 1 below.
<move index="0" point- X=793 Y=217 Time=130
<move index-'!" point- X=789 Y=232 Time=350
<move index="2" point- X=786 Y=254 Time=360
The purpose of our experiment was to determine the time to move the cursor to a specified object with the mouse and by gaze control with eye-tracker. We have to recognize that when using the mouse, test participants were already exposed to this process with some prior experience. Even for non-active computer user, during the day a mouse is engaged hundreds of times, and for active users this is likely thousands of times. However, the object measurements by means of eye tracker was the first time that all participants in the experiment used this technology and without prior training. Therefore, it can be assumed that the direct use of these experiments does not allow a fully holistic evaluation of system cursor control by gaze.
For each test we performed statistical processing, including minimal, maximal and a median time for measurement of test object by cursor. The results of the experimental data processing are shown in tabular (Table 2) and graphical (Fig. 4) forms. Specifically Figure 4(a) depicts statistics for experiments at Michigan Tech University and Figure 4(b) for Koszalin University of Technology, respectively.
Table 2. Statistical processing results comparing time for manipulating mouse cursor via manual control (Mouse) and the eye tracker (Eye).
Time (MicroSec) Mouse Eye Mouse Eye Mouse Eye Mouse Eye Mouse Eye
Min 504 520 432 520 384 552 360 504 368 408
Max 1648 1720 877 3584 1109 3488 829 5232 726 1064
Median 1080 696 528 760 480 808 480 720 496 455
6000 5000 4000 3000 2000 1000 0
6000 5000 4000 3000 2000 1000 0
min max median
Figure 4. Statistical results of experimental participants for a) Michigan Tech University and b) Koszalin University of Technology using the manual mouse (m) and eye tracker (e) cursors.
Time ratio between mouse and eye tracker modes is an average of 1.21. Details are given for both US and Polish test subjects in Table 3 and in Fig. 5.
Table 3. Time ratio for cursor setup
Subject/Ratio # 1 # 2 # 3 # 4 # 5 average
K PL 0,6 1,4 1,7 1,5 0,9 1,24
K US 0,9 1,3 1,6 1,2 0,9 1,17
1,8
1,6 1,4 1,2 1 0,8 0,6 0,4 0,2 0
nr 1 nr 2 nr 3 nr 4 nr 5 average
Figure 5. Graphical representation of mouse/cursor time ratio; here nr# is a participant id;
coefficients K are computed as (TimeMouse/TimeEyeTracker); PL-states for Poland,
US - for the USA
It is visible from Figure 5 that experiment results are practically the same in the US and Poland groups. Practically 4 participants were working with the eye tracker faster than with the mouse. It is obvious that the eye tracker results depend on:
> Eye resolution (ophthalmology factor)
> Overall reaction ability of subject
> Sizes and forms of test objects (rectangle, strip, cross)
> Distance between eyes and display
> Subject motivations
> Training and practical use of the gaze-control method
For the partial elimination of the abovementioned factors, we will analyze test results of cursor movements in a mouse mode. Sample of cursor trajectory for the statistically averaged experiment is shown in Fig. 6.
Figure 6. Average trajectory of cursor movement by mouse; X and Y are directional displacements (in Pixels); horizontal graph axe is a Time in microseconds.
It is visible from Fig. 6 that the time taken to put the cursor on a target can be represented in 4 components:
t = ti + /"2 + 13 + 14 (3)
where
t - total time;
ti - target search time;
t2 - time of cursor movement to target which depends on cursor current position and selected target;
t3 - time for correction of cursor position on target and decision making; and t4 - time to click the mouse.
Per our experimental average results, these time components are: ti= 250-300 ms;
t2=350 ms for the 510 pixels in average distance between current and targeted cursor position;
t3 =300- 350 ms; and t4 = 150-200 ms.
Similar analysis can be performed also for the eye tracking mode of our experiment and is depicted in Fig.7.
Figure 7. Average trajectory of cursor movement for eye tracking method; X and Y are directional displacements (in Pixels); horizontal graph axe is a Time in microseconds.
Analogously, for the mouse mode we can also decompose a common time to perform a task into 4 stages:
t' = t1 +t2 +t3 +14 (4)
where
t' - common time;
t'i - target search and eye inertia time;
t'2 - time for movement to target which does not depend on distance between current cursor position and selected target;
t'3 - time for correction of cursor position on target and decision making; and t'4 - time to click a target using the mouse .
In average for the eye tracking mode this time components are: t'1 = 200-250 ms;
t'2 = 60-100 ms - for the 510 pixels in average distance between current and targeted cursor position;
t'3 = 350-400 ms; and t'4 = 150-200 ms.
1. Gaze controlled cursor movement to the target is almost independent from the distance to the on-screen target and occurs in 50-100 milliseconds. This speed is on average 3 to 6 times faster than the mouse controlled cursor movement.
2. When a subject is trained in the use of an eye tracker, then time t3 is significantly reduced. Subject also may gain experience and confidence in cursor positioning by gaze and thus become an advanced user of the eye-controlled environments.
3. The command to fix the cursor position on screen, for example by a doubleblink command, could significantly reduce the t4 time.
Our future research efforts will be devoted to the development of the novel hu-man-in-the-loop geospatial imaging environments which will increase the productivity of humans in visual analysis operations. Novel geospatial environments will doubtfully be efficient for emergency situation response application scenarios.
REFERENCES
1. (COGAIN)
http://wiki.cogain.org/index.php/Eye_Trackers#Eye_Trackers_for_Assistive_Technology_and_AA C Last accessed March 2015
2. (Dupont et al. 2014) Dupont, L., Antrop, M., & Van Eetvelde, V. (2014). Eye-tracking analysis in landscape perception research: Influence of photograph properties and landscape characteristics. Landscape Research, 39(4), 417-432.
3. (Fitts 1954) Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of experimental psychology, 47(6), 381.
4. (Fritz et.al. 2008) Fritz, H.M.; Okal, E.A. Socotra Island, Yemen: Field survey of the 2004 Indian Ocean tsunami. Nat. Hazards 2008, 46, 107-117.
5. (Kowler and Anton 1987) Kowler E. and S. Anton. Reading twisted text: Implications for the role of saccades. Vision Research, 27:45-60, 1987.
6. (Land and Furneaux 1997) Land, M. F., & Furneaux, S. (1997). The knowledge base of the oculomotor system. Philosophical Transactions of the Royal Society B: Biological Sciences, 352(1358), 1231-1239.
7. (MacKenzie and Buxton 1992) MacKenzie,I.S., and Buxton,W. (1992). Extending Fitts' law to two dimensional tasks. Proceedings of the CHI '92 Conference on Human Factors in Computing Systems. New York:ACM.
8. (Mansourian et al. 2005) Mansourian, A.; Rajabifard, A.; Valadan Zoej, M.J. SDI Conceptual Modeling for Disaster Management. In Proceedings of the ISPRS Workshop on Service and Application of Spatial Data Infrastructure, Hangzhou, China, 14-16 October 2005
9. (Manfre et. al.2012), L. A., Hirata, E., Silva, J. B., Shinohara, E. J., Giannotti, M. A., Larocca, A. P. C., & Quintanilha, J. A. (2012). An analysis of geospatial technologies for risk and natural disaster management. ISPRS International Journal of Geo-Information, 1(2), 166-185.
10. (O'Regan 1990) O'Regan, J.K. Eye movements and reading. In E. Kowler, editor, Eye Movements and Their Role in Visual and Cognitive Processes, pages 455-477. New York: Elsevier, 1990.
11. (TheEyeTribe) https://theeyetribe.com/ (last accessed on Mar 9 2015)
12. (Voigt et al 2011) Voigt, S., Schneiderhan, T., Twele, A., Gähler, M., Stein, E., & Mehl, H. (2011). Rapid damage assessment and situation mapping: learning from the 2010 Haiti earthquake. Photogrammetric Engineering and Remote Sensing, 77(9), 923-931.
13. (Wobbrock et al 2011) Wobbrock, J.O., Shinohara, K. and Jansen, A. The effects of task dimensionality, endpoint deviation, throughput calculation, and experiment design on pointing measures and models. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '11). Vancouver, British Columbia (May 7-12, 2011). New York: ACM Press, pp. 16391648.
14. (Yarbus 1967). Yarbus, A. Eye Movements and Vision. Plenum Press, New York, 1967.
© E. Levin, A. Banaszek, J. McCarty, A. Zarnowski, 2015
