CC BY
92
Psychology in Russia: State of the Art Volume 9, Issue 4, 2016
Lomonosov Moscow State University
NEUROPSYCHOLOGY
Psychophysiological methods for the diagnostics of human functional states: New approaches and perspectives
Alexander M. Chernorizov3*, Sergei A. Isaychev3, Yury P. Zinchenkoa, Irina A. Znamenskayab, Petr N. Zakharov b, Andrej V. Khakhalinb, Olga N. Gradoboevab, Vladimir V. Galatenkoc
a Faculty of Psychology, Lomonosov Moscow State University, Moscow, Russia
b Faculty of Physics, Lomonosov Moscow State University, Moscow, Russia
c Faculty of Mechanics and Mathematics, Lomonosov Moscow State University, Moscow, Russia
"•"Corresponding author. E-mail: [email protected]
L. S. Vygotsky in his famous methodological essay "The historical meaning of psychological crisis" (1928) emphasized the importance of studying any psychological process or state as a "whole" — that is, as characterized from the subjective and objective sides at the same time. This position is fully relevant for studying the human functional states (FSes). Today the objective psychophysiological diagnostics of human FSes in activities associated with a high risk of technological disasters (in nuclear-power plants, transportation, the chemical industry) are extremely relevant and socially important. This article reviews some new psychophysiological methods of FS assessment that are being developed in Russia and abroad and discusses different aspects of developing integral psy-chophysiological FS assessment. The emphasis is on distant methods of FS diagnostics: the bioradiolocation method, laser Doppler vibrometry, eye tracking, audio and video recordings, infrared thermography. The possibilities and limitations of the most popular emotion atlases — the Facial Affect Scoring Technique (FAST) and the Facial Action Coding System (FACS) — in developing distant visual-range and infrared-range systems for automated classification of facial expressions are analyzed. A special section of the article concentrates on the problem of constructing an integral psychophysiological FS index. Mathematical algorithms that provide a partition of FS indicators into different FS types are based on various methods of machine learning. We propose the vector approach for construction of complex estimations of the human FSes.
Keywords: functional states, distant diagnostics, integral estimating, vector approach
ISSN 2074-6857 (Print) / ISSN 2307-2202 (Online) © Lomonosov Moscow State University, 2016 © Russian Psychological Society, 2016 doi: 10.11621/pir.2016.0403 http://psychologyinrussia.com
Introduction
In general, functional state (FS) is defined as a state of the mental and physiological activity of a person against which one or another professional activity is implemented (Danilova, 1992; Leonova, 2007, 2009; Leonova & Kuznetsova, 2015). In physiology, FS is closely associated with the level of activation (excitation/inhibition) of the nerve centers. In accordance with this interpretation, FSes are considered "functional states of the brain" and are presented as values on a "sleep-wake" scale (Danilova, 1992). Psychologists added psychological factors to physiological FS determinants: for example, motivation, activity content, individual characteristics of mental processes and states (Leonova, 2007, 2009; Leonova & Kuznetsova, 2015). As a result the set of physiological FSes (sleep, wakefulness, fatigue, monotony, slumber) was completed with such psychophysiological FSes ("psychoe-motional states") as emotions, psychological stress, altered-consciousness states, anxiety, fears, depression.
The problem of objective (psychophysiological) human FS diagnostics in different modes of behavior associated with the high risk of technological disasters (in nuclear-power plants, transportation, the chemical industry) is extremely relevant and socially important. A critical factor in determining the efficiency of an activity is the "optimal (for this activity) human FS". Deviation from this factor leads to serious failures (Chernobyl, Fukushima-1). The European Union spends tens of millions of euros for development of this urgent topic (for example, the international project "Advanced sensor development for attention, stress, vigilance, and sleep/ wakefulness monitoring"). Starting in 2004 within NATO scientific and technological programs Operator Functional State Assessments have been performed. Applied studies related to the development of objective and distant FS identification methods in real-time mode are of particular interest. In the struggle against terrorism the development of optical-electronic equipment and software for identifying FSes based on individual behavioral characteristics and appearance turns out to be a task for fundamental and applied psychophysiology.
Bioradiolocation of FS indicators
Bioradiolocation is held to create new distant methods for human FS identification on the basis of such physiological parameters as heart rate and breathing, which are usually controlled by contact sensors. Four groups of biomechanical movements generate the signals recorded by radar methods: (1) contraction of heart muscle (of 0.8-2.5 Hz, thorax oscillation amplitude is 0.1 mm); (2) vibrations of thorax during breathing (0.2-0.5 Hz, oscillation amplitude is 0.5-1.5 cm); (3) the movement of organs of speech (the basic tone oscillation frequency of vocal cords equals about 100 Hz), and (4) movements of head, lips, hands, and feet (Bugaev, Ivashev, & Immoreev, 2010). Ultra-wideband (UWB) radar) for remote measuring of cardiac activity and respiration has been developed and tested (Immoreev, Isaychev, Samkov, & Pavlov, 2010). Radar belongs to the class of ultra-wideband when the impulse length in space (impulse duration x speed of light) becomes comparable to or less than the spatial extent of the observed object. UWB radar significantly increases resolution and the accuracy of the measurement of the distance to the monitoring object, decreases the radar "dead zone", increases resistance to all types
of passive interference, and simplifies monitoring of a moving target against strong reflections from stationary objects in the background. To evaluate the accuracy of the measurements obtained by using UWB radar and the quality of playback information about cardiac activity, a special study comparing the signals captured by UWB radar with the signals of an electrocardiograph was carried out (Immo-reev et al., 2010). Value distributions of instantaneous (from beat to beat) changes in the period of cardiac contractions were compared. The correlation coefficient between these two measurement methods was ~ 0.91, and the average measurement error was less than 3%. Results of these experiments confirm the possibility of using UWB radar for monitoring human cardiac activity and respiration rate. Such properties of UWB radar as noncontact, remoteness of measurements, device portability, easy maintenance and handling make this method promising and reliable for use in systems for remote FS diagnostics.
Laser Doppler vibrometry
Laser vibrometry is designed for remote measurement of vibration velocity (vi-brooscillations) of the test object or a part of it in the range from 0.01 to 50 mm/s on frequencies of vibration from 80 Hz to 11 kHz. The working distance for the measurements between the laser vibrometer and the object ranges from 1.5 to 10 m and more. The first domestic portable laser vibrometer with increased sensitivity was developed by JSC FSPC NSRP "Quartz" (http://www.kvarz.com/general/ aboutR.html). The working principle of the laser vibrometer is the Doppler shift of radiation of optical (laser) frequency reflected from a moving object. In fact, laser Doppler vibrometry (LDV) is close to widely used methods in human psychophysiological studies such as accelerometry and plethysmography, which in contact application convert signals about changes in skin state to information about the state of the physiological processes that caused these changes. Use of LDV allows one to register a greater number of physiological parameters than does radar. In particular, by using LDV it is possible to measure muscle tension, heart rate, heart tones, and various respiratory parameters. With LDV it is possible to register muscle tension in the form of an "acoustic myogram", which correlates well with regular EMG and can be used to determine patterns of the activity of facial mimic muscles corresponding to the expression of different emotions. It is important to note that LDV signals are weakly influenced by various noises and other external conditions.
Oculomotor activity and pupillary response
Methods for eye-movement detection have been effectively used in experimental psychology for over half a century. Eye-tracking units operate on the principle of videooculography, the essence of which is video registration of pupil shifts and glints arising on the cornea because of infrared radiation directed into the eye. Glint, formed on the cornea, is recognized by video camera as a bright spot, and the pupil is detected as a black one. Moreover, the position of an infrared backlight is analyzed, and thus it becomes possible to determine the orientation of the eyeball optical axis. The main task of eye tracking is the real-time (with a frequency of 250 Hz) registration and transfer of data about eye movements. Eight types of human
eye movements are known; of these fixations and saccades are of the greatest interest to psychophysiologists. Eye fixations are the complex of micromovements in a small spatial area consisting of microsaccades (fast jumps with angular speed up to 200 °/s), drift (slow movements with speed below 0.5 °/s), and tremor (quick eyeball trembling at a rate of 30-80 c/s) (Barabanschikov & Milad, 1994; Barlow, 1952). Fixation (as a certain part of the drift) is the drift — that is, the slow and smooth eye movement — in a small spatial area (Barabanschikov & Milad, 1994). It is considered that during fixation receipt the processing of visual information occurs. Saccades are sudden eye leaps that change the eye position and, as a consequence, change the location of the focus of visual attention ("the zone of clear vision"). Oculomotor activity is determined by the current task, and study of its dynamics opens up the possibility for the objective study of cognitive-perception processes (image construction). Initially, the main task of eye tracking was to study the psychological patterns of visual perception (Klingner, Kumar, & Hanrahan, 2008). Numerous studies have shown that eye movements are critically related to many aspects of visual perception (Wade & Tatler, 2005; Yarbus, 1967), attention (Hoffman & Subramaniam, 1995), and different types of psychological disorders (Kovalev, Kli-mova, & Menshikova, 2016; Menshikova, Kovalev, Klimova, & Chernorizov, 2015; Shapiro, 1989).
The reflex reaction of the constriction/extension of pupil diameter (RPD) can be used as one of the indicators of distantly registered electrooculography for FS diagnostics. Anatomically and functionally the RPD is under the control of antagonistic effects of sympathetic and parasympathetic autonomic-nervous-system divisions. The RPD is affected by a number of factors related to the influence of external stimulation (light, contrast, duration) as well as to individual characteristics of the response subject (emotionality, cognitive processes, fatigue) (Fukuda, Stern, Brown, & Russo, 2005). According to some researchers, RPD can also be effectively used for stress diagnostics in lie detection (Heaver & Hutton, 2011). Currently a method of complex FS diagnostics based on a combination of eye tracking with EEG-analysis data is being developed.
Diagnostics of psychoemotional states according to facial expression
Facial expressions are an important source of information about human emotions, intentions, and affective states. In this regard, their analysis is increasingly being used in brain-computer interfaces along with analysis of audio signals and body movement (postures, gestures) (Ekman, 1999).
Emotion atlases FAST and FACS
Based on studies of the expression and perception of human emotions carried out using the method of visual analysis (classification) and the method of registration and analysis of mimic-muscle activity, P. Ekman and colleagues developed two human "atlases of emotions" — the Facial Affect Scoring Technique (FAST) and the Facial Action Coding System (FACS) (Ekman, Friesen, & Hager, 2002; Ekman, Friesen, & Tomkins, 1971). In the FAST atlas sketches of emotional expressions for
three levels of the face (eyebrows and forehead, eyes and lids, lower part of face) were presented for different gaze directions and head orientations. The FACS atlas is a complete description of the activity patterns of certain facial muscles, made on the basis of individual facial muscles' registration of electrical activity during the expression of different emotions. In the FACS atlas nearly 10,000 patterns of facial muscle activity are presented, and the relation of these patterns to different emotional expressions is established (Ekman et al., 2002). Thus, according to the FACS atlas, muscular correlates of the sign and the intensity of expressed (experienced) emotion are a specific combination (pattern) of the activity of facial muscles and the magnitude of this activity, respectively. FAST and FACS atlases are widely used for studying emotions in fundamental psychology, for training profilers in applied psychology (Turvey, 2012), and for developing systems for automated facial-expression classification and automated affect recognition in artificial-intelligence programs (as described in the following two sections).
Visible-range systems for automated facial-expression classification (AFEC) and systems for automated affect recognition (AAR)
Technical intelligence systems with elements of automated facial-expression classification (AFEC) and automated affect recognition (AAR) can be effectively applied not only in artificial intelligence but also in modern detectors of hidden knowledge based on analysis of thermal signals from facial muscles (Pavlidis, 2004). In the field of recognition of faces and emotional facial expressions there are two main approaches: visual and thermal. The visual approach analyzes the usual video sequence; the thermal approach analyzes video obtained from thermal imaging cameras that are able to capture infrared radiation from the human face.
Most modern AFEC devices are based on the analysis of video data (visual-based AFEC, vAFEC) (Abidi, Huq, & Abidi, 2004). The reliability of these systems in emotion recognition based on visual face images reaches 70%. The theoretical basis for such systems and the mathematical algorithms and technical components used to create them are described in reviews by Pantic and Rothkrantz (2000). Despite their relatively high accuracy in emotion recognition in laboratory conditions, AFEC systems of this type are inefficient in field conditions for the following reasons (Fasel & Luettin, 2003): the impact of changes in the intensity of background light, the dependence of the accuracy of identification on pose, and the accuracy of the face-structure model used in the algorithm for emotion recognition. The limitations of visually oriented AFEC and AAR systems in real conditions were also not removed when they were used as a reference database for recognition of faces from the FACS atlas. All these difficulties are triggers for the development of AFEC systems based on other types of (nonvisual) information (nonvisual-based AFEC, nvAFEC) or based on a combination of visual and nonvisual signals (for example, thermovision and/or acoustic signals) (Christie & Friedman, 2004). In most vAFEC visual evaluation of facial muscles, movement is used. All types of observed movements of facial muscles are usually divided into four groups: static (tonic) signals, reflecting the stable properties of the activity of facial muscles; gradual (slow) changes in muscle activity that contribute to overall facial expression; artificial signals (artifacts) associated with the peculiarities of hair covering,
glasses, and jewelry; fast signals, reflecting changes in neuromuscular facial activity (clenching of jaws, opening of mouth). In the FACS atlas fast signals are the basis for encoding emotions.
In addition to the FACS atlas two other methods are used in systems of emotion recognition based on facial expressions: optical flow analysis and the 3D wireframe face model. Optical flow analysis (OFA) is based on measurement of local changes in "face-image" brightness arising from the activity of facial muscles (De Carlo & Metaxas, 2000). The accuracy of this method for classification of emotions varies in rigidly controlled test conditions from 80% to 98%. The 3D wireframe face model is more accurate than the OFA method but requires large calculations (Gur et al., 2002).
Common drawbacks of visual methods for detecting facial expression are low efficiency in conditions of weak and/or uneven illumination and the dependence of expression-identification reliability of the same person on shooting angle.
Infrared-range systems for automated facial-expression classification (AFEC) and systems for automated affect recognition (AAR)
Infrared thermography is a method of remote visualization and registration of object thermal fields within the range of electromagnetic waves of infrared radiation from 770 nm to 1000 microns. Thermography is a powerful research tool used in almost all fields of natural science — medicine, geology, biology, energy saving, nondestructive testing (Vavilov, 2013). Increased interest in thermography is connected with the appearance of a new generation of thermal imagers and modern methods of digital processing, analysis, and storage of thermographic images.
One of the most important applications of thermography is noncontact registration of psychophysiological parameters through systems of distant detection and identification of human appearance and behavior features in situations of anxiety, increased emotional excitement, and stress. Human skin has high emissivity, close to absolute blackbody. Therefore, a change in its temperature leads to a significant change in the power of emitted infrared radiation. Low reflectance of skin minimizes the environmental influence on the detection of skin temperature. In thermographic imaging low reflectance allows accurate capturing of local temperature changes.
Cardone, Pinti, and Merla (2015) review the main achievements of thermography in human FS monitoring. They also provide strong evidence of the ability to use infrared images as the basis for a quantitative estimate of such autonomic-ner-vous-system activity parameters as local blood perfusion, heart rate, and respiratory rate. Pavlidis & Levine (2001) show that the level of perfusion in the eye-socket region allows the registration of small temperature changes associated with human FS. Puri, Olson, Pavlidis, Levine, and Starren (2005) discovered that the rectangular forehead area containing central vessels is the most informative for stress diagnostics. A number of studies suggest that one of the most informative methods for human FS evaluation is remote thermographic analysis of breathing flows. For example, Lewis, Gatto, and Porges (2011) provide experimental data showing that infrared thermography allows sufficiently accurate evaluation of breathing rhythm and relative tidal volume.
Study of facial thermal radiation by infrared cameras (thermal imagers) to develop infrared-range AFEC and AAR is a new and actively developing area at the intersection of fundamental science and practice (Jarlier et al., 2011; Kong, Heo, Abidi, Paik, & Abidi, 2005). This method is the most promising and rapidly developing technique for the distant detection of emotions. Temperature mapping has a high temporal resolution and allows identification of face features and expressions in low-light conditions. Generally, a thermovisor is used in combination with a regular video camera. The facial image in the visible range has more pronounced details that allow more accurate determination of interesting facial areas on the image; it compensates for possible distortions, such as head rotation and mimic distortions, and allows more accurate estimation of the distortion parameters. Such imaging requires the performance of two tasks: (1) searching for a face in the image according to basic contours and points (eyes, nose); (2) describing face parameters (eye contours, eyelids, eyebrows, lips, nose wings) in detail. Special biometric software has been developed to perform these tasks. Detailed evaluation of face rotation and detection of biometric (mimic) control points on the face are done using Active Appearance Models (AAM) (Leinhart & Maydt, 2002) and Active Shape Models (ASM) (Cristinacce & Cootes, 2007). Combining data from AAM/ASM models with a thermal image allows more accurate determination of the location of the desired temperature control points on the thermal image (Wang, He, Wu, He, & Ji, 2014). These methods also allow evaluation of human facial expressions, and the information can be used to assess emotional states (Huang & Ren, 2013). Use of these methods on a video sequence allows one to track control-point trajectories in space and to evaluate dynamic characteristics of facial expressions.
In experiments on rats (Vianna & Carrive, 2005), monkeys (Nakayama, Goto, Kuraoka, & Nakamura, 2005), and humans (Puri et al., 2005) the possibility of using facial thermal mapping for emotion and stress diagnostics has been demonstrated.
Avinash, Buddharaju, Pienta, & Pavlidis (2012) compared the effectiveness of methods for recognizing facial features and expressions using data from infrared video, ordinary video, or both on the basis of the FACS atlas recognition system. The comparison was based on an analysis of 13 facial regions of interest (ROIs) that are critically important for the recognition of basic emotions (Ekman et al., 2002). For each ROI the average value of the parameters was extracted, and then principal-component analysis was used to identify the degree of deviation of each of the 13 regions from neutral facial expression. It turned out that under different levels of illumination but constant temperature the thermal method was better than the visual method, as was expected. Under conditions of constant illumination and variable temperature (a fan with heated air of varying intensity was directed on the face of the subject) the visual method showed greater efficacy, whereas the performance of the thermal method, contrary to expectations, remained virtually unchanged. The authors note that if we consider the visual and the thermal methods separately, not in combination, temperature mapping is preferable. However, for the most effective identification of facial features and emotional expressions the combination of thermal and visual methods is optimal (Avinash et al., 2012).
A modern cross-cultural research challenge for the universality of emotional communication: Does culture shape how we look at faces? Fundamental cross-cultural studies of emotional expression among representatives of more than 20 countries of East and West showed that facial expressions of basic emotions do not depend on culture and are universal for humans (Ekman, 1999). Studies of facial emotional expressions among chimpanzees have confirmed the ideas of Charles Darwin and P. Ekman about the evolution of emotional communication in primates and served as the basis for the creation of the Chimpanzee Facial Action Coding System (ChimpFACS) (Parr, Waller, Vick, & Bard, 2007). The ChimpFACS is an objective, standardized observational tool for measuring for facial movements in chimpanzees based on FACS (Ekman et al., 2002). This tool enables direct structural comparisons of facial expressions between humans and chimpanzees based on their common underlying musculature. The authors provided data on the application of the ChimpFACS to validate existing categories of chimpanzee facial expressions using discriminant-functions analyses. As a result, the authors suggested a potential homology between these prototypical chimpanzee expressions and human expressions based on musculature.
However, works have now appeared the results of which indicate cross-cultural differences in ways of expression and the perception of emotions (Engelmann & Pogosyan, 2013; Park, Barash, Fink, & Cha, 2013). Such cultural distinctions may lead to missed cues or misinterpreted signals about emotions during cross-cultural communications. These data challenge the common theory that facial expressions are a hard-wired human behavior with solely evolutionary origins and therefore facial expressions do not differ across cultures. These data on the cultural influence on expression and perception are not doubted. The only question debated is how these influences apply to the so-called basic emotions (Ekman, 1999). The answer to this question is important for understanding whether (and how) to consider the impact of cultural differences on emotional communication in emotion atlases (FAST, FACS) and systems of automated emotion recognition (AFEC, AAR).
Diagnostics of human psychoemotional states using speech analysis
A number of psychophysiological data show that each human emotion has its specific sound expression, which is manifested in the change of so-called vocal utterance — tone, rhythm (tempo), and timbre (Potapova, Potapov, Lebedeva, & Agibalova, 2015). Psychologists and psycholinguists suppose that about 90% of emotional communication occurs on the nonverbal level. For example, although speech is perceived regardless of its content, we can discover the emotional state of the person (for example, melancholy or anxiety) based on such indicators as the rate and average duration of pure speech, the length of pauses, the ratio of pause time to total time of utterance, the speed of articulation, hesitations, the character of respiratory movements, voice pulsation related to blood flow, changes in the main tone of voice, and the analysis of vibrations of the vocal-apparatus muscles (Bachrowski & Owren, 2003; Lebedeva & Karimova, 2014; Potapova et al., 2015). The restriction of voice analyzers of emotions is a requirement for well manifested (clearly, loudly) verbal expressions; this requirement limits application of this method in field conditions.
Hardware-software complexes for distant registration of fs parameters
Currently, the market for commercial services for distant FS diagnostics presents a limited number of choices. The most famous is a product of Siemens VDO, the Driver Attention System (DAS). Based on a driver's facial expression and eye movements, a DAS camera placed on the dashboard monitors FSes (on the scale sleep-wake) during car movement and, in case of deviations from the norm, it beeps to the driver (http://techfreep.com/automatically-adjust-to-speed-limits-with-sie-mens-car-camera.htm).
In Russia, scientific and applied research and development of systems similar to DAS are only beginning. One of them, VibraMed, is a system for the analysis of human psychophysiological and emotional states ; it was developed by Elsis (in St. Petersburg, Russia) (Minkin, 2007). The system is based on the registration of the frequency and amplitude of human head micromovements ("vibroimages of human head"). Vibroimaging registers micromovement and spatial oscillations of the object by determining the vibration parameters (frequency and amplitude) for each element (pixel) of the test image. The theoretical basis of the method is that the vestibular system is linked to all other functional systems of the human body and responds promptly to any mechanical, level-of-pain, or emotional change. According to the author, head-vibration parameters (frequency range 0.1-10.0 Hz and amplitude in the range of 10-1000 ^m) for the steady emotional state of a person are stable over time. Vibration parameters fluctuate only after a change in emotional state. At this time the link of emotions to head micromove-ments cannot be considered to be sufficiently studied and unambiguous. For example, the VibraMed program promises quantitative evaluation of such human qualities as health, extraversion, and anxiety. However, these characteristics are intrinsically integral and reflect the complex interaction of multiple physiological and psychological processes that cannot be tested using only one physiological index.
Developments in the integral psychophysiological evaluation of Fses: challenges, approaches
Existing contact and distant FS monitoring technologies face the following major problems: (1) the difficulty of taking individual differences into account; (2) the variability of psychophysiological characteristics corresponding to one FS type, which lies within its "corridor" around an unknown centroid corresponding to the characteristics typical for it; (3) developing and adjusting reliable mathematical data-processing algorithms for providing accurate classification of FSes in real time; (4) the forecasting and predicting FS changes in the release of psychophysiological characteristics beyond the limits acceptable for the given type of FS.
In order to solve these problems we propose a vector approach based on the presentation of multiple psychophysiological characteristics (FS indicators) as a vector, the n dimension of which is determined by a number of psychological (tests) and physiological (heart rate, galvanic skin response, EEG) indicators (Lebedev, Isaychev, Chernorizov, & Zinchenko, 2013). Vector elements can be both standard "first-level" characteristics and characteristics obtained from less standard signal processing (for example, coefficients of EEG signal decompo-
sition). A set of such vectors forms the n-dimensional space of FS indicators, where each FS type has its own spatial area formed by a set of vectors corresponding to its type. An FS-indicator space-location characteristic (for example, metric) is introduced and is formulated based on its algorithm, which determines FS classification by correlating measured FS indicators, represented by the corresponding n-dimensional vector, with one or another space area. Such a model of integral FS evaluation opens the following possibilities for solving FS monitoring problems:
(1) Spatial areas can be determined for each person separately according to individual characteristics; this procedure eliminates the problem of individual differences.
(2) FS localization in separate spatial areas allows determination of the cen-troid for each FS type; the centroid reflects both the indicator values typical for the given FS type and the oscillation limits beyond which a transition to another FS type occurs.
(3) The trajectory of vector displacement in an FS indicator's space allows estimation of the sustainability level of the FS type and forecasts of the direction and the probability of change of the FS type.
(4) Multivariate analysis (multidimensional scaling, factor analysis) can provide a reasonable reduction in the number of registered signals and FS indicators without reducing the reliability of the determination of FS type.
(5) The accumulation of training samples refines the true partitioning of an FS indicator's space to areas corresponding to given FS types; this procedure results in the increased accuracy of the determination of FS type.
(6) The geometric model of FS space allows the visualization of results in the context of psychological and psychophysiological training, which can be used in developing skills for FS self-regulation.
The high efficiency of the vector approach was successfully demonstrated in the field of sensory and cognitive psychophysiology (Sokolov, 2013), as well as in human FS diagnostics (Lebedev et al., 2013).
Algorithms that provide a partition of an FS indicator's space into areas corresponding to different FS types may be based on various methods of machine learning: for example, clustering methods, neural networks, the method of support vectors, decision (random) trees, random forests. We have developed a specialized method for the automated construction of classifier functions for binary FS classification; it provides reliable classification of stress vs. state-of-calm wakeful-ness (also referred to as a normal FS) despite the relatively small size of learning samples (Galatenko, Livshitz, Podolskii, Chernorizov, & Zinchenko, 2012). Along with an efficient procedure for dimension reduction that utilizes discrete wavelet transform, this method includes an individual tuning stage that allows the inclusion of certain information on individual peculiarities in the classification. This stage additionally requires only a short learning sample for an individual, which significantly increases the reliability of the FS classification (Galatenko et al., 2013). The significance of an individual tuning stage has also been observed for a completely different method of FS classification based on the integrated analysis of pe-
ripheral physiological parameters (Lobacheva et al., 2013). These results provide a basis for the conclusion that using an individual tuning stage, which is not standard for classical machine-learning methods, is natural and efficient when mathematical methods for FS classification are employed.
conclusion
In full accordance with the ideas of L.S. Vygotsky, the main trends in the development of modern approaches to FS diagnostics concentrate on the integration of psychology and natural science (neuroscience, mathematics). These trends are connected with the construction of complex psychophysiological expert systems for FS diagnostics based on multidimensional analysis in real time of both psychological and physiological data. From our point of view, the vector metric model for the representation of FS parameters, combined with methods of multivariate analysis, is the most promising tool for building an integral process for the evaluation and classification of FS.
Acknowledgments
This study was supported by a grant from the Russian Science Foundation (project № 16-18-00080); equipment purchased by the Program of Development of Lo-monosov Moscow State University up to the year 2020 was used.
References
Abidi, B., Huq, S., & Abidi, M. (2004). Fusion of visual, thermal, and range as a solution to illumination and pose restrictions in face recognition. In Proceedings. IEEE 38th annual 2004 International Carnahan Conference on Security Technology (pp. 325-330). New York: IEEE. doi: 10.1109/ccst.2004.1405413 Avinash, W., Buddharaju, P., Pienta, R., & Pavlidis, I. (2012). A comparative analysis of thermal and visual modalities for automated facial expression recognition. Advances in Visual Computing, 7432, 51-60.
Bachrowski, J.-A., & Owren, M. J. (2003). Sounds of emotions: Production and perception of affect-related vocal acoustics. Annals of the New York Academy of Sciences, 1000, 244-265. doi: 10.1196/annals.1280.012 Barabanschikov, V., & Milad, M. (1994). Metody okulografii v issledovanii poznavatelnyh pro-tsessov i deyatelnosti [Methods of oculography in the research of cognitive processes and activity]. Moscow: Institute of Psychology at RAN. Barlow, H. B. (1952). Eye movements during fixation. Journal of Physiology (London), 116, 290306. doi: 10.1113/jphysiol.1952.sp004706 Bugaev, A.S., Ivashev, S.I., Immoreev, I.Ya. (2010). Bioradiolokatsiya [Bioradiolokation]. Moscow: Bauman MGTU.
Cardone, D., Pinti, P., & Merla, A. (2015). Thermal infrared imaging-based computational psy-chophysiology for psychometrics. Computational and Mathematical Methods in Medicine, Article ID 984353, 1-8. doi: 10.1155/2015/984353 Christie, I. C., & Friedman, B. H. (2004). Autonomic specificity of discrete emotion and dimensions of affective space: A multivariate approach. International Journal of Psychophysiology, 51, 143-153. doi: 10.1016/j.ijpsycho.2003.08.002
Cristinacce, D., & Cootes, T. F. (2007). Boosted regression active shape models. Proceedings of
the British Machine Vision Conference, 2, 880-889. doi: 10.5244/c.21.79 Danilova, N. N. (1992). Psihofiziologicheskaya diagnostika funktsionalnyh sostoynij [Psychophysiological diagnostics of functional states]. Moscow: Lomonosov MSU Publishing. De Carlo, D., & Metaxas, D. (2000). Optical flow constraints to deformable models with applications to face tracking. International Journal of Computer Vision, 38(2), 99-127. doi: 10.1023/A:1008122917811 Ekman, P. (1999). Basic emotion. In T. Dalgleish and M. Power (Eds.), Handbook of cognition
and emotion (pp. 45-60). Sussex, U.K.: Wiley. doi: 10.1002/0470013494 Ekman, P., Friesen, W. V., & Hager, J. C. (2002). Facial action coding system investigator's guide
(2nd ed.). Salt Lake City: Research Nexus eBook. Ekman, P., Friesen, W. V., & Tomkins, S. S. (1971). Facial affect scoring technique: A first validity
study. Semiotica, 3(1), 37-58. doi: 10.1515/semi.1971.3.1.37 Engelmann, J. B., & Pogosyan, M. (2013). Emotion perception across cultures: The role of cognitive mechanisms. Frontiers in Psychology, 4(118), 1-10. doi: 10.3389/fpsyg.2013.00118 Fasel, B., & Luettin, J. (2003). Automatic facial expression analysis: A survey. Pattern Recognition, 36, 259-275. doi: 10.1016/S0031-3203(02)00052-3 Fukuda, K., Stern, J. A., Brown, T. B., & Russo, M. B. (2005). Cognition, blinks, eye-movements, and pupillary movements during performance of a running memory task. Aviation, Space, and Environmental Medicine, 76, 75-85. Galatenko, V. V., Livshitz, E. D., Podolskii, V. E., Chernorizov, A. M., & Zinchenko, Yu. P. (2012). Automated real-time classification of psychological functional state based on discrete wavelet transform of EEG data. International Journal of Applied Mathematics, 25(6), 871-882. Galatenko, V. V., Livshitz, E. D., Staroverov, V. M., Lukashenko, T. P., Galatenko, A. V., Podolskii, . E., ... Zinchenko, Yu. P. (2013). A remark on the most informative EEG signal components in a super-scalable method for functional state classification based on the wavelet decomposition. Procedia — Social and Behavioral Sciences, 86, 18-23. doi: 10.1016/j. sbspro.2013.08.518
Gur, R. C., Sara, R., Hagendoorn, M., Maron, O., Hughett, P., Macy, L., . Gur, R. E. (2002). A method for obtaining 3-dimensional facial expressions and its standardization for use in neurocognitive studies. Journal of Neuroscience Methods, 115(2), 137-143. doi: 10.1016/ S0165-0270(02)00006-7 Heaver, B., & Hutton, S. B. (2011). Keeping an eye on the truth? Pupil-size changes associated
with recognition memory. Memory, 19(4), 398-405. doi:10.1080/09658211.2011.575788 Hoffman, J. E., & Subramaniam, B. (1995). The role of visual attention in saccadic eye movements. Perception & Psychophysics, 57(6), 787-795. doi: 10.3758/BF03206794 Huang, Zh., & Ren, F. (2013). Facial expression recognition based on active appearance model & scale-invariant feature transform. In 2013 IEEE/SICE International Symposium on System Integration (SII) (pp. 94-99). New York: IEEE. doi: 10.1109/SII.2013.6776620 Immoreev, I. Ja., Isaychev, S. A., Samkov, S. N., & Pavlov, S. V. (2010). Distantsionnyj control psi-hofisiologicheskogo sostoyaniya cheloveka s pomoshch'yu SSH [Distant control of human psychophysiological state using SSH]. In Bioradiolokatsia (pp. 202-220). Moscow: MGTU. Jarlier, S., Grandjean, D., Delplanque, S., N'Diaye, K., Cayeux, I., Velazco, M. I., ... Scherer, K. R. (2011). Thermal analysis of facial muscles contractions. IEEE Transactions on Affective Computing, 2(1), 2-8. doi: 10.1109/T-AFFC.2011.3 Klingner, J., Kumar, R., & Hanrahan, P. (2008). Measuring the task-evoked pupillary response with a remote eye tracker. Proceedings of the 2008 symposium on Eye tracking research and applications, Savannah, Georgia, 69-72. doi: 10.1145/1344471.1344489
Kong, S. G., Heo, J., Abidi, B. R., Paik, J., & Abidi, M. A. (2005). Recent advances in visual and infrared face recognition: A review. Computer Vision and Image Understanding, 97, 103-135. doi: 10.1016/j.cviu.2004.04.001 Kovalev, A. I., Klimova, O. A., & Menshikova, G. Ya. (2016). Evaluation of vestibular dysfunctions in athletes using eye tracking and virtual reality. International Journal of Psychophysiol-ogy, 108, 142. doi: 10.1016/j.ijpsycho.2016.07.414 Lebedev, V. V., Isaychev, S. A., Chernorizov, A. M., & Zinchenko, Yu. P. (2013). Vector approach to integral evaluating of sportsmen functional states. Procedia — Social and Behavioral Sciences, 86, 610-614. doi: 10.1016/j.sbspro.2013.08.622 Lebedeva, N. N., & Karimova, E. D. (2014). Akusticheskie harakteristiki rechevogo signala kak pokazatel funktsionalnogo sostoyaniya cheloveka [Parameters of acoustic signal as an index of human functional state]. Uspehifiziologicheskih nauk [Advances in Physiological Sciences], 45(1), 57-95.
Leinhart, R., & Maydt, J. (2002). An extended set of Haar-like features for rapid object detection. IEEE International Conference on Image Processing 2002, 1, 900-903. doi: 10.1109/ ICIP.2002.1038171
Leonova, A. B. (2007). Strukturno-integrativnyj podhod k analizu funktsionalnyh sostoyanij cheloveka [A structural and integrative approach to analysis of human functional states]. Byulleten Moskowskogo Gosudarstvennogo Universiteta. Seriya 14. Psikhologiya [Moscow University Psychology Bulletin], 1, 87-103. Leonova, A. B. (2009). The concept of functional state in Russian applied psychology. Psychology
in Russia: State of the Art, 2, 517-538. doi: 10.11621/pir.2009.0026 Leonova, A. B., & Kuznetsova, A. S. (2015). Funktsionalnye sostoyaniya i rabotosposobnost cheloveka v professionalnoj deyatelnosti [Functional states and productivity of humans in professional activity]. In E. A. Klimov, O. G. Noskova, & G. N. Solntseva (Eds.), Psihologiya truda, inzhenernaya psihologiya i ergonomika (pp. 319-346). Moscow: Yurajt. Lewis, G. F., Gatto, R. G., & Porges, S. W. (2011). A novel method for extracting respiration rate and relative tidal volume from infrared thermography. Psychophysiology, 48(7), 877-887. doi: 10.1111/jj.1469-8986.2010.01167.x Lobacheva, E. M., Galatenko, Y. N., Gabidullina, R. F., Galatenko, V. V., Livshitz, E. D., Lukashenko, T. P., ... Zinchenko, Yu. P. (2013). Automated real-time classification of functional states based on physiological parameters. Procedia — Social and Behavioral Sciences, 86, 373-378. doi: 10.1016/j.sbspro.2013.08.582 Menshikova, G., Kovalev, A., Klimova, O., & Chernorizov, A. (2015). Eye movements as indicators of vestibular dysfunction. Perception, 44(8-9), 1103-1110. doi: 10.1177/0301006615594916 Minkin, V. A. (2007). Vibroizobrazhenie [Vibroimaging]. St. Petersburg: Renome. Nakayama, K., Goto, S., Kuraoka, K., & Nakamura, K. (2005). Decrease in nasal temperature of rhesus monkeys (Macaca Mulatta) in negative emotional state. Journal of Physiology and Behavior, 84, 783-790. doi: 10.1016/j.physbeh.2005.03.009 Pantic, M., & Rothkrantz, L.J.M. (2000). Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis and Machine Understanding, 22(12), 1424-1445. doi: 10.1109/34.895976
Park, J., Barash, V., Fink, C., & Cha, M. (2013). Emoticon style: Interpreting differences in emoti-cons across cultures. In Proceedings of the Seventh International AAAI Conference on Weblogs and Social Media (pp. 466-475). Palo Alto, CA: AAAI Press. Parr, L. A., Waller, B. M., Vick, S. J., & Bard, K. A. (2007). Classifying chimpanzee facial expressions using muscle action. Emotion, 7(1), 172-181. doi: 10.1037/1528-3542.7.1.172
Pavlidis, I. (2004). Lie detection using thermal imaging. In Proceedings of SPIE Thermosense XXVI (April) (pp. 270-279). Orlando, FL, The International Society for Optical Engineering, Bellingham, WA. doi: 10.1117/12.547806 Pavlidis, I., & Levine, J. (2001). Monitoring of periorbital blood flow rate through thermal image analysis and its applications to polygraph testing. In Engineering in Medicine and Biology Society, Proceedings of the 23rd Annual International Conference of the IEEE (Vol. 3, pp. 2826-2829). New York: IEEE. doi: 10.1109/iembs.2001.1017374 Potapova, R. K., Potapov, V. V., Lebedeva, N. N., & Agibalova, T. V. (2015). Mezhdisplinarnost v issledovanii rechevoj poliinformativnosti [Interdisciplinary approach to the study of polyinformation in speech]. Moscow: Yazyki Slavaynskoj Kultury. Puri, C., Olson, L., Pavlidis, I., Levine, J., & Starren, J. (2005). StressCam: Non-contact measurement of users' emotional states though thermal imaging. In CHI 2005: Extended abstracts on human factors in computing systems (pp. 1725-1728). New York: ACM. doi: 10.1145/1056808.1057007 Shapiro, F. (1989). Eye movement desensitization: A new treatment for post-traumatic stress disorder. Journal of Behavior Therapy and Experimental Psychiatry, 20, 211-217. doi: 10.1016/0005-7916(89)90025-6 Sokolov, E. N. (2013). The psychophysiology of consciousness. New York: Oxford University Press.
Turvey, E. T. (Ed.). (2012). Criminal profiling. An introduction to behavioral evidence analysis
(4th ed.). Amsterdam: Elsevier. Vavilov, V. P. (2013). Infrakrasnaya termografiya i teplovoj control [Infrared thermography and
heat control]. Moscow: Spektr. Vianna, D., & Carrive, P. (2005). Changes in cutaneous and body temperature during and after conditioned fear of context in the rat. European Journal of Neuroscience, 21, 2505-2512. doi: 10.1111/j.1460-9568.2005.04073.x Wade, N. J., & Tatler, B. W. (2005). The moving tablet of the eye. Oxford, UK: Oxford University
Press. doi: 10.1093/acprof:oso/9780198566175.001.0001 Wang, S., He, S., Wu, Y., He, M., & Ji, Q. (2014). Fusion of visible and thermal images for facial expression recognition. Frontiers of Computer Science, 8(2), 232-242. doi: 10.1007/s11704-014-2345-1
Yarbus, A. L. (1967). Eye movements and vision. New York: Plenum. doi: 10.1007/978-1-48995379-7
Original manuscript received August 02, 2016 Revised manuscript accepted October 28, 2016 First published online December 01, 2016
