CC BY-NC-ND
51
Psychophysiological Approaches to Instructional Design for Immersive Environments
Yu. Eremenko, O. Zalata
Received in Yulia Eremenko
June 2020 Candidate of Sciences in Economics, Associate Professor, Department of Marketing, Trade and Customs Affairs, V. I. Vernadsky Crimean Federal University. Address: 21/4 Sevastopolskaya St, 295015 Simferopol. Email: jul [email protected] Olga Zalata
Candidate of Sciences in Medicine, Associate Professor, Department of Normal Physiology, V. I. Vernadsky Crimean Federal University. Address: 5/7 Lenina Blvd, 295015 Simferopol. Email: olga [email protected]
Abstract. As part of the Digital Learning Environment federal project, school education programs in Russia are expected to implement modern technology, including virtual and augmented reality. The integration of immersive technology in education should be based on research findings about the influence of virtual environments on learning effectiveness. Specific aspects of immersive technology include the sense of presence, interactivity, social interaction and multisensory stimulation, which cumulatively exert quite a controversial influ-
ence on learning experience and outcomes. Since little data is available, instructional design decisions are often based on practical or economic considerations.
Therefore, it has become vitally important to use objective methods in assessing the learning content in order to understand its effects on the learner's cognitive and emotional processes immediately involved in the perception and digestion of educational material. Evaluation of virtual learning content in the process of its design and integration is suggested to be based on such parameters as presence, cognitive load, emotional response, social interaction, and the risks of experiencing cybersickness symptoms.
Analysis of literature and the resulting structured set of methodological tools will aid further studies in the field that use psychophysiological research methods to design effective learning content in virtual reality environments. Keywords: immersive virtual environment, learning, learning content, neu-rophysiological methods of research.
DOI: 10.17323/1814-9545-2020-4-207-231
Immersive technology, creating a sense of embodiment in virtual real-Translated ity and allowing the user to interact with the surrounding space, infor-from Russian by mation and content, has been applied more and more widely in vari-I. Zhuchkova. ous spheres of social life.
Virtual reality (VR) is a three-dimensional, computer-generated environment that can be interacted with by a person through full or partial immersion [Selivanov, Selivanova 2014]. VR has been used in such diverse areas of learning as programming, tourism, marketing, medicine, linguistics, etc. It helps making the presentation of educational material more effective, immersing the learner in scenarios that are very hard to create in the real world (e. g. rescue actions in case of fire at a hazardous factory) and providing the opportunity to master new skills, the acquisition of which is associated with risks to other people's health (e. g. sophisticated microsurgical manipulations).
As VR technology has been growing more accessible due to the continuing drop in hardware costs and the emergence of diverse and open content, theoretical research in the field has grown as well. A number of studies explore the opportunities that VR provides in the development of abstract thinking, e. g. in geometry or vector algebra [Hwang, Hu 2013; Roussou, Oliver, Slater 2006; Roussou 2009; Kaufmann, Schmalstieg, Wagner 2009]. Others analyze the use of immersive environments in practical training [Alaraj et al. 2011], endoscopic surgery [van Dongen et al. 2011], engineering education [Ewert et al. 2014; Alhalabi 2016] and neurosurgical training simulators with hap-tic feedback [Müns, Meixensberger, Lindner 2014]. A lot of researchers study the use of VR in the humanities, such as linguistics [Wang, Newton, Lowe 2005; Lin, Lan 2015] or history [Blanco-Fernández et al. 2014] i.
There are several reasons why VR is an excellent tool for education. First, it can change the abstract into the tangible. This could be especially powerful in the teaching of mathematics. Second, it supports "doing" rather than just observing. Third, it can substitute methods that are desirable but practically infeasible even if possible in reality [Slater, Sanchez-Vives 2016].
Successful application of any innovative technology in education is contingent on its appropriate use in accordance with the educational goals and objectives, but also on the quality of learning environment and instructional design (Figure 1).
VR is a world created by technical means, which is transmitted to a person through their senses. The technical goal of VR is to replace real sense perceptions by the computer-generated ones. If sensory perceptions are indeed effectively substituted, it creates an effect of immersion, i. e. illusion of presence in a virtual world. In education, immersion allows students to acquire learning experience. Immersion is achieved through the technical capabilities of a system, and a subjective correlate of immersion is presence [Slater, Sanchez-Vives 2016].
1 Takahashi D. (2019) Holocaust Memorial Museum Uses Augmented Reality to Make History Visceral. Available at: https://venturebeat.com/2018/08/31/hol-ocaust-memorial-museum-uses-augmented-realitv-to-make-historv-vis-ceral/
Figure 1. The structure of immersive learning environments
Immersive learning environment
Hardware
Interaction technique
Learning content
i
I
I
1
Scenarios Environment
Simulation Instructor/ Learner module avatar support
module module
design (colors, sounds, smells, etc.)
Factors that are critical for successful immersion, or presence, include effective sensory substitution (wide field-of-view vision, stereo, high-resolution displays, etc.), degree of interactivity, multisensory stimulation, emotional response, and social interaction. These factors can be designed to stimulate the necessary cognitive and affective processes in learning.
Available findings indicate that the role of immersive environments in providing interactivity, multisensory stimulation, social interaction and emotional response has a controversial influence on cognitive and affective processes and, consequently, on the learning outcomes.
In particular, mixed results have been reported in studies examining the impact of presence on learning effectiveness. A number of researchers [Bayraktar 2002; Bonde et al. 2014; Clark, Tanner-Smith, Killingsworth 2016; Merchant et al. 2014; Rutten, van Joolingen, van der Veen 2012] have found that low-immersion virtual realities correlate with better cognitive outcomes. A possible reason for this could be that a high level of immersion leads to higher cognitive load and, as a result, less learning [Makransky, Terkildsen, Mayer 2017]. However, there are contradictory findings [Salzman et al. 1999; Lee, Wong, Fung 2010] showing that immersive environments with higher presence levels lead to higher learner engagement and motivation and richer cognitive benefits.
Table 1 provides a summary of findings obtained in a number of studies on how specific parameters of immersive environments affect learning.
There is no doubt that immersive technology can have a significant impact on learning outcomes due to its ample opportunity for simulating emotional experiences, increasing the learner's attention to and interest in the learning content, and allowing them to "live" their educa-
Table 1. The impact of immersive reality parameters on cognitive and emotional processes in learning.
Parameter/ measure of immersion Author(s) Impact on cognitive and emotional processes in learning Statistical analysis results
Interactivity [Zhang, Bowman, Jones 2019] 1. Post-tests revealed no significant difference in students' learning gains between VR experiences with different levels of interactivity. 2. Effectiveness of VR learning experiences: in post-tests, the medium interactivity condition was perceived as significantly more effective than the low interactivity condition. 1. Analysis of learning outcomes in three versions of experience 2. x2 (2) = 6,107; p = 0,047 (t = -12,445; p = 0,05)
[Zha ng et al. 2006] The post-gain of the group with interactive video was higher than that of the other three groups (with no interaction). A positive correlation between the level of student satisfaction and content interactivity. 1. F (3, 134) = 9,916; p = 0,00 2. F (3, 134) = 23,696; p = 0,00
Emotional response [Allcoat, von Muhlenen 2018] A significant increase in positive emotions for the VR condition. A significant decrease in positive emotions for the traditional (textbook style) condition. 1. t(30) = 4,73; p < 0,001 2. t(33) = 4,92; p < 0,001
Social interaction [Ravaja et al. 2006] Playing against a friend or a stranger elicited higher anticipated threat compared to playing against a computer. Spatial presence was higher when playing with a friend or a stranger compared to playing against the computer. Playing with a human elicited higher engagement than playing with a computer. Playing against a human elicited a more positive emotional response compared to playing against a computer. 1. F (1, 32) = 7,55; p = 0,010; x2 = 0,19 2. F (1, 32) = 5,22; p = 0,029; x2 =0,14 3. F (1, 32) = 17,83; p = 0,001; x2 =0,36 4. F (1, 32) = 24,19; p = 0,001; x2 = 0,43
Multimedia learning / multisensory stimulation [Moreno, Mayer 2000] Students remembered significantly less verbal material in the condition with background music. (M = 7,65, SD = 3,73) and (M = 11,37, SD = 3,29); F (1, 71) = 21,99; MSE = 11,61; p < 0,0001
[Kartiko, Kavakli, Cheng 2009] Application of animated-virtual actors (increased visual complexity) did not affect the learning outcomes. x2 (2, N = 200) = 0,12; p = 0,94
tional experiences. Whether these outcomes are positive or negative depends on the fundamental principles used in instructional design.
As can be seen from Table 1, the most debatable aspect of instructional design is the assessment of cognitive load, which is largely contingent on the levels of multisensory stimulation and interactivity. John
Sweller, the developer of cognitive load theory, contended that cognitive load depended on the amount of information in working memory [Sweller 1998]. Since short-term memory is limited, the teaching methods should not overload it with additional processes that do not contribute to learning directly. Therefore, it is impossible to determine how exactly modern multimedia affect learning outcomes, as they make students process additional sensory information of various types.
The cognitive theory of multimedia learning [Mayer 2009] suggests three types of processing: extraneous processing (which does not support the instructional goal and is caused by poor instructional design or distractions), essential processing (which is aimed at representing the essential presented material in working memory and is caused by the complexity of the material), and generative processing (which is aimed at making sense of the presented material and is caused by the learner's motivation to learn. Given that the learner's cognitive capacity is limited, an excessive amount of sensory stimuli or distractions may have negative effects on the learning outcomes. That is to say, high-immersion virtual environments may significantly increase irrelevant cognitive load.
Cognitive processes are also substantially influenced by emotions, which modulate the selectivity of attention as well as motivate action and behavior [Tyng et al. 2017; Immordino-Yang 2015]. Positive emotions improve performance on tasks requiring creative ingenuity [Isen, Daubman, Nowicki 1987; Fredrickson 2001; Greene, Noice 1988]. Negative emotions have negative effects on learning, directing a student's attention to themselves because they try to find ways to get rid of the bad feeling [Hascher 2010]. Student enjoyment was empirically proven to correlate with self-regulated learning and creative problem solving (0.43, p < 0.001) [Goetz et al. 2006].
Since VR technology can elicit a very strong emotion, is has great potential for simulating emotional states conducive to effective learning. Intense emotional responses in immersive environments are linked to presence: on one side, the feeling of presence is greater in the "emotional" environments; on the other side, the emotional state is influenced by the level of presence [Riva et al. 2007].
Therefore, the key to effective application of immersive environments in education is the development of fundamental principles of instructional design using psychophysiological methods of assessment that provide the most objective quantitative data on cognitive processes and emotional states. These methods measure unconscious physiological responses to environmental stimuli and allow for real-time data collection.
2 Milk C. (2015) How Virtual Reality Can Create the Ultimate Empathy Machine. Available at: https://www.ted.com/talks/chris milk how virtual reality can create the ultimate empathy machine
Table 2. Psychophysiological assessment methods and vital signs monitored in VR learning environments
Presence Cognitive load Emotional response Social interaction VR sickness (cybersickness)
Eye tracking, Eye tracking, Eye tracking, Eye tracking, electrocardiography,
electrocardiography, electroencephalogra- electrocardiography, electrocardiography, electrodermal
electromyography phy, electromyography, electromyography, activity,
event-related electroencephalogra- electroencephalogra- facial coding,
potentials phy, phy, heart rate
electrodermal electrodermal
activity, activity,
functional magnetic facial coding,
resonance imaging, respiratory rate,
facial coding, heart rate
respiratory rate,
heart rate
The most important parameters for the assessment of VR learning content include presence, cognitive load, emotional response, social interaction, and VR sickness caused by low-quality content that can lead to dizziness, nausea, high blood pressure, etc.
Table 2 provides an overview of the psychophysiological assessment methods and the vital signs monitored in VR learning environments.
Presence The sense of presence, or "living" a learning experience, is also referred to as the feeling of "being there" or "place illusion". Place illusion can occur in a static environment where nothing happens. When there are events in the environment that respond to the user or correlate with their actions, the user experiences an illusion of plausibility that the events are really happening. In VR learning environments, presence increases the learner's attention, interest and motivation.
Most methods of measuring presence in immersive environments are based on psychological approaches. For example, questionnaires are used to measure such presence parameters as curiosity, concentration, challenge, control, comprehension and emotion [Qin, Pei-Lu-en, Salvendy 2007]. Full immersion into VR is described by eight major components: clear goals, deep involvement, no concern for the self, alteration of the concept of time, immediate feedback, reasonable chance of completion, sense of control over one's actions, and useful experience [Csikszentmihalyi 1990].
Eye tracking is a commonly used psychophysiological technique measuring the level of immersion. In particular, findings indicate that an increase in immersion causes a significant increase in median star-
tle eyeblink amplitudes [Parsons et al. 2009]. The authors contend that "high-immersion" scenarios evoke a stronger physiological reaction than "low-immersion" scenarios. In immersive learning environments, an individual's number of fixations per second decreases, as their attention becomes more focused [Jennett et al. 2008].
Cognitive load Multisensory and interactive experiences offered by immersive environments can cause cognitive overload. High levels of cognitive load caused by irrelevant cognitive processing may have a negative impact on learning. Electroencephalography, including EEG signals based on event-related potentials, is the most objective measure of cognitive load [Sterman, Mann 1995; Gerjets et al. 2014; Mühl, Jeunet, Lotte 2014; Kumar, Kumar 2016].
An increase in functional brain activity (intense attention or mental effort) suppresses a-band EEG oscillations, stimulating irregular and low-amplitude activity instead. Physiologically, this response is interpreted as desynchronized neural activity and is referred to as "neural desynchronization" or "arousal", depending on the context in which it is recorded. Arousal occurs as a response to a new factor in the environment that requires a different level of orientation [Danilova 2001, Zenkov 2001]. A few studies using digital filters and computer analysis of EEG signals have recorded a special type of activity in the ß2 band (35-45 Hz) associated with cognitive processes, in particular active attention and sensorimotor integration [Danilova 2001]. Therefore, typical changes in EEG patterns reflecting the dynamic changes in brain cortex activity may be recommended for use as indicators of mental processes corresponding to the absorption of new information, which is also true for immersive environments.
Eye movement-based analysis of cognitive load [Goldberg, Kot-val 1999] reveals a number of patterns. First, more difficult processing is associated with fewer fixations (the brain cannot search for a target when busy doing a cognitive task). Second, longer fixation duration is associated with higher cognitive load (the user spends too much time processing). Third, higher ratios of time spent processing (fixation) to time spent searching (saccade) correlate with higher cognitive load in virtual environments.
Therefore, eye tracking in instructional design allows analyzing and directing the learner's attention, minimizing the distractions, and measuring the learner's attention to and interest in specific content elements in order to improve learning effectiveness.
Emotional Emotions exert an essential influence on perception, decision mak-response ing, attention, memory and other cognitive processes that are critical for learning. Automated emotion quantification and recognition is called affective computing. This methodology combines knowledge
from psychophysiology, computer science, biomedical engineering and artificial intelligence. Two approaches to emotion modelling have commonly been proposed in psychology: discrete and dimensional models. The former posits the existence of small innate sets of basic, universally expressed emotions, such as happiness, anger, irritation, etc. Dimensional models consider a continuous multidimensional space where each dimension stands for a fundamental property common to all emotions. Two of the most accepted dimensions are valence (i. e. pleasure, positive versus negative affect) and arousal (low versus high level of activation).
The past two decades have seen the arrival of new neuro- and psychophysiological techniques for studying emotional processing and its neural correlates. Several computational methods for emotion recognition based on variables associated with the central nervous system have been proposed, the most widespread one being EEG, which allows measuring emotion valence and intensity. A wider class of affective computing methods are based on changes in the autonomic nervous system. These include heart rate variability (HRV) analysis, electrocardiography (ECG), electromyography (EMG), electrodermal activity (EDA), eye tracking, and facial expression coding.
Combinations of methods provide more reliable and objective data. For instance, a new methodology for the automatic assessment of emotional responses based on ECG, EDA and respiration activity (RSP) was proposed. Results show that, when nonlinearly extracted features are used, the percentages of successful recognition dramatically increase [Valenza, Lanata, Scilingo 2012]. Another emotion recognition system based on physiological signals uses ECG and RSP data [He, Yao, Ye 2017]. In this study, a support vector machine (SVM) method was applied, which achieved a recognition accuracy of 81.82, 63.64, 54.55, and 30.00% for joy, sadness, anger, and pleasure, respectively. An emotion recognition system for affective states involving the recording of EEG and ECG signals was proposed, the model's accuracy being 75.00% along the arousal dimension and 71.21% along the valence dimension [Marin-Morales et al. 2018].
Emotion recognition using eye tracking is largely based on measuring variations in the diameter of the pupillary aperture of the eye (pupillometry) [Granholm, Steinhauer 2004; Steinhauer et al. 1983]. Pupil dilation is normally associated with viewing pleasant pictures [Bradley et al. 2008]. However, not only emotion or sympathetic activation in response to stress, but an increase in cognitive load, atten-tional allocation or working memory maintenance as well can result in pupil dilation.
Facial coding is based on specific motion recognition and identification algorithms. Facial coding technology normally implies an algorithm of three steps: (i) automatic face detection using the Viola-Jones cascade classifier; (ii) detection of characteristic facial features within the detected face using such landmarks as eyes, eye and
Table 3. An example of presenting the output of an emotion recognition algorithm (Microsoft Azure)
Numeric values Interpretation
Output 1: 0.001000087;0.050074505;0;0;0.001000474; True; Output 2: 0;0;0;0;0,8; True; Output 3: 0;0;0;0;0; False Anger Contempt Disgust Fear Happiness True—data recorded False—data not recorded
'identification of the respective emotion in a photo corresponds to a value between 0 and 1.
mouth corners, nose tip, mouth shape, etc.; and (iii) identification of characteristic features, or translation of facial features into metrics [Yarosh et al. 2020]. An example of this technology is Microsoft Azure (Table 3), which offers cloud-based algorithms for face detection, recognition and analysis [Eremenko, Ulanovskaya 2020].
Emotion recognition based on facial coding has been quite a success. For example, matching scores of 89% were reported for recognition of facial expressions when validating FaceReader3, automated facial coding software [Lewinski, Uyl, Butler 2004].
Social interaction "Eye-to-eye" interaction is a critical component of learning that can make statements sound more approving, supportive or credible.
There has been little research on social interaction in VR learning environments. Most of the available studies explore social interaction in online games or in "communication" with computer avatars as well as the perception of avatars, e. g. the influence of the type of opponent in video games (computer, friend, or stranger) on presence, emotional reactions and threat and challenge appraisals [Ravaja et al. 2006]. Analysis of EDA and facial EMG activity as well as self-reported perceptions indicate that playing with a friend elicits more arousal and positive valence than playing with a stranger. The presence of another person increases player involvement and enthusiasm, prompting them to choose a higher level of difficulty.
Psychophysiological methods have been used to examine social interaction involving three-dimensional digital human representations—virtual human avatars or computer "agents". In particular, interpersonal distance was found to be regulated by participant gender, perceived avatar gender, and mutual gaze behavior [Bailenson, Blas-
3 https://www.noldus.com/facereader
covich, Beall 2003]. The authors introduced the concept of "digital chameleon" (the chameleon effect): participants showed more preference to and kept the smallest interpersonal distance with the avatars that mimicked their own nonverbal behavior. People were also more likely to choose to be represented by anthropomorphic avatars that matched their gender and race [Nowak, Rauh 2005].
VR sickness Learning content of poor quality inhibits the absorption of educational material. In extreme cases, it causes VR sickness (cybersickness), which may involve dizziness, nausea, high blood pressure and increased heart rate. A questionnaire proposed to measure cybersickness consisted of 16 questions about symptoms, which were divided into three main categories: (1) nausea (general discomfort, increased salivation, sweating, nausea, difficulty concentrating), (2) oculomotor symptoms (general discomfort, fatigue, headache, eyestrain, difficulty concentrating, blurred vision); and (3) disorientation (difficulty focusing, nausea, blurred vision, dizziness with eyes open/closed) [Kennedy et al. 1993].
Post-tests are used to assess the overall body reaction to virtual learning content, and psychophysiological methods allow identifying what exactly and when caused the discomfort and symptoms of cybersickness. This is achieved by monitoring the EDA signals, hemod-ynamic metrics (blood pressure and heart rate), and respiratory rate and rhythm. Symptoms of cybersickness and changes of brain regional activity were investigated using EEG-based source localization, before and after a VR experience involving a smartphone-assisted head mount display [Kim et al. 2019].
Hemodynamic metrics are good indicators of stress. For example, an analysis of systolic and diastolic blood pressure as well as heart rate before and after a VR and hypermedia-based learning experience revealed an increase in blood pressure in VR learning environments, which may indicate an increased sympathetic tone of the autonomic nervous system that mediates response to stress—in this case a new format of learning. Meanwhile, learning in hypermedia-based environments, more familiar to students, showed a decrease in blood pressure, which may indicate less strain in the sympathetic division of the nervous system [Astafurov et al. 2020].
Neurophysiological techniques have been applied successfully in instructional design. Assessment of the learning content may be performed before, during and after the interaction, which is an undeniable advantage of such methods.
Table 4 presents the most accessible tools for neurophysiological research—metrics and their interpretation.
Therefore, objective psychophysiological methods can be used to achieve strategic and tactical goals in VR instructional design. Strategically, they allow evaluating the feasibility of innovative education-
Table 4. Methods and metrics of VR instructional design assessment.
Cognitive and affective processes Neurophysio-logical methods of measurement Metrics Interpretation Applications
Attention and interest Video-oculog-raphy Oculomotor activity metrics Managing the learner's active attention to and interest in the learning content. The optimal response criteria include maintained interest and patterns of visual attention typical of effective learning
Heat maps Color contrasts are used to demonstrate areas of visual attention and interest
Time to first fixation (TTFF) Describes the process of searching for the area of interest, i. e. which parts of the visual stimulus are of priority to the observer
Ratio of first fixation duration to TTFF A short TTFF and a long first fixation duration indicate that the visual object is highly attractive
Saccades The number of saccades increases when viewing uninteresting content [Grobelny et al. 2006]
Fixations Refixations indicate the number of times the person revisits the area of interest
Blink rate A low blink rate indicates a high level of concentration [Yarosh et al. 2020]
EEG Changes in the typical EEG power spectral frequencies Theta rhythm is used to measure the level of interest in the presented material
Emotions Automated facial coding Metrics for the automatic emotion recognition based on analysis of typical facial features Identification of emotions: joy, fear, anger, surprise, contempt Modelling the desired emotional states in learning
Cognitive load EEG Changes in the typical EEG power spectral frequencies Indicators of cognitive processes: neural desynchronization; increase in the high-frequency bands of the EEG power spectrum (beta-1 and beta-2 rhythms) Achieving an optimal level of cognitive load for VR learning environments. Neural desynchronization is considered the criterion for optimal response in EEG patterns of learners in immersive environments
Video-oculog-raphy Oculomotor activity metrics
Fixations Duration of fixation is dependent on how easy or difficult the display is to process [Renshaw et al. 2004]. The number of fixations increases in high cognitive load scenarios [Grobelny et al. 2006]
Pupillary response Pupil dilation is associated with higher cognitive load [Poole, Ball, Phillips 2006]
VR sickness Arterial tonometry with pulse (heart rate) measurement Hemodynamic metrics (systolic and diastolic blood pressure, heart rate) Emotional stress assessment based on changes in blood pressure and heart rate (increased BP and HR) Identifying and preventing the symptoms of VR sickness in immersive learning environments. A maximum increase of 10% in heart rate and blood pressure is considered the criteria for optimal hemodynamic response
http://vo.hse.ru/en/
al technology and discriminating between efficient learning tools and fashion statements. Tactically, psychophysiological techniques may be used for elaborating the fundamental principles of effective VR instructional design, assessing the level of physical and psychological comfort in immersive environments with the help of various devices, and analyzing the individual behavior of students in VR learning environments.
Conclusion This review of literature on the impact of virtual environments on learning effectiveness shows that the study of such VR parameters as presence, interactivity, social interaction and multisensory stimulation should be focused on assessing the quality of knowledge acquired in such environments as well as the risks of experiencing cybersickness symptoms.
Objectiveness in assessing the impact of immersive environments on the quality of learning can be provided by a broad range of psychophysiological measures (eye movement tracking, electroencephalography, electrocardiography, electromyography, functional magnetic resonance imaging, etc.) and physiological parameter monitoring in field experiments. It may be recommended to record a minimum of three vital signs reflecting the level of active attention, cognitive load and the risks of experiencing VR sickness symptoms to optimize psychophysiological observations.
VR instructional design assessment using objective psychophysiological methods at the stage of development will contribute to the integration of state-of-the-art technology in online education and improve learning effectiveness.
References Alaraj A., Lemole M. G., Finkle J. H. et al. (2011) Virtual Reality Training in Neurosurgery: Review of Current Status and Future Applications. Surgical Neurology International, vol. 2, Art. no 52. D0l:10.4103/2152-7806.80117 Alhalabi W. S. (2016) Virtual Reality Systems Enhance Students' Achievements in Engineering Education. Behaviour & Information Technology, vol. 35, no 11, pp. 919-925. http://doi.org/10.1080/0144929X.2016.1212931 Allcoat D., von Muhlenen A. (2018) Learning in Virtual Reality: Effects on Performance, Emotion and Engagement. Research in Learning Technology, vol. 26, November. Art. no 2140. DOI: 10.25304/rlt.v26.2140 Astafurov D., Verbenko P., Dorozhkin R., Zhukova A., Izmailov E., Kravchenko A., Prasolov N., Sereda E. (2020) Kharakteristika fiziologicheskikh i psikholog-icheskikh reaktsiy u studentov pri predyavlenii informatsionnogo kontenta v srede virtualnoy realnosti [Characteristics of Physiological and Psychological Reactions of Undergraduates Presented with Information Content in a Virtual Reality]. Proceedings of Theoretical and Applied Aspects of Modern Medicine, the 92nd International Academic Conference for Students and Young Scientists (Simferopol, April 16, 2020), Simferopol: Medical Academy named after S. I. Georgievsky, V. I. Vernadsky Crimean Federal University, pp. 349-351.
Bailenson J., Blascovich J., Beall A. (2003) Interpersonal Distance in Immersive Voprosy obrazovaniya / Educational Studies Moscow. 2020. No 4. P. 207-231
Virtual Environments. Personality & Social Psychology Bulletin, vol. 29, July, pp. 819-833. D0l:10.1177/0146167203029007002 Bayraktar S. (2002) A Meta-Analysis of the Effectiveness of Computer-Assisted Instruction in Science Education. Journal of Research on Technology in Education, vol. 34, no 2, pp. 173-188. https://doi.org/10.1080/15391523.2001.10782344 Blanco-Fernández Y., López-Nores M., Pazos-Arias J.J., Gil-Solla A., Ramos-Cabrer M., García-Duque J. (2014) REENACT: A Step Forward in Immersive Learning about Human History by Augmented Reality, Role Playing and Social Networking. Expert Systems with Applications, vol. 41, no 10, pp. 4811-4828. Bonde M., Makransky G., Wandall J., Larsen M., Morsing M., Jarmer H., Sommer M. (2014) Improving Biotech Education through gGamified Laboratory Simulations. Nature Biotechnology, vol. 32, no 7, pp. 694-697. DOI: 10.1038/nbt.2955 Bradley M. M., Miccoli L., Escrig M. A., Lang P. J. (2008) The Pupil as a Measure of Emotional Arousal and Autonomic Activation. Psychophysiology, vol. 45, no 4, pp. 602-607.
Calvo R.A, D'Mello S. (2010) Affect Detection: An Interdisciplinary Review of Models, Methods, and their Applications. IEEE Transactions on Affective Computing, vol. 1, no 1, pp. 18-37. DOI: 10.1109/T-AFFC.2010.1 Clark B. D., Tanner-Smith E.E., Killingsworth S. S. (2016) Digital Games, Design, and Learning: A Systematic Review and Meta-Analysis. Review of Educational Research, vol. 86, no 1, pp. 79-122. https://doi.org/10.3102/0034654315582065 Csikszentmihalyi M. (1990) Flow: The Psychology of Optimal Experience. New York: Basic Books.
Danilova N. (2001) Psikhofiziologiya [Psychophysiology], Moscow: Aspekt. Dongen van K.W., Ahlberg G., Bonavina L. et al. (2011) European Consensus on a Competency-Based Virtual Reality Training Program for Basic Endoscopic Surgical Psychomotor Skills. Surgical Endoscopy, vol. 25, no 1, pp. 166-171. D0I:10.1007/s00464-010-1151-6 Eremenko Ju., Ulanovskaya O. (2020) Sovremennye metody i tekhnologii prikladno-go neyromarketinga [Modern Methods and Technologies of Applied Neuromarketing]. Journal of Marketing in Russia and Abroad, no 2, pp. 37-45. Ewert D., Schuster K., Johansson D., Schilberg D., Jeschke S. (2014) Intensifying Learner's Experience by Incorporating the Virtual Theatre into Engineering Education. Automation, Communication and Cybernetics in Science and Engineering 2013/2014 (eds S. Jeschke, I. Isenhardt, F. Hees, and K. Henning), Heidelberg, New York, Dordrecht, London: Springer, pp. 255-267. Fredrickson B. L. (2001) The Role of Positive Emotions in Positive Psychology. American Psychologist, vol. 56, no 3, pp. 218-226. http://dx.doi.org/10.1037/0003-066X.56.3.218
Friedman D., Steed A., Slater M. (2007) Spatial Social Behavior in Second Life. Intelligent Virtual Agents. IVA 2007. Lecture Notes in Computer Science. Vol. 4722. (eds C. Pelachaud, J. C. Martin, E. André, G. Chollet, K. Karpouzis, D. Pelé), Berlin, Heidelberg: Springer, pp. 252-363. https://doi.org/10.1007/978-3-540-74997-4_23
Gerjets P., Walter W., Rosenstiel W., Bogdan M., Zander T. O. (2014) Cognitive State Monitoring and the Design of Adaptive Instruction in Digital Environments: Lessons Learned from Cognitive Workload Assessment Using a Passive Brain-Computer Interface Approach. Frontiers in Neuroscience, vol. 8, Art. no 386. DOI: 10.3389/fnins.2014.00385 Goetz T., Hall N. C., Frenzel A. C., Pekrun R. (2006) A Hierarchical Conceptualization of Enjoyment in Students. Learning and Instruction, vol. 16, no 4, pp. 323338. DOI: 10.1016/j.learninstruc.2006.07.004 Goldberg J. H., Kotval X. P. (1999) Computer Interface Evaluation Using Eye Movements: Methods and Constructs. International Journal of Industrial Ergonomics, vol. 24, no 6, pp. 631-645.
Granholm E., Steinhauer S. R. (2004) Pupillometric Measures of Cognitive and Emotional Processes. International Journal of Psychophysiology, vol. 52, no 1, pp. 1-6.
Greene T., Noice H. (1988) Influence of Positive Affect upon Creative Thinking and Problem Solving in Children. Psychological Reports, vol. 63, no 3, pp. 895-898.
Grobelny J., Jach K., Kulinski M., Michalski R. (2006) Sledzenie wzroku w bada-niach jakosci uzytkowej oprogramowania. Historia i mierniki [Tracking of Eye Movements in Studies of Software Quality. History and Measurement Methods]. Paper presented at Interfejs uzytkownika—Kansei w praktyce Conference (Warszawa 2006). (Unpublished).
Hascher T. (2010) Learning and Emotion: Perspectives for Theory and Research. European Educational Research Journal, vol. 9, no 1, pp. 13-28.
He C., Yao Y., Ye X. (2017) An Emotion Recognition System Based on Physiological Signals Obtained by Wearable Sensors. Wearable Sensors and Robots. Lecture Notes in Electrical Engineering. Vol. 399 (eds C. Yang, G. Virk, H. Yang), Singapore: Springer, pp. 15-25.
Hwang W.-Y., Hu S.-S. (2013) Analysis of Peer Learning Behaviors Using Multiple Representations in Virtual Reality and their Impacts on Geometry Problem Solving. Computers & Education, vol. 62, March, pp. 308-319. D0I:10.1016/j. compedu.2012.10.005
Immordino-Yang M.H. (2015) Emotions, Learning and the Brain: Exploring the Educational Implications of Affective Neuroscience. New York: W. W. Norton & Co.
Isen A. M., Daubmann K., Nowicki G. (1987) Positive Affect Facilitates Creative Problem Solving. Journal of Personality and Social Psychology, vol. 52, no 6, pp. 1122-1131. http://dx.doi.org/10.1037/0022-3514.52.6.1122
Jennett Ch., Cox A. L., Cairns P., Dhoparee S., Epps A., Tijs T., Walton A. (2008) Measuring and Defining the Experience of Immersion in Games. International Journal of Human-Computer Studies, vol. 66, no 9. P. 641-661. DOI: 10.1016/j. ijhcs.2008.04.004
Kartiko I, Kavakli M., Cheng K. (2010) Learning Science in a Virtual Reality Application: The Impacts of Animated-Virtual Actors' Visual Complexity. Computers & Education, vol. 55, no 2, pp. 881-891.
Kaufmann H., Schmalstieg D., Wagner M. (2009) Construct 3D: A Virtual Reality Application for Mathematics and Geometry Education. Education and Information Technologies, vol. 5, iss. 4, pp. 263-276.
Kennedy R. S., Lane N. E., Berbaum K. S., Lilienthal M. G. (1993) Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness. The International Journal of Aviation Psychology, vol. 3, no 3, pp. 203-220. DOI: 10.1207/s15327108ijap0303_3
Kim J.-Y., Son J.-B., Lee H.-S., Lee S.-H. (2019) Psychophysiological Alteration after Virtual Reality Experiences Using Smartphone-Assisted Head Mount Displays: An EEG-Based Source Localization Study. Applied Sciences, vol. 9, no 12, Art. no 2501. D0I:10.3390/app9122501
Kumar N., Kumar J. (2016) Measurement of Cognitive Load in HCI Systems Using EEG Power Spectrum: An Experimental Study. Procedia—Computer Science, vol. 84, December, pp. 70-78.
Lee E. A.-L., Wong K. W., Fung C. C. (2010) How Does Desktop Virtual Reality Enhance Learning Outcomes? A Structural Equation Modeling Approach. Computers & Education, vol. 55, no 4, pp. 1424-1442. http://doi.org/10.1016/j. compedu.2010.06.00
Lewinski P., Uyl T., Butler C. (2014) Automated Facial Coding: Validation of Basic Emotions and FACS AUs in FaceReader. Journal of Neuroscience, Psychology, and Economics, vol. 7, no 4, pp. 227-236.
Lin T.-J., Lan K. (2015) Language Learning in Virtual Reality Environments: Past, Present, and Future. Educational Technology & Society, vol. 18, no 4, pp. 486497.
Makransky G., Terkildsen Th.S., Mayer R. E. (2017) Adding Immersive Virtual Reality to a Science Lab Simulation Causes More Presence but Less Learning. Learning and instruction, Vol. 60, April, pp. 225-236.
Marin-Morales J., Higuera-Trujillo J.L., Greco A., Guixeres J. (2018) Affective Computing in Virtual Reality: Emotion Recognition from Brain and Heartbeat Dynamics Using Wearable Sensors. Scientific Reports, vol. 8, no 1, Art. no 3657. DOI: 10.1038/s41598-018-32063-4
Mayer R. E. (2014) Cognitive Theory of Multimedia Learning. The Cambridge Handbook of Multimedia Learning (ed. R. E. Mayer), New York: Cambridge University, pp. 43-71.
Merchant Z., Goetz E., Cifuentes L., Keeney-Kennicutt W. Davis T. (2014) Effectiveness of Virtual Reality-Based Instruction on Students' Learning Outcomes in K-12 and Higher Education: A Meta-Analysis. Computers & Education, vol. 70, no 1, pp. 29-40. DOI: 10.1016/j.compedu.2013.07.033
Moreno R., Mayer R. E. (2000) A Coherence Effect in Multimedia Learning: The Case for Minimizing Irrelevant Sounds in the Design of Multimedia Messages. Journal of Educational Psychology, vol. 92, no 1, pp. 117-125.
Mühl Ch., Jeunet C., Lotte F. (2014) EEG-Based Workload Estimation across Affective Contexts. Frontiers in Neuroscience, vol. 8, Art. no 114.
Müns A., Meixensberger J., Lindner D. (2014) Evaluation of a Novel Phantom-Based Neurosurgical Training System. Surgical Neurology international, vol. 5, Art. no 173. D0I:10.4103/2152-7806.146346
Nowak K., Rauh Ch. (2005) The Influence of the Avatar on Online Perceptions of Anthropomorphism, Androgyny, Credibility, Homophily, and Attraction. Journal of Computer-Mediated Communication, vol. 11, no 1, pp. 153-178. DOI:10.1111/j.1083-6101.2006.tb00308.x
Parsons Th.D., Iyer A., Cosand L., Courtney Ch., Rizzo A. A. (2009) Neurocognitive and Psychophysiological Analysis of Human Performance within Virtual Reality Environments. Paper presented at the 17th Annual Medicine Meets Virtual Reality (MMVR17) (Long Beach, CA, January 19-22, 2009). DOI:10.3233/978-1-58603-964-6-247
Pausch R., Proffitt D., Williams G. (1997) Quantifying Immersion in Virtual Reality. Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (Los Angeles, August 03-08, 1997), SIGGRAPH '97, New York, NY: ACM Press/Addison-Wesley, pp. 13-18.
Picard RW. (1997) Affective Computing. Cambridge: MIT.
Poole A., Ball L. J., Phillips P. (2004) In Search of Salience: A Response-Time and Eye-Movement Analysis of Bookmark Recognition. Proceedings of HCI 2004. People and Computers XViii—Design for Life (eds S. Fincher, P. Markopolous, D. Moore, R. Ruddle), London: Springer-Verlag Ltd, pp. 363-378.
Qin H., Pei-Luen P.R., Salvendy G. (2007) Player Immersion in the Computer Game Narrative. Proceedings of the 6th International Conference nEntertainment Computing—ICEC2007a (Shanghai, China, September 15-17, 2007), pp. 458461.
Ravaja N., Saari T., Turpeinen M., Laarni J., Salminen M., Kivikangas M. (2006) Spatial Presence and Emotions during Video Game Playing: Does It Matter with Whom You Play? Presence: Teleoperators & Virtual Environments, vol. 15, no 4, pp. 381-392.
Renshaw J. A., Finlay J. E., Tyfa D, Ward R. D. (2004) Understanding Visual Influence in Graph Design through Temporal and Spatial Eye Movement Characteristics. interacting with Computers, vol. 16, iss. 3, pp. 557-558.
Riva G., Mantovani F., Capideville C.S, Preziosa A., Morganti F, Villani D., Gaggi-oli A., Botella C., Alcaniz M. (2007) Affective Interactions Using Virtual Reality: The Link between Presence and Emotions. CyberPsychology & Behavior, vol. 1, no 1, pp. 45-56. http://doi.org/10.1089/cpb.2006.9993
Roussou M. (2009) A VR Playground for Learning Abstract Mathematics Concepts. IEEE Computer Graphics and Applications, vol. 29, no 1, pp. 82-85. DOI:10.1109/ MCG.2009.1
Roussou M., Oliver M., Slater, M. (2006) The Virtual Playground: An Educational Virtual Reality Environment for Evaluating Interactivity and Conceptual Learning. Virtual Reality, vol. 10, no 3, pp. 227-240. DOI:10.1007/s10055-006-0035-5 Rutten N. P.G., van Joolingen W., van der Veen J. T. (2012) The Learning Effects of Computer Simulations in Science Education. Computers & Education, vol. 58, no 1, pp. 136-153. https://doi.org/10.10167j.compedu.2011.07.017 Salzman M. C., Dede C., Loftin R. B., Chen J. (1999) A Model for Understanding How Virtual Reality Aids Complex Conceptual Learning. Presence: Tel-eoperators and Virtual Environments, vol. 8, no 3, pp. 293-316. http://doi. org/10.1162/105474699566242 Selivanov V., Selivanova L. (2014) Virtualnaya realnost kak metod i sredstvo obu-cheniya [Virtual Reality as a Method and Means of Learning]. Obrazovatelnye tekhnologii i obshchestvo, vol. 17, no 3, pp. 378-391. Slater M., Sanchez-Vives M.V. (2016) Enhancing our Lives with Immersive Virtual Reality. Frontiers in Robotics and AI, vol. 3, December, Art. no 74. Steinhauer S. R., Boller F., Zubin J., Pearlman S. (1983) Pupillary Dilation to Emotional Visual Stimuli Revisited. Psychophysiology, vol. 20, pp. 472. Sterman M. B., Mann C. A. (1995) Concepts and Applications of EEG Analysis in Aviation Performance Evaluation. Biological Psychology, vol. 40, no 1-2, pp. 115-130. Sweller J. (1998) Cognitive Load during Problem Solving: Effects on Learning. Cognitive Science, vol. 12, iss. 2, pp. 257-285. https://doi.org/10.1207/ s15516709cog1202 4
Tyng Chai M., Hafeez U.Amin, Mohamad N. M. Saad, Malik Aamir S. (2017) The Influences of Emotion on Learning and Memory. Frontiers in Psychology, vol. 8, Art. no 1454.
Valenza G., Lanata A., Scilingo E. P. (2012) The Role of Nonlinear Dynamics in Affective Valence and Arousal Recognition. IEEE Transactions on Affective Computing, vol. 3, no 2, pp. 237-249. DOI: 10.1109/T-AFFC.2011.30 Wang R., Newton S., Lowe R. (2015) Experiential Learning Styles in the Age of a Virtual Surrogate. Archnet-IJAR: International Journal of Architectural Research, vol. 9, iss. 3, pp. 93-110. https://doi.org/10. 26687/archnet-ijar.v9i3.715 Yarosh O., Kalkova N., Velgosh N., Eremenko Y., Mitina E. (2020) Vizualny neyro-marketing: fundamentalnye i prikladnye issledovaniya [Visual Neuromarketing: Fundamental and Applied Research], Simferopol: Arial. Zenkov L. (2001) Klinicheskaya elektroentsefalografiya (s elementami epileptologii) [Clinical Electroencephalography (With Elements of Epileptology)], Moscow: MEDpress-inform.
Zhang D., Bowman A., Jones C. N. (2019) Exploring Effects of Interactivity on Learning with Interactive Storytelling in Immersive Virtual Reality. Proceedings of the 11th International Conference on Virtual Worlds and Games for Serious Applications (VS-Games) (Vienna, Austria, September 04-06, 2019), pp. 1-8. DOI: 10.1109/VS-Games.2019.8864531 Zhang D., Zhou L., Briggs R., Nunamaker J. F. (2006) Instructional Video in e-Learn-ing: Assessing the Impact of Interactive Video on Learning Effectiveness. Information and Management, vol. 43, no 1, pp. 15-27.
