CC BY
7
RUDN Journal of Language Studies, Semiotics and Semantics
2313-2299 (print), ISSN 2411-1236 (online) 2022 Vol. 13 No. 2 396-415
Вестник Российского университета дружбы народов. Шр^рш^.ш^.ш^шюйю^таткя
Серия: ТЕОРИЯ ЯЗЫКА. СЕМИОТИКА. СЕМАНТИКА
DOI: 10.22363/2313-2299-2022-13-2-396-415 UDC 81'221'42:7.04
Research article / Научная статья
Event Construal in Text and Image. Gaze Behavior and Discourse Interpretations
Maria I. Kiose И
Moscow State Linguistic University,
119034, Russian Federation, Moscow, Ostozhenka, 38 И [email protected]
Abstract. The study explores the combinatorial prevalence effect in Event construal techniques in text and image components of heterosemiotic book pages. We hypothesize that their activity and contingency affect their interpretation, here tested in the oculographic experiment and discourse responses check. To proceed, we develop the parametric system applied for 100 book pages annotation and further statistical analysis. This study reveals the relevance of Truth, Type, Relation, Manageability, Completeness, Instantness, Achievement, Evaluation, Space location, Time location, Repeatability, Cause and effect parameter groups in Event construal in text and image as well as their resonance in concomitant activity. To select the samples serving as stimuli in the oculographic experiment, we apply Principal component analysis, which assigns Uniqueness indices to the samples, here ranging from 0.111 to 0.675, and provides diversity of Event construal techniques to be tested in terms of their interpretation. The results evidence that participants applied different text and image attention distribution patterns with longer fixations on text component in case the image displayed physical contact, static and desirable events. When the creation or destruction events, events-achievement, events located in time or causal events were not present in the text, the participants were more likely to address the image, not the text. Parameter activity also affects the choice of Descriptive, Narrative and Speculative discourse responses, with a restricted number of parameters stimulating Narrative discourse, with a restricted in text and vast in image number ofparameters stimulating Speculative discourse, which evidences in favor oftheir more predetermined and predicted character. Hopefully, the results may be used to predict the interpretation effects and to further cognitive linguistic and semiotic research coordination.
Key words: heterosemioticity, event construal, text and image, construal parameters, oculographic experiment, gaze patterns, descriptive, narrative, and speculative discourse
Article history:
Received: 01.06.2021 Accepted: 01.03.2022
Аcknowledgment. Financing
This research is financially supported by the Russian Foundation for Basic Research, project No. 20-012-00370 "Textual heterogeneity and the factors of its successful reading comprehension" at Moscow State Linguistic University.
©Kiose M.I. ,2022
Q This work is licensed under a Creative Commons Attribution 4.0 International License I^S^KS^« https://creativecommons.org/licenses/by/4.0/
For citation:
Kiose, M.I. (2022). Event Construal in Text and Image. Gaze Behavior and Discourse Interpretations.
RUDN Journal of Language Studies, Semiotics and Semantics, 13(2), 396—415. https:// doi.org/10.22363/2313-2299-2022-13-2-396-415
Особенности конструирования события в тексте и иллюстрации.
Глазодвигательные реакции и дискурсивные интерпретации
М.И. Киосе Э О
Московский государственный лингвистический университет,
119034, Российская Федерация, Москва, ул. Остоженка, 38 О [email protected]
Аннотация. В работе исследуются комбинаторные эффекты в выборе техник конструирования события в гетеросемиотическом единстве текста и изображения на материале иллюстрированных книг. Гипотеза исследования, заключающаяся в том, что активность и сопряженность параметров конструирования события будет определять интерпретацию гетеросе-миотического единства, верифицируется в ходе окулографического анализа и последующего анализа дискурсивных ответов участников. Анализу подвергаются 12 групп параметров: параметры истинности, общего типа, типа отношений, управляемости, завершенности, мо-ментальности, результативности, оценочности, локализации в пространстве, локализации во времени, повторяемости, причинно-следственных отношений; в ходе анализа устанавливается их роль в создании текста и изображения, а также особенности их резонанса, или особенности дублирования их активности в тексте и изображении. Для отбора стимульных образцов используется методика определения их типичности и уникальности с помощью факторного анализа с привлечением показателей параметров конструирования; показатели уникальности установлены в диапазоне от 0.111 до 0.675, что обеспечивает вариативность привлекаемых образцов. В ходе эксперимента выявлено, что продолжительность пребывания взгляда в зонах текста и изображения обусловлена особенностями конструирования события. Увеличение продолжительности фиксаций в зонах текста наблюдается, если событие в изображении представлено как задействующее отношения физического контакта, как статическое и желательное. Если в тексте не представлены событие создания и разрушения, событие-достижение цели, событие, имеющее временные отсылки, событие с выраженными причинными и следственными отношениями, то возрастает продолжительность фиксаций в зоне изображения. Активность параметров также определяет выбор дискурсивных паттернов в дескриптивном, нарративном дискурсах и дискурсе-рассуждении. Установлено ограниченное количество параметров конструирования события, сопряженных с выбором нарративных паттернов; выбор дискурса-рассуждения сопряжен одновременно с ограниченным количеством параметров конструирования текста и с достаточно большим их количеством в изображении. Полученные данные позволяют использовать разработанную систему параметров конструирования события в тексте и изображении для реализации прогностической оценки успешности интерпретации гетеросемиотических единств. Результаты также способствуют интегрированию на единых основаниях достижений лингвокогнитивной и когнитивно-семиотической парадигм к анализу гетеросемиотических единств.
Ключевые слова: гетеросемиотичность, конструирование события, текст и иллюстрация, параметры конструирования, окулографический эксперимент, глазодвигательные паттерны, описательный, нарративный и спекулятивный дискурс
Благодарности. Финансирование:
Исследование выполнено при финансовой поддержке Российского фонда фундаментальных исследований, проект № 20-012-00370 «Гетерогенность текста и факторы ее успешной интерпретации читателем» в Московском государственном лингвистическом университете.
История статьи:
Дата поступления: 01.06.2021 Дата приема в печать: 01.03.2022
Для цитирования:
Kiose M.I. Event Construal in Text and Image. Gaze Behavior and Discourse Interpretations // Вестник Российского университета дружбы народов. Серия: Теория языка. Семиотика. Семантика. 2022. Т. 13. № 2. С. 396—415. https://doi.org/10.22363/2313-2299-2022-13-2-396-415
Introduction
The studies of text and image in cognitive semiotics have recently integrated experimental methods since one of the key areas in heterosemiotic research is now prognostic analysis aimed at predicting how heterosemiotic construal techniques might influence interpretation. Prognostic assessment in the studies follows a series of experiments exploiting various heterosemiotic stimuli displaying diverse construal patterns. As a result, some behavior reactions contingent with the construal patterns are detected, which allows to range the construal patterns as more or less significant in terms of interpretation reactions they cause. Among the most influential experimental methods testing this contingency we will name oculographic analysis since it helps reveal the gaze patterns and the quantitative data on reading techniques susceptible to construal patterns in heterosemiotic unities of text and image, for instance, see [1-3]. Experimental studies in multimodal semiotics (exploring various heterosemiotic forms), although not exploiting prognostic algorythms of interpretation, still succeed in detecting both typical and atypical gaze patterns of experiment participants perceiving the stimuli diverse in heterosemiotic construal. It is noteworthy, that these results are used in applied research and technology, for instance in gamification, publishing industry, educational literature design.
However, in most cases these are the semiotic characteristics of text and image unities that enjoy experimental research, among them we might name the occupancy space of text and image, the color techniques, the order of appearance [4; 5], the combination techniques like domineering, coordination, and reduplication [6; 7], the semantic types of information presented, for example its denotative or connotative character, participants type, dynamic or static type [8]). It is worth mentioning that the pre-experiment analysis of heterosemiotic construal techniques is seldom carried out and even in this case it is most restricted to discourse type studies [9] or communicative functions [10]. Therefore, cognitive construal techniques with participants, events or perspective construal techniques are rarely considered, although they might as well hamper or stimulate interpretation of heterosemiotic unities.
In this study, we test the event construal techniques in text and image, and later explore their effect onto the experiment participants' gaze patterns. We then proceed to studying the discourse patterns which the participants employed when they gave account on the
information they had received while reading. It is expected that diverse event construal techniques will affect the gaze patterns and the discourse patterns employed, although it is hard to predict which construal techniques will stimulate the diversity in cognitive load redistribution. Experimental studies mostly use monomodal stimuli, still among the most influential factors determining the increase in cognitive load expressed for instance in longer fixation duration and return saccadic movements they name the factors of salience (mostly expressed in the construal techniques frequency) and focusing (focalization of construal techniques), for instance, see [11-13]. In case of event construal techniques, we can also distinguish between more and less focal techniques, cf. achievement and non-achievement events, instant and durative events, intersubjective and mono-participant events, where the first event in each group is more focal, although the effects of salience will be applicable only in context. The research question is whether these more focal event construal techniques will affect the gaze behavior and interpretation responses. Therefore, the effects of combinatorial prevalence are explored in this study.
Event construal techniques in text and image
Event construal techniques have received attention in both cognitive semiotics [14-17] and in cognitive linguistics [18-21], among the key areas of study are dynamicity, aspectuality and entrenchment as well as embodiment construal types. Event typologies are thoroughly discussed in cognitive linguistics, however they cannot in all cases be easily adapted to conform to other semiotic systems analysis, still they may suffice to develop the event typologies applicable to heterosemiotic systems. It is worth mentioning, that event construal studies originated with the famous work of Z. Vendler who differentiated between events-achievements, accomplishments, states, and activities. Vendler's work [22] has become highly resonant in linguistics, especially in logical semantics. Following Vendler, N. Arutunova [23] and E. Paducheva [24] give a more detailed and linguistically oriented accounts on event types. Communicative and discursive characteristics of events were mostly explored in European and American studies of the 1980s, for instance, see [25; 26]. In later theories event construal becomes associated with participants' construal and their relations; this approach is a key one in construction grammar [27].
In this research, we will mostly rely on semiotic approaches to linguistic analysis which pave the way to interpretation theories and allow to synchronize several semiotic modes on the same grounds. Semiotic foundations for modes synchronization on linguistic terms received attention in the works of L. Novikov in his theory of ornamental field accumulating different means of poeticism construal [28] and in disclosing dominant structures in composition [29]. These dominant structures creating an ornamental field comprised objects and participants, their actions, inner world construal, event location in time and space, poeticism construal. However, these components need specification to be analyzed in heterosemiotic forms. The most influential typological research in this area is the study presented by V. Demyankov
who develops the notion of interpretation coordinates (or landmarks) [30] which may serve to develop the procedure of parametric analysis of event construal in different semiotic systems, not restricted to linguistic system. Among these landmarks, V. Demyankov names stativity / dynamicity, manageability (controllability) / non-manageability, integrity / non-integrity, momentarily / durative character, repeatability / uniqueness, achievements / non-achievements [30. P. 323-328].
The interpretation value of these landmarks lies in their more and less focusing character for the interpreter, it may help assess (and predict) the interpretation effects produced on the viewer (reader). For instance, the landmark "event-achievement" is more focal than "non-achievement" as it lacks result or action outcome, however in some cases the decision on focal character depends on the reader's expectations as well as on the event construal type. Therefore, the character of manageable or non-manageable event, for instance, may be interpreted as focal or non-focal depending on whether it conforms to the macroevent (discourse event) expected structure or is totally unexpected. That explains our decision to address the parameters which might be both focusing and defocusing in the context of heterosemiotic construal. The annotation parameter system with the coded parameter values is given in Table 1. This system will be applied to annotate both text and image components of heterosemiotic unity.
The parameter code values start with 1 for annotating text and with 2 for annotating image. Applying two types of codes allows to conduct statistical processing of two data sets. For instance, annotating Sample 1 (see Fig. 1) we may select the following event construal parameters for image construal: True, veritable, real event (201), Physical contact, perception event (205), Intersubjective event (207), Manageable, planned event (211), Incomplete, interrupted event (213), Continuous event (215), Purposeless, aimless event (217), Event lacking evaluation (220), Event located in space (221), Event without location in time (223), Sporadic, unitary event (226), Descriptive event (228).
¿1
Качели • •
i HP # • <fl
52
Fig. 1. Sample 1
Рис. 1. Образец 1
Table 1/Таблица 1
Event construal parameters / Параметры конструирования события
Parameter groups/ группа Parameters/napaMeTpbi Codes/ коды
Truth True, veritable, real event 101/201
Untrue, fictional, unreal event 102/202
Type Creation, building, molding / demolishing, destruction event 103/203
Shifting, changing event 104/204
Physical contact, perception event 105/205
Static event 106/206
Relations Intersubjective event 107/207
Interactional event (between man and object) 108/208
Event not displaying any relations 109/209
Manageability Non-manageable, spontaneous, accidental event 110/210
Manageable, planned event 111/211
Completeness Completed, resultant event 112/212
Incomplete, interrupted event 113/213
Instantness Instantaneous event 114/214
Continuous event 115/215
Achievement Event-achievement 116/216
Purposeless, aimless event 117/217
Evaluation Desirable event 118/218
Undesirable event 119/219
Event lacking evaluation 120/220
Space location Event located in space 121/221
Event without location in space 122/222
Time location Event located in time 123/223
Event without location in time 124/224
Repeatability Repeatable event 125/225
Sporadic, unitary event 126/226
Cause and effect Causal event 127/227
Descriptive event 128/228
Annotating its text component, we observe that in some cases events display a different character. The event is also given as True, veritable, real (101) as the text displays a sequence of events which might be real (the event on the children's playground), however it is Shifting, changing (104) as we see a chain of subsequent microevents employing a change in scene without creating or destroying any event components. The event is shown as Intersubjective (107) which activates several participants, grandma, children, grandma's own grandson, at the same time as opposed to the event in the image, it is Manageable, planned (111) which describes the deliberate action of grandma presenting it in detail. We may conclude that the text shows a Completed, resultant event (112) as it is a completed narrative with all narrative components described in [31], Orientation in the first two sentences presenting the scene and its participants and objects, Complicating Action in the third sentence presenting assessment of the grandma's action, although the action itself will be described later. Therefore, the sentences five and six display an amalgam of Complicating Action in Grandma put a 8-kilo weight into her grandson's pocket and Resolution in Her grandson turned out to be the heaviest of all children. Coda in the final sentence Then Grandma boasted that they give the best food to their grandson allows to present the Grandma's explanation for what has happened. The event is Continuous (115) as the actions described are mostly of unbounded aspect type [32] in competed, behaved, overweighted, boasted (cf. with the actions like won, tore, killed, eaten or similar). At the same, we encounter Event-achievement (116) because the reasons for the grandma's action are explained and the desired result is achieved. The event is classified as Undesirable (119) as this action is justified as dishonest, although explicitly. It might have been classified as desirable if the text perspective were the one that is construed by the grandma herself, however it is not so. The event is located in space (121) as there are several objects which help to orient it (swings, pocket), however, this is Event without location in time (124) because there are no time identifiers apart from the time identifier then in the final sentence, but this one co-locates two microevents [33] and does not allow to locate the whole event in time. The event is Sporadic, unitary (126) as the action is not repeated, and Causal (127) because the reasons for such a dishonest action are clearly presented.
Therefore, the system elaborated allows to provide annotation in both text and image and to analyze them on the same grounds, which can further be used for statistical processing and exploring the techniques of event construal specifics developed in text and image in contrast.
Methods and procedure
The study applies the triangulation methodological procedure [34], here it comprises the methods of cognitive semiotic modelling (parametrization), annotation and statistical methods to explore the distribution and contingency
techniques in event construal in text and image, and cognitive psychological study (oculographic experiment) testing the contingency between construal and interpretation techniques and assessing role of combinatorial prevalence factor. Below, I will briefly describe these three research steps.
Parametrization procedure was described in detail above, it includes the elaboration of sufficient parameter system applicable to both text and image construal analysis. It involves 12 groups of event construal parameters which have coded values for further annotation and statistical processing.
Annotation was carried in UAM Image Tool (www.corpustool.com) which allows to conduct annotation at various levels and annotate image and text separately. The research data are 100 book samples presenting the fragments of text accompanied with illustration, addressed to adults and children. The basic criterion for samples selection was the criterion of diversity, that is the diversity in author and illustrator range, period of issue, style of writing and artistic style, genre of literature, addressee age group. The samples were selected from the works of A. Chekhov, B. Akunin, I. Ilf and Ye. Petrov, S. Marshak, N. Gogol, A. Barto, L. Kaminsky, F. Nonn, E. Tarle, Z. Turlonnas, among others, with illustrations made by Kukryniksy, L. Vladimirsky, V. Chizhikov, R. Bogdanova, I. Oleynikov, A. Bekker, A. Petrova, K. Boldt, V. Vidali, among others. In Figure 2 we demonstrate the structure of Event level and the annotation procedure of the Sample 1 given above.
Fig. 2. Event level structure and annotation procedure in UAM Image Tool Рис. 2. Структура уровня события и процедура аннотации в UAM Image Tool
There is specificity of annotating in UAM Image Tool which is the necessity to select only one parameter within the group, it explains why we applied paired parameters (and not only one of them) within the parameter system.
The results were further exported and statistically processed in specially designed software HETEROSTAT [35], which allows to receive the activity values of each parameter and each parameter group as well as the correlation values of every pair of parameters. We then may contrast the parameter distribution in text and image and explore the event construal techniques prevailing in text or in image. This analysis is carried out adopting the methods of vector semantic analysis [36-38] which helps detect the contingency between the multi-vector profiles of text and image event construal. With the contingency results we observe the differences between the text and image text construal profiles.
Apart from contingency analysis, we conduct the resonance analysis applying the procedure of overlapping the annotation results in text and image event construal. The procedure involves the search for the parameters with the same activity in text and image; in case a parameter is active in both matrices, it is marked as active in the heterosemiotic unity, in case it is active in text or image only, it is marked as inactive.
To select the samples for further oculographic analysis testing interpretation techniques of event construal in text and image, we conduct factor analysis (principal component analysis as one of its version) in JAMOVI program (https:// www.jamovi.org) to receive typicality and uniqueness values of each sample. Only the samples with varied uniqueness indexes, low and high, should be selected due to their different event construal techniques. In case we detect the changes in eye movement behavior demonstrated by the experiment participants which are contingent with any of the samples, we can easily deduce which event construal parameters, or the combinations of event construal parameters might have caused them. We finally select 6 samples (presented in Open Access web-resources) with different uniqueness index to serve as experiment stimuli (see Figure 3).
The paired samples 1-6 were further used in the experiment which tested the viewers' oculographic reactions and linguistic reports in terms of the discourse strategies used, which were descriptive, narrative, and speculative. We hypothesized that different Event construal patterns will correlate with specific eye-movement strategies and discourse strategies, which will allow ranging the Event construal operations in terms of their significance for interpretation. 16 participants (students, average age 22) were supposed to view 6 samples, after each sample there was a blank screen for 15 seconds, during this time the participants gave their reports on what they have seen. Their eye movements as well as their reports were recorded during stimuli presentation. Only the reactions of 11 participants were finally considered as in other 5 participants' eye movement or discourse interpretation reports not all 6 samples were presented adequately.
Fig. 3. Experiment stimuli 1-6 Рис. 3. Стимулы эсперимента 1-6
The SMI Red-x eye tracker was applied in the experiment (binocular system, frequency = 60 Hz, accuracy = 0.4°, head movement 40x20 cm, operating distance = 60-80 cm). Eye movement data during task performance were sampled (fixation and saccade duration, dwell time on the areas of interest); subjects' free reports were recorded and analyzed in terms of using Descriptive, Narrative, and Speculative discourse strategies. To analyze the discourse strategies and to decide in favor of the strategy adopted we relied on the discourse markers. For Narrative discourse we considered the tense of verbs (mostly present for Descriptive discourse, mostly past or tense shifts for Narrative discourse), the use of conjunctions and adverbs of time, the use of bodily verbs like see and look as well as of actional verbs like make and take. For Descriptive discourse we considered the tense of verbs, the use of parallel clauses, objects and participants naming in parallel structures. For Speculative discourse we looked for the use of modal verbs and words with the semantics of various modality types, the use of word with the semantics of evaluation, and the use of non-referential words, among other markers.
We hypothesized that this triangulation procedure will reveal contingent event construal techniques, eye movement techniques and discourse construal techniques, which will allow to range the significance of event construal parameters for interpretation.
Results
Annotation and statistical analysis
I will proceed with the results of annotation and statistical processing of the received data and then present the results of the oculographic experiment. Following the annotation procedure, we explored parametric activity in event construal in text and image. HETEROSTAT program window is given in Figure 4.
HETEROSTAT: Event construal in Text and lmage.txt
72 201 203 207 211 213 215 216 21Э 221 224 226 228 101 103 107111 112115116119 1 21 123 1 26 73201 204 203 210 213 215 217 220 222 224 225 223 101 103 107111 112114116118 122 1 23 1 25
74 2D2 204 207 210 213 215 217 218 221 224 22Б 228102 104 107 111 112114117118121 123126
75 101 201 205 200 210 213 215 217 220 222 224 225 228 1 01 106 107111 112114116 120 1 22 1 24
76 202 204 208 211 212215217 220 221 224 226228102104 107111 112115116119122124126 77101 202 204 208 210 212 215 217 218 222 224 228 223101 106108111 112114117120122124
78 201 20S 209 210 212 215 217 220 221 224 226 228 101 104 108111 112115116 120 121 124 1 26
79 202 205 207 210 212215217 220 221 224 226 228 101 105 107110113115117116 1 22 123 1 25
80 101 201 206 203 210 213 215 217 220 222 224 228 228 1 02 105 108110113115117 1 20 122 1 24
81 201 205 207 211 212 215 217 220 221 224 226 228 101 105 107111 112115117118 1 22 123 1 25 82101 201 204 203 210212215217 220 221 224 226228101 106109111 113115117118121 124 83 201 204 207 211 212 215 216 220 221 224 226 228 101 105 107110113115117119 1 22 1 23 1 26 84101 202 205 207 210 212 215 217 220 221 224 225 228101 106107111 113114117118122124 85 1 01 201 206 208 210 212215217 220 221 224 228 228 1 01 106 107111 112114116 1 20 122 1 23 86101 202204 208211 213215217220 222224 226 228101 106107111 113114116120122123
87 202 204 207 211 213 215 216 220 221 224 226 228 102 1 04 107111 112115117 120 1 21 124 1 25
88 202 205 209 211 212 215 217 220 221 224 226 228 101 105 108110112114117 120 122 124 1 26
89 202 204 207 211 212215216 220 221 223 225 227102 1 04 107110112115116118 122 124 1 25
80 202 205 207 210 212 215 217 220 221 224 226 228 101 106 108111 113115117118 1 22 124 1 25
81 202 206 209 210 212 215 217 220 221 224 225 228 101 104 108110112115117118 121 123 1 25 92202204 207 210212215217220 221 224 226228101 104 108111 112115117120121 124125 93 201 204 207 210212 215216220 221 223 226 227 101 104 108110113115117118121 123126 34101 201 205 207 210 212 215 217 220 221 224 228 228101 104107111 112114116118122124
Sír Upload Dala | */ Chect: Dala | ^ Piocess Dala | Data corrected
128 127
127
125128
128 126128 128 128 126128 127 126128
127 126128 125128 126128
128 128 128 128 128 128 127
125128
□ X Code Tuning
128 228
Range Nmin = |1 Nrnax- [TOO
•III* Default
Й Save Code
Br Upload Code
Fig. 4. HETEROSTAT window processing the data on Event construal Рис. 4. Окно программы HETEROSTAT, обрабатывающей данные о конструировании событий
The samples numbers are given on the left, each sample demonstrates its parameter codes, the codes for event construal in text starting with 1, and the codes for event construal in image starting with 2. Parameter activity results helped reconstrue the text and image profiles of Event construal in the samples, see Figure 5.
Fig. 5. Text and Image profiles of Event construal Рис. 5. Профили конструирования события в тексте и изображении
Pearson correlation test to test profiles contingency revealed that r(26)=0.66, with critical r-values for p=0.01 equal to 0.48, and for p=0.05 the critical r-value is 0.37. High correlation value evidences that in general the direction of vector activity in text and image is similar, however, this analysis does not consider the values themselves. More detailed t-statistics (Paired samples T-test) revealed very low significance of contingency data in text and image, with t(27)=0.017, p=0.99. It shows that the techniques of Event construal are diverse in text and image. To find out which parameters display higher variance, we conducted One sample T-test. It revealed two parameters whose values are relatively similar, these are 107/207 with t(1)=10.83, p=0.059, and 111/211 with t(1)=11.5, p=0.055. This means that in most cases intersubjectivity and manageability in Event construal are reduplicated in text and image, probably for the sake of their focalization. Among the parameters with higher variance, we should name 114/214 with t(1)=1.24, p=0.431, 122/222 with t(1)=1.94, p=0.302, 119/219 with t(1)=2.14, p=0.278, 109/209 with t(1)=2.2, p=0.272, 123/223 with t(1)=2.26, p=0.265, 127/227 with t(1)=2.26, p=0.265. They are Instantaneous event with higher values in text, Event without location in space with higher values in text, Undesirable event with higher values in text, Event not displaying any relations with higher values in image, Event located in time with higher values in text, Causal event with much higher values in text. This means that the text component is more responsible for Event construal techniques, although these are several techniques displaying higher activity in image construal, with Incomplete, interrupted event, Continuous event, Event lacking evaluation, Event located in space, Event without location in time, Sporadic, unitary event, Descriptive event among them.
What is of importance, is the contingency values of Event construal parameters. With critical r(54) of 0.34 at p=0.01 and 0.26 at p=0.05, there is a large number of those which display contingency, however in almost half of the cases this contingency is the result of the specifics of parameter annotation system which allows only one choice at each parameter group and therefore, the parameters work in counteraction. Considering this fact, I will give single parameters (in case there is counteraction) or parameter groups (in case there are more than two parameters representing each group) which demonstrate coordination prevalence in text and image construal. Among parameter groups I will name the event type, with high correlation values. We observe them in the correlation pairs including the parameter Shifting, changing event in image and text, for instance in the correlations of this parameter in image and Intersubjective event in text (r(54)=0.68), this one and Manageable, planned event in text (r(54)=0.71), this one and Completed, resultant event in text (r(54)=0.68). Among single parameters of image having rigid correlations with text parameters are Continuous event
(True, veritable, real event with r(54)=0.79, Manageable, planned event with r(54)=0.83, Completed, resultant event with r(54)=0.89), Purposeless, aimless event (True, veritable, real event with r(54)=0.75, Intersubjective event with r(54)=0.72, Manageable, planned event with r(54)=0.77, Completed, resultant event with r(54)=0.77, Sporadic, unitary event with r(54)=0.69, Descriptive event with r(54)=0.7). Displaying lower, but still significant correlations are the parameters Event located in space, Event without location in time, Descriptive event. Overall, we observe almost strict regulations in Event construal parameter distribution in text and image, with continuous changing events located in space lacking evaluation in image, and resultant causal purposeful manageable events located in time in text.
However, it is worth mentioning how text and image can comply to enhance the construal specificity of one and the same parameter or parameters. The resonance analysis applying the procedure of overlapping the annotation results in text and image event construal showed that in some cases parameter activation occurs simultaneously in text and image, obviously to enhance the information transferred. In Figure 6 I present the parameter activity results in text, image and the resonance values.
Event construal parameter values
100 90 80 70 60 50 40 30 20 10 0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 ■ Test ■ Image Resouauce
Fig. 6. Contrastive data on parameter values in text, image, and their resonance values
Рис. 6. Сопоставление данных о значениях параметров в тексте, изображении и их резонансных значениях
In Figure 6 the parameter codes are presented in the unified way which allows to consider text, image and resonance parameters on the same grounds. The diagram shows that in some cases the resonance values are almost similar to the values of the parameters in text or image. Among them are the events displaying Truth, Shifting, changing event, Completed, resultant event, Continuous event (almost all cases
1. In, 1, 1,
of continuous event construal in text are reduplicated in image), Purposeless, aimless event, Event located in space (almost all cases of continuous event construal in text are reduplicated in image), Sporadic, unitary event. Therefore, we may conclude that these event types are principal for both text and image event construal.
To select the samples for further oculographic analysis testing interpretation techniques of event construal in text and image, I conducted principal component analysis and received typicality and uniqueness values of each sample. Factor analysis revealed 7 principal components within the data with the Uniqueness indices ranging from 0.111 to 0.675. Among the samples I then selected the ones displaying high uniqueness, thus being atypical and presumably requiring cognitive load increase in their interpretation, and the ones quite typical which might not cause interpretation problems. I hypothesize that their eye-movement and discourse interpretations will be different.
Oculographic experiment and discourse responses analysis
Next, I proceed with the results of oculographic experiment and discourse responses analysis.
6 samples were selected for the experiment, with the uniqueness indices in Event construal ranging from 0.23 to 0.57. They were presented to the participants one after another, however, they fall into 3 groups, demonstrating similarity in several parameters activity and distinct differences in others'.
First, I conducted contingency analysis with each Event construal parameter and eye movement reactions, here the contingency results of fixation duration within the areas of text and image. Overall, there were 54 probes, 8 probes of participants could not be considered due to inconsistent gaze paths received. In Table 2 Descriptive data on eye movement measures are presented.
Table 2 / Таблица 2
Eye movement measures of text and image AOIs / Показатели движения глаз в текстовых и графических зонах интереса
Stimuli 1-6
Mean 41.9
Text AOIs fixation duration (ms) Median 45.7
Shapiro-Wilk p <0.001
Mean 58.1
Image AOIs fixation duration (ms) Median 54.3
Shapiro-Wilk p <0.001
As seen, Shapiro-Wilk analysis reveals that the data have normal distribution, therefore I apply Pearson correlation procedure for contingency analysis. Pearson correlation analysis disclosed several parameters displaying rigid correlation with fixation duration distribution within the areas of text and image (at p=0.01 critical r(53)=0.35). The role of Event construal in the text was assessed first and the parameters contingent with gaze patterns distribution were detected. They are Event type featuring Creation, building, molding / demolishing, destruction event with r(53)=-0.45, Physical contact, perception event with r(53)=0.52, and Static event with r(53)=0.54, Event not displaying any relations with r(53)=-0.54, Event-achievement with r(53)=0.-45, Desirable event with r(53)=0.45, Event lacking evaluation with r(53)=0.45, Event located in time with r(53)=0.-45, Causal event with r(53)=0.-45, Descriptive event with r(53)=0.45. The results evidence that participants had text and image attention distribution in case the image displayed events showing physical contact, the event was static and desirable — under these circumstances the attention on the text component was longer as if helping to reconstruct the missing information. When the creation or destruction event, event-achievement, event located in time or causal event was not present in the text, the participants were more likely to address the image, not the text.
Next, I assessed the role of Event construal in the image. It is worth mentioning that the number of parameters contingent with statistically significant gaze patterns redistribution is twice smaller, which evidences that in general event construal by means of the image is less expected, its expected role is much higher in the text component. Among the Event construal parameters contingent with attention distribution were Intersubjective event with r(53)=0.45, Event without location in time with r(53)=-0.66, Desirable event with r(53)=0.52, Event lacking evaluation with r(53)=-0.64, Event located in time with r(53)=0.52.
Therefore, if the information load typical of the text component is shared in the image component it is the image that attracts longer fixations, perhaps because in this case the event construal is less typical and demands more concentration, the second explanation although is that the image component is enough to reconstrue the information. There are several parameters of event construal whose role is decisive in attention distribution between text and image components and whose activity affects both text and image construal. They are Relation construal in terms of Intersubjectivity, Event location in time and Desirable event, with the last one working differently. In case of its presence either in image or in text it makes the fixation duration longer. We might conclude that the presence of evaluation in all cases affects the fixation duration increase.
Second, I proceed with the discourse responses analysis.
The same correlation analysis procedure was carried out to see where the choice for discourse types applied (descriptive, narrative, and speculative)
was contingent with any of the Event construal parameters. The results have revealed that Descriptive discourse responses were stimulated (being highly contingent) by the presence of the following parameters in the text construal: Shifting, changing event with r(53)=0.92, Interactional event (between man and object) with r(53)=0.46, Manageable, planned event with r(53)=0.92, Purposeless, aimless event with r(53)=0.92, Event lacking evaluation with r(53)=0.92, Event without location in space with r(53)=0.46, Event without location in time with r(53)=0.44, Sporadic, unitary event with r(53)=0.44, Descriptive event with r(53)=0.92. Narrative discourse responses were stimulated by the following parameters in the text construal: Creation, building, molding / demolishing, destruction event, Non-manageable, spontaneous, accidental event, Event-achievement, Desirable event, Event located in time, Causal event, all with r(53)=0.75, Repeatable event with r(53)=0.53. Speculative discourse responses were stimulated by the following parameters in the text construal: Interactional event (between man and object), Event without location in space, both with r(53)=0.46, Repeatable event with r(53)=0.62. Thus, the choice of Speculative discourse responses is of a more individual character, although the results suffice to assume which Event construal parameters are highly provocative.
Now, I will report the results on the discourse responses contingent with Event construal in the image. Descriptive discourse responses were contingent with the presence of the following parameters: Shifting, changing event with r(53)=0.49, Intersubjective event with r(53)=0.92, Manageable, planned event and Completed, resultant event, both with r(53)=0.44, Continuous event with r(53)=0.49, Undesirable event, Sporadic, unitary event and Causal event, all with r(53)=0.44. Narrative discourse responses were stimulated by the presence of following parameters: Event not displaying any relations with r(53)=0.75, Non-manageable, spontaneous, accidental event, Incomplete, interrupted event, Event lacking evaluation, Repeatable event, and Descriptive event, all with r(53)=0.53. Speculative discourse responses were contingent with the following parameters: Shifting, changing event with r(53)=0.46, Non-manageable, spontaneous, accidental event and Incomplete, interrupted event, both with r(53)=0.62, Continuous event with r(53)=0.46, Event lacking evaluation, Repeatable event, and Descriptive event, all with r(53)=0.62.
Contrastive analysis of contingency results demonstrates that Descriptive discourse responses are stimulated by different Event construal parameters in text and image, and there are a lot of parameters displaying rigid correlation with the choice of this response type, whereas Narrative discourse responses are stimulated by partially the same parameters in text and image, and their number is significantly smaller. It might mean that the choice of Narrative discourse responses is rather predetermined and much easier predicted in case several distinct parameters of Event construal were present. The situation with Speculative discourse responses
choice is more complicated. I have detected a vast number of Event construal parameters in the image which stimulate the use of Speculative responses, whereas in the text these parameters are few. Besides, the image parameters contingent with Speculative discourse responses display non-manageable, continuous, incomplete character, thus lacking focusing potential, which in turn might have affected the choice of Speculative discourse.
Concluding remarks
The research aimed at testing Event construal specifics in the heterosemiotic format of book page has allowed to reveal the construal techniques in text and image, verify their interpretational value, and disclose the discourse response types contingent with these techniques.
The results conform with the experimental findings reported in [Kaspar, König 2011; Kirtley 2018] in the part concerning the types of parameter activity distribution in text and image components in heterosemiotic formats. However, in our study we do not focus on the synchronization techniques of domineering, coordination, and reduplication revealed in their works, but explore the contingency of Event construal parameters in text and image. The key issue is the techniques themselves in terms of their interpretation. The research has proved that the factors of focusing and typicality play a less significant role in heterosemiotic formats of information, compared with monosemiotic formats studied in terms of salience effects [Giora 2003], the effects of attractors [Gibbs, Tendal 2006], and entrenchment effects [Schmid 2016]. The study has explored the factor of combinatorial prevalence which adds to the effects of focusing and typicality.
The study has proved that Event construal parameters explored in [Tenny, Pustejovsky 2000] to describe text construal specifics can be applied to analysis of images with the idea of exploring both semiotic forms on the same grounds. Apart from that, several of these construal parameters demonstrate high interpretation value, at least in terms of text and image heterosemiotic construal. This approach integrating construal as both information generation and information processing allows to detect and verify the construal techniques outlined in cognitive linguistics and semiotics. However, the idea of their integrating in application to Event construal analysis is not a new one. The parameters bearing relevance to the interpretation were explored in [Demyankov 1983], but in his study mostly morphological and syntactic techniques were deduced, with the non-verbal parameter groups left unverified, although they were specified.
Among the most important findings of this study are the parameter activity and contingency results in Event construal in text and image, with the system of 28 parameters developed and applied for annotation and statistical processing. Their coordination and reduplication effects as components of combinatorial prevalence
were also disclosed, which may pave the way for prognostic fundamental and applied studies assessing the role of these parameters or aiming to devise a successful (with the readers) heterosemiotic format.
Parameter activity and contingency results served to detect the uniqueness and typicality effects among the data samples, which in their turn later became stimuli in the oculographic experiment accompanied by comprehension checks aimed at exploring the interpretation value of diverse construal techniques. The results suffice to state that the presence of several parameters in text or image is highly resonant with the gaze reactions of the participants, moreover, the gaze patterns are also dependent on the semiotic format in which these parameters are activated, which suffices to specify different interpretation roles of text and image. The experiment also included the study of participants discourse reactions, which proved highly resultant, since it revealed how the choice of discourse responses is stimulated.
Overall, hopefully the procedure developed and the results achieved may be used to predict the interpretation effects, select the proper construal techniques to achieve the necessary result, and help verify cognitive linguistic and semiotic data.
References
1. Itti, L. (2005). Quantifying the contribution of low-level saliency to human eye movements in dynamic scenes. Visual Cognition, 12 (6), 1093-1123.
2. Komogortsev, O. & Khan, J.I. (2006). Perceptual attention focus prediction for multiple viewers in case of multimedia perceptual. In: ETRA 2006 — Proceedings of the Eye Tracking Research and Application Symposium, Räihä K-J., Duchowski A.T (eds.). California: San Diego. pp. 101-108.
3. Wilming, N., Onat, S., Ossandon, J.P., Agik, A., Kietzmann, T.C., Kaspar, K., Gameiro, R.R., Vormberg, A. & König, P. (2016). An extensive dataset of eye movements during viewing of complex images. Scientific Data, 4. URL: https://www.nature.com/articles/sdata2016126. pdf (accessed: 12.05.2020).
4. Lautenbacher, O.P. (2012). From still pictures to moving pictures: eye-tracking text and image. In: Eye Tracking in Audiovisual Translation, E. Perego (Ed.). Roma: Aracne Editrice. pp. 133-154.
5. Antipov, V.N. & Zhegallo, A.V. (2017). Three-dimensional perception of flat images in computerized environment. Experimental Psychology, 7(3), 97-111.
6. Kaspar, K. & König, P. (2011). Viewing behavior and the impact of low-level image properties across repeated presentations of complex scenes. Journal of Vision, 11, 1-29.
7. Kirtley, C. (2018). How images draw the eye: an eye-tracking study of composition. Empirical Studies of the Arts, 36 (1), 41-70.
8. Steptoe, W., Oyekoya, O. & Steed, A. (2010). Eyelid kinematics for virtual characters. Journal of Visualization and Computer Animation, 21 (3), 161-171.
9. Voroshiliva, M.B. & Zlokazov, K.V. (2017). Psychological methods of Creole texts study. Political Science, 2, 126-137. (In Russ.).
10. Baldry, A.P. & Thibault, P. J. (2006). Multimodal Transcription and Text. London: Equinox.
11. Giora, R. (2003). On our mind: Salience, context, and figurative language. Oxford: Oxford University Press.
12. Gibbs, R. & Tendal, M. (2006). Cognitive effort and effects in metaphor comprehension: Relevance theory and psycholinguistics. Mind and Language, 21(3), 379-403.
13. Schmid, H-J. (ed.) (2016). Entrenchment and the psychology of language learning. How we reorganize and adapt linguistic knowledge. Washington DC: Mouton de Gruyter.
14. Merleau-Ponty, M. (2005). Phenomenology of perception, C. Smith (transl.). London: Routledge.
15. Clark, A. (2006). Language, embodiment and the cognitive niche. Trends Cogn. Sci., 10, 370-374.
16. Danziger, E. & Rumsey, A. (eds.) (2013). Intersubjectivity across Languages and Cultures. Special issue of Language and Communication. Amsterdam: Elsevier,
17. Sinha, C. (2015). Language and other artifacts: Socio-cultural dynamics of niche construction. Frontiers in Psychology, 6. URL: https://www.frontiersin.org/articles/10.3389/ fpsyg.2015.01601/full (accessed: 12.01.2021).
18. Pustejovsky, J. (1995). The Generative Lexicon. Cambridge, Massachusetts: MIT Press.
19. Talmy, L. (1996). Fictive motion in language and caption. In: Language and Space, P. Bloom, M. Peterson, L. Nadel, and M. Garrett (eds.). Cambridge, MA: MIT Press. pp. 211-276.
20. Jackendoff, R. (2002). Foundations of language: Brain, meaning, grammar, evolution. Oxford: Oxford University Press.
21. Bergen, B. (2015). Embodiment. In: Handbook of Cognitive Linguistics, E. Dabrowska, D. Divjak (eds.). BerlIn: Walter de Gruyter. pp. 10-30.
22. Vendler, Z. (1967). Facts and Events. In: Linguistics in Philosophy. Ithaca, NY. pp. 12-146.
23. Arutunova, N.D. (1988). Types of linguistic meanings: Assessment, event, fact. Moscow: Nauka. (In Russ.).
АрутюноваН.Д. Типы языковых значений. Оценка, событие, факт. М.: Наука, 1988.
24. Paducheva, E.V. (1985). Utterance and its coreference with reality. Moscow: Nauka. (In Russ.) Падучева Е.В. Высказывание и его соотнесенность с действительностью (Референциальные аспекты семантики местоимений). М.: Наука, 1985..
25. van Dijk, T.A. (1981). Studies in the pragmatics of discourse. The Hague—BerlIn: Mouton.
26. Tomlin, R.S. (1987). Linguistic reflections of cognitive events. In: Coherence and Grounding in Discourse, R.S. Tomlin (Ed.). Amsterdam: Benjamins. pp. 455-480.
27. Tenny, C. & Pustejovsky, J. (eds.) (1991). Events as Grammatical Objects: The Converging Perspectives of Lexical Semantics and Syntax. Stanford, Ca: CSLI Publications.
28. Novikov, L.A. (2001). Dynamic composition. In: Selected Papers. Vol. II. Esthetic aspects in language. Miscellanea. Moscow: RUDN publ. pp. 409-412. (In Russ.).
29. Novikov, L.A. (1991). Stylistic of ornamental prose of Andrej Belyi. Moscow: RUDN publ. (In Russ.).
30. Demyankov, V.Z. (1983). Event in semantics, pragmatics and interpretation coordinates. Izvestiya ÁN USSR. Literature and language series, 42 (4), 320-329. (In Russ.).
31. Labov, W. (1972). Language of the Inner City. Studies in the Black English Vernacular. Philadelphia: University of Pennsylvania Press.
32. Cienki, A. & Iriskhanova, O. (eds.) (2018). Aspectuality across languages. Event construal in speech and gesture. Amsterdam: John Benjamins.
33. Boldyrev, N.N. & Vinogradova, S.G. (2016). Composite sentence and its metacognitive modeling. Issues of cognitive linguistics, 3, 69-76. (In Russ.). https:// doi.org/10.20916/1812-3228-2016-3-69-76
34. Baker, P. & Egbert, J. (eds.) (2016). Triangulating methodological approaches in corpus-linguistic research. London: Routledge.
35. Kiose, M.I. & Efremov, A.A. (2020). HETEROSTAT software. Registered in FIPI 21.09.2020, registration number 2020661240.
36. Mikolov, T., Chen, K., Corrado, G. & Dean, J. (2013). Efficient estimation of word representations in vector space. In: Proceedings of ICLR Workshop. pp. 430-443. URL: https://www.arxiv-vanity.com/papers/1301.3781/ (accessed: 1.04.2021).
37. Paradis, C., Löhndorf, S., van de Weijer, J. & Willners, C. (2015). Semantic profiles of antonymic adjectives in discourse. Linguistics, 53 (1), 153-191.
38. Kiose, M.I. (2021). Linguistic creativity and discourse profiles of English language children's novels. Russian Journal of Linguistics, 25 (1), 147-164. https://doi. org/10.22363/2687-0088-2021-25-1-147-164
Information about the author:
Maria I. Kiose is Doctor habil. of Philology, Associate Professor, Leading Researcher of the Centre for Socio-Cognitive Studies of Moscow State Linguistic University. Research interests: cognitive semantics, indirect naming in text, referential semantics, oculographic and corpus-based research of text inference and generation process. e-mail: [email protected] ORCID ID: http://www.orcid.org/0000-0001-7215-0604; Scopus Author ID: 56642747500; Web of Science ResearcherIDAAB-7989-2019
Сведения об авторе:
Киосе Мария Ивановна, доктор филологических наук, доцент, ведущий научный сотрудник Центра социокогнитивных исследований дискурса Московского государственного лингвистического университета. Научные интересы: когнитивная семантика, непрямое наименование в тексте, референциальная семантика, окулографические и корпусные методы анализа восприятия и порождения текста. e-mail: [email protected] ORCID ID: http://www.orcid.org/0000-0001-7215-0604; Scopus Author ID: 56642747500; Web of Science ResearcherIDAAB-7989-2019
