CC BY
28
Научная статья УДК 81-119
DO110.52070/2542-2197_2022_1_856_119
Корпусный анализ поворотов головы в мультимодальном дискурсе
Н. В. Сухова
Национальный исследовательский технологический университет «МИСиС» suhova.nv@misis.ru
Аннотация. В статье представлены формальный, функциональный и прагматический подходы к исследова-
нию поворотов головы в дискурсе. Показано, что физическая форма движения определяет его прагматику, функциональный потенциал (коммуникативную роль в дискурсе) и способы его аннотирования. Цель данной статьи - описать формальный подход к кинетической структуре поворотов головы и предложить их полный список, описать роль поворотов головы в качестве смены позы и проанализировать их взаимодействие с мануальными движениями. Работа выполнена на материале русскоязычного мультимодального корпуса «Рассказы и разговоры о грушах».
Ключевые слова: движения головы, повороты головы, смена позы, мануальные движения, цефалический канал
Для цитирования: Сухова Н. В. Корпусный анализ поворотов головы в мультимодальном дискурсе // Вестник Московского государственного лингвистического университета. Гуманитарные науки. 2022. Вып. 1 (856). С. 119-127. DOI: 10.52070/2542-2197_2022_1_856_119
Original article
Corpus Analysis of Head Turns in Multimodal Discourse
Natalya V. Sukhova
National University of Science and Technology 'MISiS', Moscow, Russia suhova.nv@misis.ru
Abstract.
Keywords: For citation:
The paper discusses formal, functional and pragmatic approaches to study head turns in the discourse. The work shows that the physical form of a move determines its pragmatics, functional potential (communicative role in the discourse) and the ways it is annotated. The paper aims at the formal analysis of the turns kinetic structure and provides their full inventory, at the description of their role as a posture change and at the explanation of their concomitance with the manual movements. The study is done on the material from multimodal "Russian Pear Chats and Stories" corpus.
head movements, head turns, posture change, manual movements, cephalic channel
Sukhova, N. V. (2022). Corpu analysis of head turn in multimodal discourse. Vestnik of Moscow State Linguistic University. Humanities, 1(856), 119-127. 10.52070/2542-2197_2022_1_856_119
INTRODUCTION
The diversity of communication modes undergoes a very deep analysis in multitude of studies nowadays. Assuming a broader perspective, there are works devoted to speech production and gesture production which look at the joint or separate source of origin for these two modalities (see [Fritz et al., 2019; Goldin-Meadow, 2018]).
The studies on gestural components of multimodal communication vary a lot from some general explorations into it (cf. [Language and gesture, 2000; Body - Language - Communication, 2013; From gesture in conversation to visible action as utterance, 2014] to more specific ones which mostly entail investigations on kinetic activity of hands (see [Beattie, 2016; Litvinenko, Nikolaeva, Kibrik, 2017]).
Cephalic movements (or head movements) have not been widely addressed so far except for nods, tilts, shakes or wags (see [Pederson, 2014; McClave, 2000; Calbris, 2008]) from the formal point of view as well as the functional one (see [Hadar, Steiner, Clifford, 1985; Kousidis et al., 2013]). Despite the uni-directed focus of these studies to negation and affirmation, recent years have seen an increase in research considering such functions as back-channel, feedback, as markers of syntactic boundaries within the utterance; as markers of turn-takings; as rhythmic means of speech organization, etc. (see [Grishina, 2017]).
This paper discusses the broader categorization of cephalic movements in the discourse. Head movements are studied on the corpus resource called "Russian Pear Chats and Stories" (for more information see www.multidiscourse.ru). The material is an 18-minute video of one recording session (Pears22N1). Narrator is a Russian-speaking 24-year-old young woman. For more details on the settings and the material see Sections 3-4.
Within our corpus project we adopt a formal and functional approach to defining boundaries in the whole repertoire of kinetic behaviour (e.g. see annotation of manual gestures [Litvinenko, Nikolaeva, Kibrik, 2017]). The idea is to approach the kinetic units (movements) formally: i.e. how the annotator sees their motor / kinetic form in a mute video.
CEPHALIC MOVEMENTS PHYSIOLOGY
The head movements are quite complex as they are produced by a diverse net of muscles. The fact that
1 This is the way the material is registered in the project: Pears means this corpus, 22 stands for the number of the recording session and N stands for Narrator.
Fig. 1. The head position around three axes and in three planes: where A - mediolateral axis and a sagittal plane; B - vertical axis in the horizontal plane; C - anteroposterior axis in the frontal plane
those muscles are always active brings difficulty to the annotator. Having intuitively agreed that the head movement is a visible movement of the neck or head itself, we can face the situation when the head is actually moving all the time. To have a solid ground to annotate the head movements and thus extract them from the kinetic flow we have determined a definite set of characteristics. The head is able to move across three planes and along three axes (Fig. 1)2.
1) around mediolateral axis (imaginary line right ear-nose-left ear) in a sagittal plane. For example, TiltLeft3 (picture A);
2 Adapted from [Multimodal Annotation Guidelines, 2015].
3 Hereafter we use bold type and tag meaning to exemplify the discussion. About tags see below (Table 1).
2) around the vertical axis (imaginary line forehead-nose-chin) in the horizontal plane. For example, a TurnRepeated (picture B);
3) around anteroposterior axis (imaginary line nose-nape / back of the neck) in the frontal plane. For example, Backwards (picture C).
The basis of mark-up consists of the sequence of head movements and periods of stillness that appear between these movements.
SINGLE AND SIMULTANEOUS CEPHALIC MOVEMENTS. TAGGING
Head movements are given meaningful tags, which describe their physical characteristics (see Table 1). These characteristics cover a list of tags. Tags describe the movements in a sort of a 3D fashion exactly as we can observe movement's physical characteristics, e.g. head moves upwards (Up); head moves backward (Backward) or there is a tilt to the right (TiltRight, Fig. 2).
Table 1
TAGS ASCRIBED TO HEAD MOVEMENTS
Tag Description
Down Downward head move
Up Upward head move
Forward Move of the head forwards
Backward Move of the head backwards
TiltRight Move of the head leaning on the right side
TiltLeft Move of the head leaning on the left side
TurnRight Move of the head turning on the right side
TurnLeft Move of the head turning on the left side
Straighten Head is straightened (about Straighten see below in more details)
Slide Displacement of the head from side to side
Rock Head leans from side to side in a single continuous movement
Rotation Head makes a circular movement; this need not be a complete circle, it may be a part of an arc
TurnRepeated Head turns from side to side (cf.: shake)
Other It is difficult to describe the movement
Fig. 2. An example of a head movement TiltRight (Pears22N)
Fig. 3. The design and settings of the shooting session (2017) to collect the recorded material
(courtesy of O. V. Fedorova)
We have conducted an empirical study on the occurrence of single, simultaneous and compound head movements. Single cephalic movement is a movement which is independent from other movements of head and can be related to any explanatory tag, e.g. Down. Simultaneous cephalic movement is not formed by a 'combination', but rather an inseparable unity of usually two and, less commonly, three movements, which is found within the nomenclature of single movements, for example, downwards and tilting left (Down + TiltLeft1), i.e. this downward movement is simultaneously accompanied by tilting the head to the left. It is worth noting that the first element in this 'equation' indicates that this movement is marked. The markedness can be described in terms of such characteristics as velocity (slow, average, fast); amplitude (maximum, average, minimal) and intensity (prominent / strong, average, weak). Further, we mark the instances like Down+TurnLeft as Before Turn and cases such as TurnRight + Up as After Turn, which indicate the position of Turn movement in a simultaneous movement. We have also tested the occurrence of a compound cephalic movement (as suggested by some studies, see [Multimodal annotation guidelines, 2015]). Compound cephalic movements are those movements that go together in a consecutive mode, building a combination of usually two and, less
1 The sign plus (+) describes simultaneous movements in our annotation.
commonly, three gestures, for example upwards and backwards (UpBackward), i.e. backward movement follows upward movement. Our study has shown that it is generally possible to separate the first and the second movements and make them both single. The main criterion to distinguish between a single movement and a movement which is a part of a compound two-movement unity is their duration. A single movement should last for a minimum of 200 ms. If the movement is shorter than it becomes a part of a compound set.
In the majority of cases single and simultaneous types are sufficient enough to decipher and annotate what we see as a head movement.
CEPHALIC TURNS:
THEIR FORMAL DESCRIPTION
In the present study with the framework of our corpus we have decided to start examining the cephalic channel from the family of Turns. The Turns family in our cases is an umbrella term for representing all single and all simultaneous movements where there is a turn of head. Thus, they are: TurnLeft, TurnRight, TurnRepeated and all their simultaneous derivatives, like TurnLeft + Up (that means that while turning left a head also lifts up) or Down +TurnRight (that means that the head goes down and at the same time turns to the right). Hypothetically, we can have all combinations (see Table 1) of single movements and Turn family in progressive or regressive fashions,
i.e. Turn goes as the first (as a marked movement) or the second (as a concomitant second-marked movement) movement.
Firstly, we tackle the form of the head movements which we call 'Turn'. The form of the kinetic movement is described along the planes and axes the head moves (see Fig. 1 above). Here we have a starting1 position and the maximally possible position of the head in different planes. These two positions are the key to understand the boundaries and essential kinetic characteristics of a head movement.
The starting position can be illustrated in the following way: the head is straight and the straight line goes along the line head-neck-chest-pelvis-legs. In case when the head moves to this position we speak about Straighten. Regardless of the previous movement, posture and so on, the communicant will go back to this very position as it seems to be quite physiological. Now in our current annotation scheme we do not indicate the direction from where the head is straightened. That means that if the head was in the position TurnLeft and it goes back we tag the movement Straighten, the same is true about TurnRight, TiltRight / TiltLeft.
The finding that a starting point is not necessarily a neutral point of a head is complimented by the fact that body can be positioned in other way than directly towards the camera. The settings, where the material was recorded, suggest 3 main participants (see Fig. 3). Narrator looks to the left (turns left) as this is the place where Commentator is located. Commentator has seen the film, too, and should talk, commenting on what Narrator has said after the Narrator has finished his part. Both of them straighten their head / body positions as they both address the Reteller who is in front of them.
In other words, what we have witnessed in out settings is that Narrator and Commentator are sitting half-turned to each other. However, if we proceed and imagine the line head-neck-chest-pelvis-legs than it will be the starting point, where the head tends to return. That is why we distinguish between Turns and Straighten.
Besides the starting point, we identify the maximally physically possible point on the kinetic continuum where head can go in any particular direction. This point is called maximal. For example, as for TurnRight in the sagittal plane it would be the point of 90 degrees to the right.
1 The concept of a neutral position, i.e. the position where the head returns at some period of time, can be a synonym here, though sometimes head's neutral position is quite different from what we term as a starting position.
Thus, the freedom to move within sagittal plane is 180 degrees. If the head moves from the maximal point - head turned to 90 degrees to the right - to maximal point of 90 degrees to the left going through a starting point then this movement would be tagged as Rotation, even if the arc is not full. Moreover, we can presuppose that there can be a movement similar to this one but with a backward direction - it would also get a tag Rotation. Intuitively, it seems that this movement would be quite rare, and it definitely has never occurred in the analyzed material.
The same logic can be applied in other planes and across axes. These potential starting and maximal positions would enrich the picture of kinetic / motor abilities of head.
current research material
Here we present the analysis of cephalic Turns. This study aims to examine: 1) the formal structure of cephalic movement family Turns; 2) Turns as a posture change; 3) how Turns are concomitant with hand movements.
Among all the head movements, Turns family has been picked and analyzed.
Head movements were annotated according to the principles outlined above with 3D descriptive tagging. The annotation was done in ELAN programme [ELAN, 2021]. The annotation went on a mute video file. We assume that kinetic channel should be approached separately from the vocal channel on the first stage of a formal analysis. The video frame is 10 ms.
CEPHALIC TURNS: FORM
The formal approach towards Turns has shown that the most numerous group of Turns is Straighten (37 % of all Turns). Then TurnLeft follows with 33 % of all Turns. It can be explained by the settings, as we have already mentioned, when the Narrator has to turn left in order to see the Commentator. The third largest group is comprised by TurnRepeated movements (see Table 2).
All head movements in the episode under discussion is 470 cases, among those 138 are Turns, that is 29 % of all cases. Among Turns there are 83 % of single movements and 17 % are simultaneous ones (see Table 3). It should be underlined here that the movement TurnRepeated in this study has been considered as a single movement. There might be a plausible explanation to this. TurnRepeated movement is a set of movements TurnRight and TurnLeft which are definitely different from real single movements with the same names due to it
intrinsic meaning - to shake one's head. However, we cannot but agree that they are turns in their form. Thus, we include them formally in a Turn family and separate them from the analysis of pure Turns.
CEPHALIC TURNS: FUNCTION
At the first, basic level the following general kinetic units are annotated for the head: these are movements which are described with the help of tags according to formal physical principles (see Section 2). Then these movements are classified according to their basic functions: gestures, adaptors, or cephalic posture changes.
1. Gestures are considerably meaningful movements which are performed with a
particular communicative goal. They are connected with speech production and speech perception (for example, nodding).
2. Adaptors are meaningful movements which are not connected with speech and have a certain practical goal (for example, moving one's head to fix one's hair).
3. Posture changes are the movements the main goal of which is to move head from one neutral position to another that would differ considerably in one of the axes with certain fixation in this position for some perceivable time.
Turns comprise 29 % of all cephalic movements and out of them there are 12 % those which cause the posture change (see Table 4).
Table 2
DISTRIBUTION OF TURN MOVEMENTS IN PEARS22N
Turns (tags) Total number of Single Turns Simultaneous Turns (cases) Total number
Turns (cases) (cases) Before Turns After Turns of Turns (%)
TurnLeft 46 37 4 5 33
TurnRight 15 11 3 1 11
TurnRepeated 26 25 0 1 19
Straighten 51 41 6 4 37
Total 138 114 13 11 100
Table 3
THE COMPOSITION OF TURN MOVEMENTS IN PEARS22N
Turns Number of cases %
Total number of Turns 138 100
Total number of single Turns (TurnRepeated are included) 114 83
Total number of simultaneous Turns 24 17
Table 4
DISTRIBUTION OF TURNS AS POSTURE CHANGE IN PEARS22N
Cephalic movements Number of cephalic movements (cases) Number of cephalic movements (%)
Cephalic movements (all) 470 100
Turns and their derivatives (TurnRepeated are included) 138 29
Turns and their derivatives (posture changes) 55 12
Table 5
THE INTERRELATION BETWEEN TURN TYPES AND MANUAL MOVEMENTS
Turn types Number of cases Number of Turns concomitant with manual movements (cases) Number of Turns concomitant with manual movements (%)
Total 138 81 59
Single Turns 114 65 57
Simultaneous Turns 23 16 67
cephalic turns: concomitance with manual movements
We have also Looked at the interchannel relations between cephalic and manual movements. The manual movements of both hands have been interrelated with Turns in a broad sense. The cephalic activity generally prevails over manual in our material video. There were 81 manual movements which correspond to 138 cephalic Turn movements (see Table 5). In other words, 59 % of Turns are concomitant with manual movements.
GENERAL DIScuSSIoN AND conclusion
This study investigated the kinetic form of cephalic Turns and how we can approach the boundaries of the movements we see in the communication both as a viewer and as a communicant. The method of 3D tagging has proved to be quite sufficient and reliable. The tag scheme suggests that the head movement can be described along three spatial axes and within three spatial planes. We can also trace the changes in the movement characteristics (velocity, amplitude and intensity).
Thus, cephalic Turns can be described as TurnLeft, TurnRight, TurnRepeated and Straighten. Those movements may be of two types: single and simultaneous. Single turn movements are one-vector movements, like TurnRight, TurnLeft or Starighten. TurnRepeated is a complex movement (one or more movements to the right and to the left) which is also viewed as a single movement due to its inherent meaning - to shake one's head. Simultaneous movements consist of one of Turn family movement and some other cephalic movement which can either precede Turn or follow it in space and time, which is reflected in the tagging. The order of the movements in the tag would imply its kinetic markedness: the most prominent movement would go first.
We have also examined the cephalic posture changes and the role of Turns in them. The notion of posture within a kinetic channel is justified by the fact that there are neutral positions of the body part or the whole body and there are some local and marked positions of them. Intuitively, Turns are those movements which change the cephalic posture. However, the survey has showed that only 12 % of Turns change the posture.
One more focus of our study was to see the interchannel correlation between cephalic Turns and manual movements. The formal and functional approach we advocate in our project made the comparison of two different communication channels possible and feasible. We have chosen one level of parameters - movements - to be compared. The whole idea was to see any correspondence between hands and head movements in contrast to periods of stillness in both channels. The well-elaborated manual scheme and ELAN [ELAN, 2021] annotations allowed a refined comparison. Hence, we looked at movements of hands which could be gestures, adaptors or manual posture changes and we correlated them to cephalic movements which also could be gestures, adaptors or cephalic posture changes. Only Turns were examined here. The results have shown that the cephalic activity prevails over manual. Turns are concomitant with manual activity in 59 % of cases of all cephalic activity. Further research is needed as to the explanation of this relation. However, by now it can be hypothesized that Turns are quite prominent movements; they can change posture in 12 % of cases of all cephalic movements and while changing the posture it can be so that the communicant is either take-turning in speech or keeping silent just looking at his interlocutor. Hence, his manual activity slows down.
Thus, the further research can be suggested as into the other types of cephalic activity and more precise categorization of other cephalic movements and Turns.
СПИСОК ИСТОЧНИКОВ
1. Fritz I. et al. Information packaging in speech shapes information packaging in gesture: the role of speech planning units in the coordination of speech-gesture production / I. Fritz, S. Kita, J. Littlemore, A. Krott // Journal of Memory and Language. 2019. Vol. 104. P. 56-69.
2. Goldin-Meadow S. Taking a hands-on approach to learning // Policy Insights from the Behavioral and Brain Sciences. 2018. Р. 1-8.
3. Language and gesture (Language Culture and Cognition) / ed. by D. McNeill. Cambridge, England: Cambridge University Press, 2000.
4. Müller C., Cienki A., Fricke E., Ladewig S., McNeill D., Teßendorf S. (Eds.) Body - Language - Communication. An international handbook on multimodality in human interaction. Berlin: de Gruyter Mouton, 2013.
5. From gesture in conversation to visible action as utterance: Essays in Honor of Adam Kendon / ed. by M. Seyfeddinipur, M. Gullberg. Philadelphia, PA: John Benjamins, 2014.
6. Beattie G. Rethinking body language. How hand movements reveal hidden thoughts. London, New York: Routledge, 2016.
7. Литвиненко А. О., Николаева Ю. В., Кибрик А. А. Аннотирование русских мануальных жестов: теоретические и практические вопросы // Компьютерная лингвистика и интеллектуальные технологии: по материалам Ежегодной международной конференции «Диалог» (Москва, 31 мая - 3 июня 2017). М.: Российский государственный гуманитарный университет, 2017. Вып. 16(23), Т. 2. С. 255-268.
8. Pederson E. Listener head gestures and the co-construction of narrative timing. International Society for Gesture Studies VI. July 8-11. San Diego, California. ISGS6 Abstract, 2014. P. 67-68.
9. McClave E. Linguistic functions of head movements in the context of speech. Journal of Pragmatics. 2014. Vol. 32, № 7. P. 855-878.
10. Calbris G. La tête de Nicolas Sarkozy, ou les fonctions des gestes de la tête durant l'enonciation. Mots. Le langages de politique. 2008. Vol. 86. P. 98-118.
11. Hadar U., Steiner T. J., Clifford F. Head movement during listening turns in conversation // Journal of Nonverbal Behavior. 1985. Vol. 9. P. 214-228.
12. Kousidis S. et al. (2013). Exploring annotations of head gesture forms in spontaneous human interaction / S. Kousidis, Z. Malisz, P. Wagner, D. Schlangen. In: TiGER 2013, Tilburg Gesture Research Meeting.
13. Гришина Е. А. Русская жестикуляция с лингвистической точки зрения (корпусные исследования). М.: Языки славянской культуры, 2017.
14. Multimodal annotation guidelines. CorpAGEst (2013-2015): A corpus-based multimodal approach to the pragmatic competence of the elderly / C. Bolly, D. Belin, D. Boutet, J. Kairet, A. Lepeut, A. Safar, A. Thomas. Manual. 2015. Vol. 1.8. P. 23-25.
15. ELAN. Version 6.2. [Computer software]. Nijmegen: Max Planck Institute for Psycholinguistics. 2021. Retrieved from https://archive.mpi.nl/tla/elan
references
1. Fritz, I., Kita, S., Littlemore, J., Krott, A. (2019). Information Packaging in Speech Shapes Information Packaging in Gesture: The Role of Speech Planning Units in the Coordination of Speech-Gesture Production. Journal of Memory and Language, 104, 56-69.
2. Goldin-Meadow, S. (2018). Taking a hands-on approach to learning. Policy Insights from the Behavioral and Brain Sciences (pp. 1-8).
3. McNeill, D. (Ed.) (2000). Language and Gesture (Language Culture and Cognition). Cambridge, England: Cambridge University Press.
4. Müller, C., Cienki, A., Fricke, E., Ladewig, S., McNeill, D., Teßendorf, S. (Eds.) (2013). Body - Language -Communication. An International Handbook on Multimodality in Human Interaction. Berlin: de Gruyter Mouton.
5. Seyfeddinipur, M., Gullberg, M. (Eds.), (2014). From gesture in conversation to visible action as utterance: Essays in Honor of Adam Kendon. Philadelphia, PA: John Benjamins.
6. Beattie, G. (2016). Rethinking body language. How hand movements reveal hidden thoughts. London, New York: Routledge.
7. Litvinenko, A. O., Nikolaeva, Yu. V., Kibrik, A. A. (2017). Annotirovanie russkikh manual'nykh zhestov: teoreticheskie i prakticheskie voprosy = Annotation of Russian manual gestures: theoretical and practical issues. Komputernaya
linguistika i intellectual'nye tehnologii [Computer linguistics and intellectual technologies]. Proceedings of "Dialogue" (Moscow, 31 May - 3 June 2017), 16(23), 2, 255-268. M.: Russian State University for the Humanities. (In Russ.)
8. Pederson, E. (2014). Listener head gestures and the co-construction of narrative timing. International Society for Gesture Studies VI (pp. 67-68). July 8-11. San Diego, California. ISGS6 Abstract.
9. McClave, E. (2000). Linguistic functions of head movements in the context of speech. Journal of Pragmatics, 32(7), 855-878.
10. Calbris, G. (2008). La tête de Nicolas Sarkozy, ou les fonctions des gestes de la tête durant l'enonciation. Mots. Le langages de politique, 86, 98-118.
11. Hadar, U., Steiner, T. J., Clifford, F. (1985). Head movement during listening turns in conversation. Journal of Nonverbal Behavior, 9, 214-228.
12. Kousidis, S., Malisz, Z., Wagner, P., Schlangen, D. (2013). Exploring annotations of head gesture forms in spontaneous human interaction. In: TiGER 2013, Tilburg Gesture Research Meeting.
13. Grishina, E. A. (2017). Russkaya gestikulyatsia s lingvisticheskoy tochki zrenia (korpusnye issledovaniya) = Russian gesticulation from a linguistic perspective (corpus studies). Moscow: Yazyki Slavyanskoy Kultury. (In Russ.)
14. Bolly, C., Belin, D., Boutet, D., Kairet, J., Lepeut, A., Safar, A., Thomas, A. (2015). Multimodal Annotation Guidelines. CorpAGEst (2013-2015): A corpus-based multimodal approach to the pragmatic competence of the elderly. Manual, 1.8, 23-25.
15. ELAN (Version 6.2) [Computer software]. (2021). Nijmegen: Max Planck Institute for Psycholinguistics. Retrieved from https://archive.mpi.nl/tla/elan
информация об авторах
Сухова Наталья Витальевна
кандидат филологических наук, доцент, заместитель заведующего кафедрой иностранных языков и коммуникативных технологий Института базового образования Национального исследовательского технологического университета «МИСиС»
INFORMATION ABOUT THE AUTHORS Sukhova Natalya Vitalievna
PhD (Philology), Associate Professor, Deputy Head of the Department of Modern Languages and Communication, College of Basic Studies, National University of Science and Technology 'MISiS'
Статья поступила в редакцию 29.11.2021 The article was submitted 29.11.2021
одобрена после рецензирования 20.12.2021 approved after reviewing 20.12.2021
принята к публикации 25.12.2021 accepted for publication 25.12.2021
