CC BY
43DOI: https://doi.oig/10.46991/AFA/2021.17.2.063
MULTIMODALITY IN CONTEMPORARY COMMUNICATION
Ani Simonyan*
Yerevan State University
Communication and representation have always been about more than language. Since the times of Ancient Greece and Rome, people have been sure that language is not the only tool of communication. This means that there are more factors involved in and contributing to successful and productive interaction. The role of language is invaluable, however, the spatial, aural and visual aspects of communication should not be overlooked or underestimated. Multimodality is about the choices of modes and the social effect produced by the mentioned aspects of communication. Modern technologies directly influence multimodality, offering new and newly emerging modes or meaning-making resources. Thus, this paper focuses on illustrating contemporary communication realized through more than one mode or, in other words, multimodal ensemble. The article reveals that each mode can add something special to the meaning and its perception that others cannot. It presents a thorough study of mode, multimodality and its key concepts as well as the interrelation between multimodality and new digital technology. The article also highlights the factors contributing to the enhancement of meaning and provides examples that are meticulously discussed through Case Study and Multimodal Analysis methodology to discover all the modes and media resources applied to the creation of meaning.
Keywords: multimodality, monomodality, mode, multimodal ensemble, meaning-making, meaning potentials, digital media.
Introduction
The term multimodality has been chosen for various means of meaning-making. In fact, it is an interdisciplinary approach deriving from social semiotics that views communication as more than language and an integrated whole of meaning-making ways. Multimodality offers concepts, methods and framework for the study of aural and visual aspects of communication and spatial aspects
* anisimonyan@ysu.am
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Received: 16.06.2021 Revised: 19.08.2021 Accepted: 26.09.2021
© The Author(s) 2021
of the environment. It aims at challenging the widespread opinion that language is the only major form of communication and focuses on studying the interaction and connection between different modes. Thus, it can be stated that language in its written and spoken forms is still considered to be of utmost importance, however, as part of the multimodal ensemble.
It has been proved that most of our communication process relies on nonverbal means (Bezemer & Mavers, 2011). Thus we can conclude that static as well as moving images, emoticons, the tone and pitch of voice and gestures can contribute to the perception of a particular message if chosen carefully to fit the context. Today there is an increasing use of technologies to communicate something and to influence people. Hence multimodality in digital technologies needs to be thoroughly studied. This kind of study should concentrate on various modes, their meaning potentials and constraints posed by this or that type of technology and the multimodal ensemble which can be applied via technology in public communication.
To achieve our goal and understand the factors included and their influence on the audience - two examples have been discussed. The first example concentrates on a piece from the Encyclopedia Britannica (2021), while the second one studies a YouTube program called Dr. Oz (2020). Two methods of analysis have been used in the article. The first is the method of Multimodal Analysis and the second - Case Study.
Theoretical assumptions on multimodality
Scholars studying multimodality adhere to such disciplines as linguistics, media studies, semiotics, psychology, education, sociology. All of them have a common interest or object of study as they want to understand what the different ways or modes of meaning-making are (Bezemer, 2008). Multimodality can be defined as various means and forms of meaning-making with language in its oral and written forms as part of the multimodal ensemble. Social semiotics is considered to be the main source of multimodality.
Multimodality came to attention due to Hodge and Kress (1998) and later Kress and van Leeuwen (2006). They brought different modes other than language to attention preparing grounds for multimodality. Kress and van Leeuwen studied visual texts to reveal semantic resources, meaning potentials and their organization to visually convey discourses and ideologies. Some core ideas are taken from linguistics: turn taking, composition and coherence. With the growing interest in multimodality scientists have also searched for other
resources to make reliable analyses. Such fields as sociolinguistics, art history, musicology, iconography, film history are also studied.
Multimodality aims at detecting the choices of modes that people make and the social effects they leave on meaning. The notion of context is of great importance here as all the meaning-making modes are chosen to suit the context in terms of design and selection. There is a strong connection between the social context and the signs, modes and meaning-making resources selected to convey meaning.
Kress (1993) comes up with the idea of the motivated sign, emphasizing the social character of meaning. The idea is simple: we make choices according to the social context. Speaking about multimodality Jewitt (2008) states that the latter is a response to the modern demand to consider more than language in communication in the era of constantly developing digital technology. Understanding the variety of modes in communication presented by digital technology like signs and writing, is essential to distinguish new types of texts and interactions in relation to the spatial, content and genre conventions.
According to Bezemer and Mavers (2011) multimodality relies on three main theoretical assumptions:
Firstly, communication and representation are realized through combinations of various modes (gestures, language, space, voice, etc.). All the factors or meaning-making resources contribute to better communication.
Secondly, meaning-making resources are socially shaped. This means that they should be used in a particular society to express a certain meaning. The frequency of use in the social life of this or that culture makes the mode more articulate. The selection of modes is of vital importance to communication and creation of meaning.
Finally, people express meaning through mode selection and configuration. The idea that modes are interrelated should be highlighted here.
In their work called Reading Images Kress and van Leeuwen (2006) claimed that there were Mctimes when Western culture was dominated by monomodality. They argued that there was one main mode in all texts and writing genres. That mode was language. To prove their assumptions - among the examples they drew attention to newspapers, books, official documents which totally lacked pictures and colors. The same was true about art: pictures all used canvas and oils, musicians all dressed alike and it was only the soloist and the conductor that could use body language. Even the disciplines studying different art forms were monomodal in essence using language as the only
mode to speak about language, music, art. The spheres of musicology, linguistics and art history were among the many fields confirming their opinions.
It was not long ago that there was a shift from monomodality to multimodality not only in mass media but also books, university, government and business documents. Now we can see pictures, colors and page design (layout and typography) which make the information more pleasant, comprehensible and impressive. Artists representing different branches of art have started to search for modes that would allow them to cross boundaries.
When speaking about multimodality we should not forget about semiotics. It is a well-established fact that the same meaning can be expressed via different semiotic modes. In other words, language should be looked at from the social functional perspective as a system of possible choices and meaning potentials (Halliday, 1978; (Jewitt, 2009).
The key concepts to multimodality
Jewitt (2008) separates six key concepts to multimodality. They are S mode,
S semiotic resource, S materiality, S modal affordance, S multimodal ensembles,
S meaning functions/functional theory of meaning. A channel or a resource can be called a mode if it is widely accepted and commonly used by the community. Halliday (1978) argues that to be considered a mode, resources should meet the following criteria: they should be meaningful, help to build social relationships and be coherent in a text. Well-established modes include gestures, gaze, image, writing, posture, etc. Mode is a set of meaning-making resources shaped by culture and society. According to Jewitt (2009, p. 254), a mode consists of elements and norms which express "well-acknowledged regularities" in a certain community.
Actions, materials and artifacts used to communicate are the semiotic resources. The connection between mode and form is highly stressed in semiotic resources. The resources such as gestures, facial expressions, posture and smiles, messages, etc. can be physiological and technological. They have the ability to convey meaning owing to their past uses and a number of affordances due to the possible uses.
Materiality refers to the way in which modes are used to express meaning, such as the written text, sound, color, etc. They are different from each other in terms of meaning-making potentials and constraints.
Kress (2009) defines modal affordance as the potentialities and constraints of modes to express meaning in a certain situation. It is about the ability to represent, express and communicate meaning due to the well-established social, cultural and historical use and perception of a certain mode. From this perspective, according to Jewitt (2009, p. 254), each mode is partial, and to communicate effectively one needs to apply several modes at the same time.
The multimodal ensemble refers to the simultaneous use of several modes in interaction. All of them are used to represent certain meaning more clearly. Each of the modes involved carries just part of what is intended to be said, and multimodality studies the impact of each mode as well as the interplay between the modes in the ensemble.
Multimodality relies on the meaning functions/functional theory of meaning, according to which, meaning is a social action expressed through modal choices. The choices are made in accordance with the content of what people want to convey, social relations between the interlocutors and the textual organization.
At the beginning of the 20th century linguistics was considered a branch of a wider science called semiology (Saussure, 2013). Since then efforts are being made to reveal the different factors contributing to the creation of meaning. With multimodal approach, however, it has become evident that not a single element or factor contributes to the meaning-making processes but rather a set of means that usually appear together: "image with writing, speech with gesture, maths symbolism with writing", etc. (Bezemer & Jewitt, 2010, p. 181). Multimodality suggests the idea that means of meaning-making do not appear in isolation. Moreover, they tend to appear together most of the time creating an inextricable whole. With the introduction of digital technologies it has become more common to use several modes at once to create meaning, and it has become more difficult to separate one from the other. That is why multimodality and multimodal ensemble should be considered essential when analyzing meaning.
Traditionally, language was considered the most important and powerful means of meaning-making and also the one which can serve various communicative functions, it is a tool, available to great masses of people. There
was a clear division between verbal and non-verbal factors included in the process of meaning-making. However, new technology came to illustrate the fact that none of the tools can be considered more important than the other. They are just different ways of expressing the same thing. The use of several modes at once makes it easier for the audience to understand the message. Thus, it is not by chance that scholars have noticed the interplay between the two means (verbal and non-verbal) and tried to understand what potentials and limitations each of them has. To sum up, there are verbal and non-verbal semiotic resources and if one wants to study meaning, it is important to go through all the levels of meaning-making.
Multimodality also focuses on situated actions. This means that it discusses both the social contexts and means of meaning-making. This approach suggests that the selected resources or means of creating meaning in a particular situation are more important than all the available ones. Consequently, multimodality enables us to study, analyze and theorize the different ways contributing to the meaning-making process. It aims at finding connections between available semiotic resources and their meaning in a particular social context (Jewitt, 2009, p. 250).
Different scholars studying multimodality choose different terms to refer to this phenomenon. Kress (2011) prefers multimodality, other scholars choose multimodal discourse, multimodal communication or multimodal interaction (Kress & van Leeuwen, 2006; Turk, 2013). There are also different terms for mode itself. It can be referred to as a semiotic resource or simply resource. In general there are no clear-cut boundaries between these resources and they are considered to be an inseparable whole.
Digital technologies in multimodal ensemble
While other modes of communication, such as gesture and gaze, have been recognized and studied extensively (McNeil, 1992), multimodality, as already mentioned, investigates the interaction between communicational means and challenges the prior predominance of spoken and written language in research (Scollon & Scollon, 2009). Speech and writing continue to be understood as significant but are seen as parts of a multimodal ensemble.
The study of language and other resources has always been a matter of great interest. Even a long time ago people wanted to understand the relations existing between speech and gestures or gaze. Multimodality questions the widespread opinion that language is the most powerful means of meaning-
making. If that were true, then how could people having hearing disorders or those who could not speak, or even babies converse and communicate? Considering these examples we can conclude that communication has always been multimodal - not only language but also gestures, facial expressions and other modes have their important role to play in achieving a meaningful communication process. The opinion that language is the only powerful means or mode of communication is the result of the fact that linguistics has a long history and there are various means of describing language. On the other hand there has been little research to find out about the properties of scent, color, gesture, gaze, etc. Most of our communication process relies on non-verbal means (Bezemer & Jewitt, 2010). This means that images/moving images, emoticons, the tone and pitch of voice and gestures can contribute to the perception of a particular message if chosen carefully.
Nowadays, scientists are more interested to find out how people make meaning in a particular context to achieve their goals in communication. And again, though multimodality does not deny the fact that language has an important role in meaning-making, it states that language is one of the resources in the multimodal ensemble, so it is more important to understand the situational use of the resources and the purpose for which they are chosen. People select a set of modes for expressing their ideas. Therefore, the interaction between the chosen modes is essential.
Though communication has always been multimodal, the introduction of digital technologies has created new challenges for social science research and analysis. Scholars have to research new environments and tools used for data collection, representation and storage. The study of digital texts, multimodal inventories, environments is of great importance as it can reveal the potentials and constraints of modern technologies in terms of the representation of information and communication. Digital technology and the internet have resulted in the emergence of new modes. Digital media has contributed to the increase in people's interest in various modes, their configurations and interactions. One mode contributes to the better understanding of another mode for they are always in an interplay (Jewitt, 2009, p. 252). Thus, new digital technologies have their direct and serious influence on multimodality as they offer new modes of communication. In other words today, more than ever, there is an increasing use of technology to communicate something and to influence people. Hence multimodality in digital technologies needs to be thoroughly studied, concentrating on various modes, their meanings, potential
and constraints posed by this or that type of technology and the multimodal ensemble which can be applied via technology in different registers or discourse types of communication, including public communication.
Lemke (2002) and Zammit et al. (2007) have examined the ways of organizing texts, such as creating hyperlinks or layering and the impact on people's navigation of digital texts. Kress and van Leeuwen (2006) attempted to examine the dynamics of the connection of image and language. In fact, they had studied visual articulation of meaning earlier in their works, and now many other scholars tried to observe the relations between image and text, image and writing, color and layout of texts (Jewitt, 2009, p.259).1
Let us discuss the example of Encyclopedia Britannica (2021)2 As soon as one enters the site, various colorful pictures, headlines and an eye-catching layout provoke interest. Everything here is planned carefully to attract the reader or to trigger curiosity towards the information available on the page. All the details including the font, the color of subtitles, the pictures and videos contribute to the better understanding of information. If there was only the text or the image, the data wouldn't be impressive and appealing. Even the position and the territorial relationship between the text and images is of utmost importance.
ENCYCLOPEDIA BRITANNICA
IN THE NEWS ■ SPOTLIGHT • DEMYSTIFIED • QUIZZES - GALLERIES • LISTS
I Search Britannica..,
V^VSS
Obama Acts to Protect the Arctic
On December 20,2016, U S Pres. Barack Obama issued memorandums banning future oil exploration in the Arctic and along much of the Atlantic Coast.
SPOTLICHT / SOCIETY
The International Criminal Court (ICC)
Picture 1
Multimodal ensemble: Encyclopedia Britannica
Public communication is about understanding, planning, realizing and assessing successful communication campaigns in relation to public service. It
aims at informing, convincing, relationship building, encouraging open dialogue which serves the public interest. This goal is generally achieved with the help of modes of persuasion and rhetorical strategies or appeals.
The idea of ethos, logos and pathos as three essential components of public communication defined by Aristotle (1954) in 350 BC, is often used in speech, writing and advertisements. Ethos is an appeal to credibility. It refers to the necessity of establishing credentials as a presenter of a certain subject matter. To be convincing and to attract more people the presenter should show the authority, right or qualification to introduce the subject. The audience should believe in what is said and be sure it is trustworthy. The presenter can succeed showing that s/he is a significant figure or someone competent and knowledgeable in the field by referring to a relevant stock of terminology. Being introduced by another authority or described as genuine will be another asset. Pathos is an appeal to human emotions. It comes in various forms, such as simile, metaphor, a hint at injustice3. Pathos is especially effective when there is an agreement in terms of the underlying values between the speaker and the listener, the writer and the reader, etc. Pathos can also appeal to the listener's hopes and imagination. Logos is the simulation of logic. Logos supports the speaker's ideas via facts and figures. Using data and information, it contributes to the creation of the image of a knowledgeable speaker, which in its turn can enhance ethos. However, not every piece of information is a complete truth. Sometimes the speaker can deliberately use manipulation, false, misleading or inaccurate information to enhance pathos (Roberts, 1984). More than ever, today multimodality has greatly enhanced persuasion and rhetorical strategies suggested by Aristotle. With the application of multimodal approach it is possible to create a repository of meaning potentials which might be helpful when using technology in all types of communication, especially in public contexts. This goal can be achieved via the systematic description of semiotic resources and modes, modal affordance, materiality and organizing principles of applications and devices introducing the idea of creating networks, or mapping the meaning potentials of modes.
Multimodality has discovered and developed new semiotic resources. The process is connected with changes in society and the social need for novelties or new uses of existing resources. Among the examples can be mentioned the use of digital technology to reshape human voice in terms of tone and rhythm. As far as texts and ways of linking and layering printed texts are concerned, they have also been significantly revised by digital technology.
Nowadays there is a wide range of TV programs which successfully combine different modes and new digital technology. Their target is general public and the aim is to inform people and influence human behavior. In this respect, let us have a look at a TV show called Dr. Oz launched by Oprah Winfrey's Harpo production and Sony Pictures Television (Dr. Oz, 2020). The episodes are also available on Youtube. The aim of this TV show is to popularize knowledge on health. To achieve the goal a number of multimodal elements are used. To make information more digestible real life experiments are made on people and special manikins. In each episode the studio is designed with placards, manikins, human skeletons, etc., chosen in accordance with the topic of interest to make people understand the essence of the problem, the reasons and solutions. The program has an innovative format. It is sometimes interrupted by a video of an expert to share his experience or some real-life stories. The host of this Daytime Emmy Award-winning show is Dr. Oz himself who is Vice-Chair and professor of surgery at Columbia University. He also directs the Cardiovascular Institute and Contemporary and Complementary Medicine Program at NY Presbyterian. He is usually dressed in a smart suit and a shirt or a medical uniform. The guests are usually doctors or experts in various fields. In many episodes the audience also takes part in some real life experiments and assists the host to reveal the whole truth about the issue. The program is quite popular among different age groups. Essential information is usually presented on a large screen in the studio. We can see that the colors, uniforms, visual materials, videos all contribute to the better comprehension of the situation under discussion. The title of the show Dr. Oz itself is very impressive as the host himself is a prominent doctor and a university professorBelow is presented an episode from the mentioned show with Dr. Oz whose character fills the viewer with the confidence that whatever is said and illustrated in the program is reliable and trustworthy.
Linguistic mode being part of the multimodal ensemble is still dominant as the whole show has a format of conversation, so turn-taking is one of the essential aspects of this kind of communication. Also, the information is usually coherent to make it easier and more comprehensive for the listener. Information is mainly presented using necessary terminology in a logical sequence. To enhance the perception of academic language for the listeners, the hosts usually interrupt the expert to ask questions of general interest.
Picture 2 Multimodal ensemble: Dr. Oz
Conclusion
Thus, we can conclude that communication (more than ever today) is usually realized through a combination or amalgamation of modes each of which contributes to the better understanding of the message. With the invention of modern digital technology new modes have appeared making the information exchange process more feasible and interesting. Multimodality aims at detecting the choices of modes that people make and the social effects that they leave on meaning. Mutimodality thoroughly studies what can and cannot be done using technology, hence providing useful, obvious and explicit information in terms of new modes. It points out ways of redesigning and using semiotic innovations. On the whole, the thorough study and awareness of various modes and the effects created by them can help people make better choices for conveying messages.
Notes
1. Jewitt (2008) tried to explore the basic connections between a variety of modal resources, such as color, gesture, gaze, movement, sound. The author showed how digital technologies have changed the ways in which students and
teachers interact and exchange information. His study has revealed that new technology has reshaped the variety of modes, semiotic resources, materiality and modal affordance.
2. Encyclopedia Britannica (2021) has an ample supply of multimodal articles at https://www.britannica.com/, see for example, https://www.britan-nica.com/biography/Alexander-the-Great.
3. According to Merriam-Webster online dictionary the words pathetic, empathy and sympathy originated from the same Greek word - pathos meaning "emotion", "experience" or "suffering". The word was borrowed into English in the 16th century. Retrieved from; https://www.merriam-webster.com/-dictionary/pathos; https://www.merriam-webster.com/dictionary/apathy
References
Aristotle. (1954) Rhetoric. New York: Modern Library.
Bezemer, J. (2008). Writing in multimodal texts: A social semiotic account of designs for learning. Sage Journals, 25(2), 166-195. https://doi.org/10.1177/0741088307313177 Bezemer, J., & Jeweitt, C. (2010). Multimodal analysis: Key issues. In L. Litosseliti (Eds.). Research methods in Linguistics (pp. 180-197). London: Continuum. Bezemer, J., & Mavers, D. (2011). Multimodal transcription as academic practice. International Journal of Social Research Methodology, 14(3), 191-206. https://doi.org/10.1080/13645579.2011.563616 Halliday, M.A.K. (1978). Language as social semiotic: The social
interpretation of language and meaning. London: Edward Arnold. Hodge, R., & Kress, G. (1998). Social semiotics. Cambridge: Polity Press. Jewitt, C. (2008). Multimodal methods for researching digital technologies. London: Routledge.
Jewitt, C. (2009). The Routledge handbook of multimodal analysis, (17). Retrieved from https://www.researchgate.net/publication/2932751-74_Multimodal_ methods_for_researching_digital_technologies Kress, G. (1993). Against arbitrariness: The social production of the sign as a foundational issue in critical discourse analysis. Discourse and Society, vol. 4, issue 2, 169-193. https://doi.org/10.1177%2F0957926-593004002003
Kress, G. (2009). Multimodality: A social semiotic approach to contemporary communication, London: Routledge.
Kress, G. (2011). Multimodal discourse analysis. London: Routledge.
Kress, G., & van Leeuwen T. (2006). Reading images: The grammar of visual design. London: Routledge.
Lemke, J. (2002). Travels in hypermodality. Visual communication. London: Sage Publications.
McNeil, D. (1992). Gesture and thought. Chicago: University of Chicago Press.
Roberts, W. (1984). The 'Rhetoric' and the 'Poetics' of Aristotle. New York: The Modern Library.
Saussure, F. (2013). Course in general linguistics. Trans.by R. Harris. New York: Bloomsbury Publishing Plc.
Scollon, R., & Scollon, S. (2009). Intercultural communication: A Discourse Approach, London: Wiley Blackwell.
Turk, M. (2013). Multimodal interaction: A review. United States: Elsevier.
Zammit, K., McCabe, A., O'Donnell, M., & Whittaker, R. (2007). Popular culture in the classroom: Interpreting and creating multimodal texts. Advances in Language and Education. Retrieved from http://ezproxy.uws.edu.au/login?url=http://site.ebrary.com/lib/sydney/Do c?id=10362028&ppg=69
Sources of Data
Encyclopedia Britannica. (2021), Retrieved from https://www.britannica.com/
Dr. Oz. (2020). In Youtube. Retrieved August 10, 2021 from https://www.youtube.com/user/DoctorOz
¿UT.nnu^ônh0-3U-u UbS
UUfr Ufrtfnüjrnü
Zmqnpqm^gnLpjnLUp m^t^U t, pmU tf^mjU itq^fr tf^2ngn4 stqt^nLpjmU ^n^mUgnLtfp, pmU^ np tfmUpm^p^m pUspnLpjnLU t ^mmmp^ntü mtqt^mm^nLpjmU hmqnpqtfmU Àktp^ nL tf^2ngUtp tf^h: UjU pnLnP tqmUm^Utpp, npnUf oqmmqnp&^nLtf tU mtqt^mm^nLpjmU hmqnpqtfmU hmtfmp, ^mqtfnttf tU pmqtftqmUm^mj^U mtfpn^nLpjnLU: Stqt^mm^m-^mU Unp p^mj^U mt^Unjnq^mUtp^ qmpqmgtfmUp hmtfpUpmg" mnm^m-UnLtf tU Unp àhtp, npnUf mtqt^nLpjnLUp hmumpm^ntpjmU m^t^ nL
HjU tU hmugUnLl: ¿minpqm[gnLp]mU pmqltimUm-
[m]UnLp]mUp ^tpmptpnq q^mt^pUtp^ m^pm^tmnLlp oqUnLl tU dmJmUm[ml^g hmumpm[nLp]mUp mpqjnLUm^trn hmqnpqm^g^t^ U pUrnpt^ rntqt^mm^nLpjmU hminpqlmU ^Utp nL qnp&^pUtp:
PmhmjJi pmnhp. pmqiftiiiaUial[iajUnLpjnLU, liJuutnuUmlnipjniU, pmq-iftqwUmliwjfiU tuifpnipnLpjnLU, JiiimuinJi uintibnl, JuImuinmjJiU hUm-pwilnpnipjiiLUUtp, jptaunltaltaUpilmjJiU qmUqilmbtajJiU IfipngUtp:
