CC BY
12
DOI: 10.24234/wisdom.v3i2.782
Nataliia HOLUBENKO
Kyiv National Linguistic University, Kyiv, Ukraine E-mail: [email protected]
©
PHILOSOPHICAL AND DISCURSIVE APPROACHES TO THE CATEGORISATION OF MODAL MEANINGS IN MULTIMODAL TEXTS
Abstract
The question of overcoming barriers between the study of language and philosophical, immanent perception of the image and other aspects that formulate a coherent text is still open. A multimodal approach to understanding the news coverage strategy in the Telegram can reveal the discursive function and the multimodal perspective. One uses a combination of linguistic and philosophical approaches to analyze multimodal texts. This combination proves that the homogeneous use of the model of proving empirical reality employing philosophical and linguistic segments reveals the modulations. The article aims to explore the theoretical foundations of philosophical and discursive approaches to categorizing modal meanings in multimodal texts. The study results of the ideational architectonics of the 65 analysed multimodal posts on the Ukrainian Telegram channel "SUSPILNE NEWS" as of April 18, 2022 have pointed to a tendency of disconnection of categorical modalities. 19 multimodal texts out of 65 revealed a complete symbiosis of philosophical and linguistic modus operandi (the presence of all linguistic and extralinguistic resources, including video sequences), while 46 posts acted as monomodal tests. Regarding the number of "likes" and views, the posts with all extralinguistic and semiotically heterogeneous resources were more popular and, therefore, more exciting and easier to perceive.
Keywords: modus, plurisemioticity, functionality, indexicality, symbiosis of approaches, semiotically heterogeneous resource.
Introduction
This article analyses the theoretical founda-
tions of the distribution of categorical types of
Multimodal texts shape receptive processes. To understand the multimodal test, we are not talking exclusively about movies, novels, or comics, which constitute the classic forms of multimodality. The problem of multimodal tests created due to new digital technologies is new and relevant. Recent research in this area (Metros, 2008; Chia & Chan, 2017); (Jancsary, Hollerer, & Meyer, 2016) has demonstrated that the XXI century reader, called upon by new media in everyday life, devotes less and less time to traditional media and deals more and more with multimodal texts.
modal meanings in multimodal texts. Conse-
quently, the tasks of:
• to investigate the basic theoretical positions of multimodality;
• to carry out the analysis and description of the full range of means of creating multimodal test meaning;
• to prove that extra-lingual, semiotically heterogeneous resources are used to create multimodal text in Telegram channels, which are more effective than linear tests; to identify the main categories of multimodality as a heterogeneous complex of philosophical and lin-
guistic means;
• to identify the types of metafunctional organisation of semiotic resources and categorisation of modes in a multimodal text;
• to determine the nature of relations between modes of expression of multimodal texts. The multimodal text synthesises different
symbolic and textual modes, gravitating toward transgressive connections (Caple, 2018).
It differs from a monomodal text (i.e., a document consisting exclusively of writing) by the presence of an image or sound, the ability to combine one or more modes of expression (Mu-khametshina, 2020).
Moreover, depending on the medium it was developed, it can be linear (e.g., a comic strip on paper) or nonlinear (e.g., hyperlinked text on the web). This essential characteristic explains that multimodal text encompasses a deployed set of combined modes of expression that can be broadcast on paper and through various electronic gadgets: iPods, iPads, and laptops (Dahlström, 2021).
Multimodal text (media document) belongs to the media environment, which combines different modalities of expression, including print, audiovisual, telephone, and computer. In short, it is a digital type of document (digital document). It is composed of various components according to variable geometry: text (written texts or spoken speeches), images (e.g., photos, maps, animated images - video or fixed images), hypertext links, sounds (e.g., texts, music, and various sound systems) (Yong & Qing, 2020). Therefore, multi-modality, which is inherent in the presence of different sign, speech, and auditory modalities, expresses the category of intensional modality (philosophical concept). The process of their perception is possible on two levels: first, there is a juxtaposition or combination of different modes, and next, these same modes have a multimodal character (for example, the video sequence in a digital channel Telegram document includes animated images and sounds, both being transmitted together). Because multimedia is still widely
regarded as an adjunct to texts, sounds, and images (broadly speaking), it is given the qualities of unifying modes. Nevertheless, as Fouche argued, it is clear that multimedia cannot be represented solely based on a pure and straightforward juxtaposition of the qualities attributed to its constituent modes. The articulation of modes of expression (modes) forms a multimodal text; the same applies to media, which requires the reader to have a metaphorical mindset to perceive information assembled from different multimodal documents.
The scientific research in the context of our work leads us to the fact that multimodality is a natural phenomenon in the communicative process, which includes visual, auditory and other modalities. The author defines multimodality as a kind of symbiosis of scientific fields that have differences in methods, research directions and theoretical foundations, but also that illustrate the interdisciplinary nature of multimodality and the need for further development of interdisciplinary multimodal research (Gu, 2006). The distinction between meaning and word reference is demon-stated by Frege, supplemented by Peirce's reflections on indexicality as a mode of signification (Rajendra & Kaur, 2022).
The signification of nonverbal means, which constitute an autonomous and closed system in multimodal texts can be treated as elements of ordered semiotic nonverbal systems, as well as individual disordered elements in the system. It is essential and relevant today to say that the multimodality actively used nowadays is being unpredictable in the actual embodiment. Subsequently, in 1945, Wittgenstein laid the foundations for a philosophy of language that integrated into Illocutionary Logic and modern pragmatics (Wanselin, Danielsson, & Wikman, 2021).
We identify discursive approaches to categorizing modal meanings in multimodal tests as discourse appears to be integral to sociocultural factors (Stepanyan, Manukyan, Tevosyan, & Il-yushina, 2022).
Discursive approaches are closely related to
primary philosophical approaches and are revealed in texts through intertextual connections where individual multimodal texts acquire the characteristics of other modified sociocultural discourses. Thus, the combination of social practice and discourse reflects the philosophical origin of multimodality. Therefore, the modal meanings of multimodal texts can be also considered from the point of view of philosophy.
This paper is guided by Kantian followers
These typologies allow it to reveal the in-tersemiotic synergies operating in multimodality and adequately explain their deictic character.
Let us consider each of the items of the three types of organisation of semiotic resources and categorisation of modal meanings of multimo-dality and metafunctional organisation of the text. From the perspective of structuring idea-tional denomination, we can say that the relationship and equivalence between text and image are competitive. Hill and Dorsey (2019) call them complementary or dissimilar. The degree of text/image homogeneity depends on the context and lexical cohesion, resulting in intersemiotic coherence. The complementarity of text/image depends on the functional specialization of each. Text gravitates toward sequential relations and categorical distinctions, while image emphasizes spatial relations and modifications of degrees and proportions.
The construction of the interactive/interpersonal meaning of text-image correlativity is reflected in the role of interactants, i.e., information requests and transmissions (Veum, Silian, & Ma-
Kress and Van Leeuwen's (1996) theory of text/ image relationships in multimodal texts, drawing on Holliday's (1985) work from the perspective of the System of Functional Linguistics (SFL) (Hill & Dorsey, 2019). We distinguish three types of metafunctional organisation of semiotic resources and categorisation of modal meanings from a multimodal perspective. In the (Table 1) the author's solution is shown:
Table 1.
agere, 2020).
The variety of possibilities, probabilities, and certainty of images and text leads to varying degrees of interactant attachment to multimodal text (Wanselin et al., 2021).
Rational judgment and emotional perception play an essential role in this process in the first place.
The construction of compositional/textual meaning from the perspective of analyzing the relationship between text and image works with a reliance on theme/rhyme. The animated image opposes the theme/rhyme in the text through its linear structure (from left to right). Thus, philosophy and then linguistics and semiotics contributed to defining the nature of the relationship between the modes of expression of multimodal texts, concretizing, above all, the relationship between text and image.
Materials and Methods
The practical material of the research comprised 65 posts (multimodal and monomodal
Organisation of Philosophical-Semiotic Resources and Categorisation of Modal Meanings of Multimodality
Type Explanation
Representative or idealistic. Text and image construct the nature of the event, its participants, and its context.
Interactive or interpersonal. The construction of text and image, the nature of the relationship between sender and receiver.
Compositional or textual. Text and image are in different relationships, depending on accentuation and purpose.
texts) on the Ukrainian Telegram channel "SUSPILNE NEWS" as of 18.04.2022 (news rubric). The techniques of organisation and perception of the texts, which increase the audience's involvement, are considered. In the perspective of critical multimodal discourse analysis, the discursive categories of multimodality of news texts in Telegram channels are considered a form of socio-philosophical practice that reveals the hidden ideological intentions and presents the analytical structure in three measurable ways categories: text, discourse, and social practice. The content of users' preferences in the analysed posts are described.
The exploration methodology included an analysis of the components of multimodal news texts on the Telegram channel and a synthesis of their integration features in the textual space. In addition, the method of observation made it possible to select verbal and visual fragments. The quantitative analysis method was used to determine the frequency of multimodal means, and discourse analysis methods made the parsing of interesting text fragments with explicit semantic grounds. Structural and functional methods were used to analyze the multimodal texts of the news section on the Telegram channel to highlight the pragmatic functioning of multimodal modus op-erandi.
The ideal architectonics of 65 news texts of the news section is considered, where the analysis of full and partial multimodality by interacting verbal and non-verbal components is carried out. The functioning of extralinguistic means of multimodality expression and the bases of their multimodality as factors of formation of interactivity and compositionality of multimodal text have been investigated.
Results
The strategies of news distribution on the Telegram channel, especially under war conditions, have changed significantly. The resource "SUSPILNE NEWS" selected for the study was
guided by the fact that it is one of the leading in the segment of Telegram channels of the new rubric. The number of its subscribers since the beginning of the war on 24.02.2012 amounts to 288 thousand people. Among the Ukrainian Internet media, the channel takes first place in the number of views.
The organisation of the construction of multimodal (digital) texts and the use of necessary modalities here takes place on at least two levels: first, there is a juxtaposition or combination of different modes, and second, the same modes are multimodal in nature (for example, a video sequence including animated images and sounds, with both being transmitted together).
The results of the conducted work indicate that the idea architectonics of the 65 analysed multimodal texts (posts) on the Ukrainian Telegram channel "SUSPILNE NEWS", as of 18.04.2022 (news rubrics) is characterized by the disconnection of categorical modulations. Nevertheless, the analysis showed that using the entire symbology to create multimodal test values on this channel is used less frequently but is perceived more actively.
Out of 65 posts, a complete symbiosis of philosophical and linguistic modes (the presence of all linguistic and extralinguistic resources, including video sequences) was seen only in 19 multimodal texts, and 46 of them were monomodal tests - to create multimodal text in this Telegram means. Regarding the number of reactions of likes and views, posts using all extralinguistic and semiotically heterogeneous resources prevail.
Thus, using basic categories of multimodality as a heterogeneous set of philosophical and linguistic means determined the nature of influence and the relationship between the informative channel and the recipient. Within the framework of discourse and philosophical modes, we distinguish three basics for this work: • Linguistic modus (multimodality in the text as
such) - textual marker (linguistic design) -
structures the text, brings to the fore the phe-
nomenon of cohesion;
• Empirical modus - functions at the level of representation of reality (events, processes) and is superimposed on the linguistic system of transitivity;
• The interconnections represent interpersonal modus within society regarding speaker/author and recipient.
The complex use of this spectrum of modalities in the design of multimodal text in Telegram is more comfortable and understandable for the
Thus, the editorial board writes a short subplot for each post of a multimodal text. In it, the editorial makes it clear that there is a story but does not tell it - the discourse modus as a form of social action constructing society and culture is the most influential at 8.2%, and it is characterized by the presence of three philosophical-se-miotic modes of multimodality; critical discourse modus deals with social problems 3.2%, and it is characterized by linearity; discourse modus performing an ideological function 1.4%; - discourse modus dealing with the interpretation of textual material and the connection between text and society mediated 1.2% - simple textual message. The discourse modus as a form of social
user. The popularity and the number of views increase precisely due to the attraction of linguistic design of visual, auditory, and kinesthetic communication channels for the perception and transmission of information. This channel is of interest because it mainly posts portal news in the form of multimodal text (on average more than 100 posts a day, which is a record among the analysed channels) and links to the site. Each post was examined fractionally by the presence of modal discourse (Fig. 1).
action reveals the most popular rubrics of multimodal texts among Ukrainians in the analysed Telegram channels. The most popular are losses of the Russian army (21%), general news (evacuation, communication, business, volunteering), (21%), missile strikes (16%), mortality of Ukrainians (16%), military and tactical attacks of the enemy (12%), armed forces of Ukraine (9%), sanctions against Russia (7%). The users are mostly interested in publications with strong social charge, like losses of the Russian army and less interested in sanctions against Russia. The most popular technique of organizing multimodal text is a combination of text, photo, video, and hyperlinks to the publication's website. The most
Figure 1. Discourse Modus Multimodal Tests of Telegram News (Realized by the author).
popular method of organizing multimodal text in Telegram is the combination of text with a hyperlink to the publication website, photo, video, and the presence of the iconic turn, according to G. Kress (Stepanyan et al., 2022). Consequently, the text involved in the Telegram channel is characterized by informal presentation, emotionality, and reasonable intrigue.
Discussion
The question of the study of language and immanent philosophical perception of the phenomena of multimodality, cohesion, and coherence is still open. A basic definition of the concept of multimodality is considered in his works (Savic, 2020). Multimodality considers texts that communicate through speech, gestures, gaze, and other visual forms. The author is convinced that modern communication is increasingly considered from a multimodal perspective, and multimodal texts are becoming more relevant and attracting the attention of linguists. Even though scholars (Wanselin et al., 2021) differentiate language and images as separate communication systems in studying text multimodality (primarily visual components), their interpretation often depends on linguistic structures. Dosenko's (2022) study of text multimodality focuses on the integration of linguistic elements within the structure of the text, encompassing visual elements as well. We agree that when it comes to the written or printed word, attention is directed first to typography and content and then to linguistic structures or poetics of meaning. Typeface, colour, graphic sign, and elements of graphing are nonverbal means of conveying information in the multimodal text.
In the works of the European school of Veum, Siljan and Maagera (2020), linguists explain the main areas of multimodality covering discourse (socially formed knowledge), design (the conceptual aspect of expression, including the combination of semiotic components), creation (explicative material), distribution (through which the product reaches society).
Skaar (2009) notes that modus (semiotic resources) do not function to create communicative oppositions to the text but combine it into a whole, which is not divided into parts but is perceived as an integrated whole. Their search is associated with discovering modus operandi in the multimodal text. Let us coincide with the similarly problematic work and remark on their functionality in ensuring the coherence of textual space.
The vision of multimodality presented in Ser-afini (2010) is implicated in this work. The authors reveal a comprehensive approach to the study of the categories of multimodality; namely, they formulate their own structure of multimodal tests. Although the explanation of their multimo-dality relies on a well-known definition, we draw attention to the fact that the phenomenon of mul-timodality involves many scientific angles, ways of thinking, and multiple approaches. Another work on multimodality examines a combination of philosophical and linguistic categories and approaches (Liu, 2013). We agree that a multidimensional toolkit in studying such a complex topic cannot ignore philosophy and systems-functional linguistics. We believe that in analyzing multimodal texts and the modalities denoting them, systems-functional linguistics can illuminate all aspects of the question, revealing a critical rethinking in the formulation of texts in a new dimension.
On the involvement of the modus operandi of speech and the sound of music or the principles of integrating semiotic resources writes (Un-sworth, 2014). The author is guided by the principle of compression of meanings, about the integration of semiotic means at the text level, reaching them to higher levels, where interpretation of meanings is possible. Exploration of the combination of semiotic means is considered fundamental in the study of categories of modus in multimodal texts.
Let us add the work of Hart & Marmol Queralto (2021), which is based on the foundations of cognitive metaphor theory and system-functional grammar. The authors appeal to the use of a cog-
nitive approach to the study of multimodality since the metaphorization of multimodality produces textual lyrical images in a new connotation.
Some works involve corpus linguistics in analyzing multimodal tests (Caple, 2018). In particular, the scholar identifies multimodality as a separate genre. The author structures multimodal text by levels containing: primary, location, rhetoric, orientation, and genre. We agree that this vision of multimodality allows us to deepen the empirical analysis in the process of parsing the semiosis of multimodal meaning.
In the context of the disclosure of the goals and objectives of the work, we adopt that the concept of this study of multimodality reflects the elements of the model of proving empirical reality with the help of philosophical and linguistic segments. The symbiosis of linguistic and philosophical categories reveals the modalities that form a multimodal text. It, in turn, is characterized by explicit (external), implicit (hidden), and integrated categories. Moreover, the allocated types of multimodality, in turn, are encoded in different modus (auditory, visual, tactile, musical, ekphrasis). The representation of these concepts as a homogeneous reality in the multimodal text of Telegram channels reveals to the user the real and virtual artistic world and the transgression between material and mental informative fields. So, the encoding of the multimodal test in the Telegram channels occurs in one modus and several simultaneously.
Conclusion
Multimodality allows you to focus on the entire spectrum of semantically significant resources used in communication. Note that their choice depends on the chosen channel, the specific communicative situation, and technical capabilities. It also seems unreal to make an accurate inventory of all means since information technology creates boundless communicative opportunities. Therefore, fixing the entire spectrum is hardly possible because individuals' crea-
tive potential is limitless.
Multimodality admits that the representation and content of an utterance are always based on the interaction of the modalities that are basic to the formation of meaning. Extra-lingual, semiot-ically heterogeneous resources are used to create multimodal text in Telegram channels, but there are posts characterised by linearity (monotexts).
The combination of multimodality, cohesion, and coherence diversifies the modalities and thus improves the formation of material for use in the daily social interaction of people. Including se-miotic resources realises the material, cultural and historical aspects of modus usage in the multimodal test. In this perspective, three types of metafunctional organisation of semiotic resources and categorisation of modal meanings are identified, and the nature of relations between modes of expression of multimodal texts is determined. Bringing the empirical reality to users and readers with the help of symbiosis of philosophical and linguistic segments allows the disclosure of the modalities that form a multimodal text. Explicit, implicit, and integrated categories are encoded into different modulations, whose representations reveal to the user the real and virtual artistic world and the transgression between material and informative mental fields in the multimodal text of Telegram channels. Consequently, the symbiosis of philosophical and discursive approaches to categorizing modal meanings in multimodal texts reveals its indexi-cal purposes.
Thus, the multimodal texts of the Telegram resource "SUSPILNE NEWS" in which multi-modality, cohesion, and coherence are traced are more popular and therefore clearly fulfil their idiomatic function since they contain information-active discourse-semiotic modulations, which not only convey the main and attach secondary connotations to the main content.
Further application of these scientific search results is promising for studying the construction of multimodal texts of Internet resources. In addition, it can be used in teaching courses on "New Advances in Modern Linguistics", "Cog-
nitive Approaches and Semiotics of Internet Texts", and "Fundamentals of Language Communication Theory".
References
Caple, H. (2018). Analysing the multimodal text. In Corpus approaches to discourse (pp. 85-109). Routledge. https://doi.org/10.-4324/9781315179346-5 Chia, A., & Chan, C. (2017). Re-defining 'reading' in the 21st century: Accessing multimodal texts. Beyond Words, 5(2), 98105.
Dahlstrom, H. (2021). Students as digital multimodal text designers: A study of resources, affordances, and experiences. British Journal of Educational Technology, 53(2), 391-407. https://doi.org/-10.1111/bjet. 13171 Dosenko, A. (2022). The phenomenon of the multimodal service media text of the communication platform. State and Regions. Series: Social Communications, 4(48), 97-101. Retrieved from http://-www.zhu.edu.ua/journal_cpu/index.ph p/der_sc/article/view/864 Gu, Y. (2006). Multimodal text analysis: A corpus linguistic approach to situated discourse. Text & Talk - An Interdisciplinary Journal of Language, Discourse Communication Studies, 26(2), 127167. https://doi.org/10.1515/text.2006.-007
Hart, C., & Marmol Queralto, J. (2021). What can cognitive linguistics tell us about language-image relations? A multidimensional approach to intersemiotic convergence in multimodal texts. Cognitive Linguistics, 32(4), 529-562. https://doi.org/10.1515/cog-2021-0039 Hill, C., & Dorsey, J. (2019). Expanding the map of the literary canon through multimodal texts. In Handbook of the Changing World Language Map (pp. 1-13). Springer International Publishing.
https://doi.org/10.1007/978-3-319-73-400-2_119-1 Jancsary, D., Höllerer, M. A., & Meyer, R. E. (2016). Critical analysis of visual and multimodal texts. Methods of Critical Discourse Studies, 3, 180-204. Retrieved from http://epub.wu.ac.at/6126-/1 /Dennis_etal_2016_SAGE-critical-analysis.pdf
Liu, J. (2013). Visual images interpretive strategies in multimodal texts. Journal of Language Teaching and Research, 4(6). https://doi.org/10.4304/jltrA6.12-59-1263
Metros, S. E. (2008). The educator's role in preparing visually literate learners. Theory into Practice, 47(2), 102-109. https://-doi.org/10.1080/00405 840801992264 Mukhametshina, D. R., Danilova, J. J., & Bu-bekova, L. B. (2020). Multimodal text: Authoris linguopragmatics and extralinguistic factors. Applied Linguistics Research Journal, 4(9), 112-117. https://doi.org/10.14744/alrj.2020.199 81
Rajendra, T. D., & Kaur, S. (2022). Print-based multimodal texts: Using illustrated poems for generating ideas and writing narratives. Studies in English Language and Education, 9(1), 278-298. https://-doi.org/10.24815/siele.v9i1.21830 Savic, V. (2020). Visual literacy for young language learners: Multimodal texts in content-based instruction. In Handbook of Research on Cultivating Literacy in Diverse and Multilingual Classrooms (pp. 166-189). IGI Global. Serafini, F. (2010). Reading multimodal texts: Perceptual, structural and ideological perspectives. Children's Literature in Education, 41(2), 85-104. https://doi.-org/10.1007/s10583-010-9100-5 Skaar, H. (2009). In defence of writing: A social semiotic perspective on digital media, literacy and learning. Literacy, 43(1), 36-42. https://doi.org/10.1111/j.1741-
4369.2009.00502.x Stepanyan, A., Manukyan, E., Tevosyan, L., & Ilyushina, M. (2022). Legal regime for scientific works in the digital age. WISDOM, 21(1), 117-122. https://doi.-org/10.24234/wisdom.v21i1.625 Unsworth, L. (2014). Multiliteracies and metalanguage: Describing image/text relations as a resource for negotiating multimodal texts. In Handbook of research on new literacies (pp. 377-406). Rout-ledge. https://doi.org/10.4324/9781410-618894-24
Veum, A., Siljan, H. H., & Maagera, E. (2020). Who am I? How newly arrived immigrant students construct themselves
through multimodal texts? Scandinavian Journal of Educational Research, 116. https://doi.org/10.1080/00313831.2-020.1788147 Wanselin, H., Danielsson, K., & Wikman, S. (2022). Analysing multimodal texts in science - A social semiotic perspective. Research in Science Education, 52(3), 891-907. https://doi.org/10.1007/s111-65-021-10027-5 Yong, H. U., & Qing, Q. I. U. (2020). A study on verbal and image relations in multimodal texts from the perspective of in-tersemiotic complementarity. Canadian Social Science, 16(10), 50-56.
