Experimental methods of exploring multimodal discourse: crossmodal alignment
Автор: Kiose M., Potekhin V., Zubkov O.
Журнал: Вестник Волгоградского государственного университета. Серия 2: Языкознание @jvolsu-linguistics
Рубрика: Моделирование параметров текста для разработки систем искусственного интеллекта
Статья в выпуске: 5 т.23, 2024 года.
Бесплатный доступ
The study advances Crossmodal Alignment Framework to explore multimodal discourse in its three formats - semiotic, communicative and perceptive - via multimodal experiment. It considers the alignment patterns obtained from two semiotic modes (text and image), transferred in two communicative modes (speech and gesture), sensed by two perception modes (visual and audial). The common research framework determines the patterns as modulated by discourse tasks. The study features the results of multimodal experiments with the participants engaged in three discourse tasks: 1) receptive, which presumes obtaining information from text and image stimuli; semiotic alignment patterns are identified indirectly via participants' gaze response; 2) productive, in which the participants communicate the information in monological format; communicative alignment patterns are identified directly via their speech and gesture; 3) receptive-productive, which presupposes the participants perceive information visually and audially; alignment patterns are identified directly via participants' gaze behavior contingent on the stimuli areas of interest and indirectly via their speech response. Data analysis allows to determine and scale the degree of crossmodal alignment to discourse tasks, which helps identify the input of each mode to solving these tasks. The research framework and obtained results contribute to further development of multimodal discourse methods.
Multimodal experiment, crossmodal alignment, discourse task, information construal, semiotic mode, communicative mode, perceptive mode, alignment patterns
Короткий адрес: https://sciup.org/149147489
IDR: 149147489 | DOI: 10.15688/jvolsu2.2024.5.12
Список литературы Experimental methods of exploring multimodal discourse: crossmodal alignment
- Chafe W.L., 1976. Givenness, Contrastiveness, Definiteness, Subjects, Topics, and Point of View. Li C.N., ed. Subject and Topic. New York, Academic Press, pp. 25-55.
- Chernigovskaya T.V., Petrova T.E., eds., 2018. The Gaze of Shroedinger’s Cat: Identifying Gaze Metrics in Psycholinguistic Studies. Saint Petersburg, Saint Petersburg University Publ. 228 p.
- Cienki A., Iriskhanova O.K., 2020. Patterns of Multimodal Behavior Under Cognitive Load: An Analysis of Simultaneous Interpretation from L2 to L1. Voprosy kognitivnoy lingvistiki [Issues of Cognitive Linguistics], no. 1, pp. 5-11. DOI: https://doi.org/10.20916/1812-3228-2020-1-5-11
- Divjak D., Milin P., Medimorec S., 2020. Construal in Language: A Visual-World Approach to the Effects of Linguistic Alternations on Event Perception and Conception. Cognitive Linguistics, no. 19, pp. 37-72. DOI: https://doi.org/10.1515/cog-2018-0103
- Gavrilenko N.N., 2023. Sinteziruyushchiy etap v deyatelnosti otraslevogo perevodchika: determinatsiya otrasli i tekhnologiya formirovaniya neobkhodimykh kompetentsiy [Synthesizing Stage in the Activities of an Industry Translator: Branch Determination and Technology for Forming Required Competences]. Vestnik Volgogradskogo gosudarstvennogo universiteta. Seriya 2. Yazykoznanie [Science Journal of Volgograd State University. Linguistics], vol. 22, no. 3, pp. 57-66. DOI: https://doi.org/10.15688/jvolsu2.2023.3.5
- Iriskhanova O.K., 2014. Igry fokusa v yazyke. Semantika, sintaksis i pragmatika defokusirovaniya [Games of Focus in Language. Semantics, Syntax and Pragmatics Defocusing]. Moscow, Yaz. slav. kultury Publ. 320 p.
- Iriskhanova O.K., 2021. Polimodalnye izmereniya diskursa [Multimodal Discourse Measures]. Moscow, Yaz. slav. kultury Publ. 448 p.
- Iriskhanova O.K., Kiose M.I., Leonteva A.V., Agafonova O.V., Petrov A.A., 2023. Multimodal Collaboration in Expository Discourse: Verbal and Nonverbal Moves Alignment. Lecture Notes in Artificial Intelligence, vol. 14338, pp. 1-14. DOI: https://doi.org/10.1007/978-3-031-48309-7_29
- Keller L., Viebahn M., Hervais-Adelman A., Seeber K., 2023. Unpacking the Multilingualism Continuum: An Investigation of Language Variety Co- Activation in Simultaneous Interpreters. PLOS ONE, vol. 18, no. 11, p. e0289484. DOI: https://doi.org/10.1371/journal.pone.0289484
- Kiose M.I., Izmalkova A.I., Rzheshevskaya A.A., Makeev S.D., 2023. Sobytiye teksta i metateksta v glazodvigatelnom povedenii impulsivnykh i reflektivnykh chitateley [Text and Metatekst Event in the Gaze Behavior of Impulsive and Reflective Readers]. Nauchnyy rezultat. Voprosy teoreticheskoy i prikladnoy lingvistiki [Research Result. Theoretical and Applied Linguistics], vol. 9, no. 1, pp. 115-135. DOI: https://doi.org/10.18413/2313-8912-2023-9-1-0-8
- Kutlubayev M.A., Shagieva D.R., Karimova G.I., Izmalkova A.I., Myachikov A.V., 2023. Pupillometriya v otsenke psikhoemotsionalnogo sostoyaniya i kognitivnykh funrtsiy cheloveks [Pupillometry in the Assessment of Emotional State and Cognitive Functions in Human]. Zhurnal vysshey nervnoy deyatelnosti im. I.P. Pavlova [I.P. Pavlov Journal of Higher Nervous Activity], vol. 73, no. 5, pp. 651-665. DOI: 10.31857/S0044467723050064
- MacSaveny T., 2010. Towards a Description of Descriptive Discourse. GIALens, no. 3, pp. 1-5.
- Mandler J.M., Johnson N.S., 1977. Remembrance of Things Parsed: Story Structure and Recall. Cognitive Psychology, vol. 9, no. 1, pp. 111-151.
- McNeill D., 2006. Gesture, Gaze, and Ground. Renals S., Bengio S., eds. Proceedings of Machine Learning for Multimodal Interaction: Second International Workshop 2005. Berlin, Heidelberg, Springer Verlag, pp. 1-14.
- Nippold M.A., Scott C.M., eds., 2010. Expository Discourse in Children, Adolescents, and Adults: Development and Disorders. New York, Psychology Press. 336 p.
- Potekhin V.O., 2023. Diskursivnye skhemy i zhesty v immersivnoy kommunikatsii: vliyanie faktora tekhnologichnosti [Discourse Schemata and Gesture in Immersive Communication as Affected by Technology Factor]. Cognitivnye issledovaniya yazyka [Cognitive Studies of Language], vol. 5 (56), pp. 395-399.
- Prokofyeva O.N., 2018. Spetsifika yazykovoy i zhestovoy peredatshi zritelnogo vospriyatiya [Visual, Verbal and Gestural Focusing in Russian Descriptive Discourse]. Vestnik Moskovskogo gosudarstvennogo universiteta. Gumanitarnyye nauki [Vestnik of Moscow State Linguistic University. Humanities], no. 18 (816), pp. 257-267.
- Van Eemeren F.H., 2010. Strategic Maneuvering. Extending the Pragma-Dialectical Theory of Argumentation. Argumentation in Context 2. Amsterdam-Philadelphia, John Benjamins, pp. 93-127. DOI: https://doi.org/10.1075/aic.2
- Van Leeuwen T., 2022. Multimodality and Identity. London, Routledge. 190 p.
- Yuan L., Wang B., 2023. Cognitive Processing of the Extra Visual Layer of Live Captioning in Simultaneous Interpreting. Triangulation of Eye-Tracked Process and Performance Data. Ampersand, vol. 11, p. 100131. DOI: https://doi.org/10.1016/j.amper.2023.100131
