New possibilities of combining multimodal game modeling and cardiometric detection in instrumental cognitive science

Imprint

Alexander S. Ognev. New possibilities of combining multimodal game modeling and cardiometric detection in instrumental cognitive science. Cardiometry; Issue No. 28; August 2023; p. 13-18; DOI: 10.18137/cardiometry.2023.28.1318; Available from: new-possibilities-combining-multimodal-game

Introduction and identification of the research problem

Such a rapidly developing interdisciplinary area in modern sciences as cognitive science successfully integrates the theory of knowledge, the achievements in cognitive psychology and neurolinguistics. The latter requires the development and the active application in this field of science of everything that was previously the prerogative of physiology and medicine. However the correlates found in these natural scientific disciplines between the subjective significance of various kinds of events, objects and phenomena and those responses, which accompany the experience of this subjective significance, have been opening new possibilities for understanding the mechanisms of a person’s work with the information he/she receives. Thanks to instrumental recording of cardiac activity and gaze parameters, it becomes possible to identify the most essential elements of a person’s personal cognitive schemes [1-12]. The combination of such measurements with instrumental recording by means of modern polygraphs of the physiological manifestations of a person’s psycho-emotional reactions makes it possible to assess the strength and the nature of the connection between the information we receive and the context in which cognitive processes take place [2, 6, 8].

The majority of these research works somehow involves the use of specially selected unified visual stimuli, the samples or prototypes for which are images created in the second half of the last century for projective testing [1, 2, 7, 8, 11-19]. However, despite their high statistical validity, such stimuli cannot always be classified as really significant for a given particular respondent. And when we are talking about the individual features of the course of cognitive processes of this particular person, then in order to study exactly how and on the basis of what information he/she makes his/her decisions that are vital for him, we also need stimuli that are subjectively significant exactly at the moment of measuring. In this regard, modern cognitivists face a number of questions regarding how to quickly create such stimuli and how then to find common grounds for comparing the data obtained in this case when working with different respondents.

This problem is partly solved by using digital photographs of thematic sand compositions, created by the respondents themselves, as visual stimuli [3, 6-9, 16, 18, 19]. The diverse nature of such constructions and the reflection of deep cognitive processes in them, have shown a high level of productivity of this kind of research. But at the same time, rather large scope of time is spent on building such compositions themselves and on making visual stimuli for subsequent instrumental diagnostics. Added to this is the some- times unwillingness of respondents to build such compositions at a certain time for conducting research.

Partially, this problem was solved by using such a kind of multimodal game modeling as an illustration of a way for solving a problem or a respondent’s story about an object using metaphorical associative cards [6-8, 14, 17]. But in this case, we again partially return to using, even a very large, but however stylized, set of images.

Much greater individualization of this kind of metaphors can be achieved through the use of kinetic sand [18, 19]. However, the respondents are not always immediately ready to use just such means of metaphorical display of their experiences at the very beginning of the experiment.

The study of an ever-expanding set of forms and methods of visual game modeling allows us to assume that the use of thematic patterns, which the respondents themselves create on the surface of light tablets, has quite good prospects in terms of solving this problem. At the same time, it has been taken into account that modern varieties of these devices are equipped with a large set of color illumination of compositions quickly created by the respondents. When formulating this hypothesis, it has been also taken into account that drawing with sand on the surface of a light tablet takes much less time and is less labor-intensive than building metaphorical pictures using other means of multimodal game modeling.

Material and methods

To test the formulated hypothesis, metaphorical compositions were used as a stimulus material, which were created by the respondents in the form of thematic drawings on the surface of the most common light mini-tablets. The working surface of the plates used was a rectangle with sides of 300 and 400 mm. All of them were equipped with low-voltage control panels for color illumination. A set of colors was selected based on the presence of blue, green, yellow and red, which were classified as those basic in the Luscher test. This was done with the expectation that in the further processing of the obtained results, the use of such a means of their operational assessment as a balance between the parasympathetic and sympathetic systems of the respondent, for example, using integrated coefficient SHIPOS [15].

The universal tasks for all respondents were as follows:

14 | Cardiometry | Issue 28. August 2023

• –    task No. 1: an image in a spontaneous mode on the surface of the light tablet of any pattern that the respondent wants to create when thinking about the problem that worries him at the moment;

• –    task No.2: an image in a spontaneous mode on the surface of the light tablet of any pattern that the respondent wants to create when he/she imagines his/ her own ideal solution to the problem that worries him at the moment;

• –    task No. 3: selection of such a lighting option that is most compatible with this “problem” image;

• –    task No. 4: selection of such a lighting option that is most compatible with this “problem-free” image.

Each of these patterns with the selected lighting option has been photographed with the respondent’s personal smartphone. Then these images have been used as stimulus materials during cardiometric measuring accompanied in parallel by recording of respondents’ electrodermal activity. As means of instrumental diagnostics, computer-assisted cardiometric device Cardiocode and “Barrier-14” as a recorder of human physiological reactions were employed.

The patterns created by 78 respondents and reactions to each of the images chosen as visual stimuli were discussed individually during the post-test conversation, which was conducted in the form of an indepth interview or an interview described in detail in [6-8, 11-19].

Results and discussion

Figure 1 given herein shows some examples of respondents completing tasks Nos.3 and 4. In the photographs, the left of the patterns on each of the visual stimuli was created as an illustration of the problem that the respondent was worried about after completing task No. 3, and the right as what he wanted to depict in the course of thinking about what the result of a successful resolution of this problem after completing the task No. 4 might be.

Figure 2 given herein shows an example of how the respondents rank the various options for highlighting according to the degree of acceptability for themselves in the course of performing tasks Nos. 3 and 4. In the upper left corner there is an image with an illumination, which this respondent has considered to be the most appropriate to the pattern that reflects his feelings of a problem that is significant for him/ her. In the upper right corner there is an illuminated image, which this respondent has considered to be

Figure 1. Examples completing by the respondents of task No. 3, which consisted in selecting such a highlighting option that was most compatible with the experiences of a problem that was significant for the respondent (left photo) and of task No. 4 selecting such a highlighting option that was most compatible with ideas about the successful resolution of this problem (right photo).

Figure 2. Examples of how the respondents are looking for the most appropriate illuminating options that best match either the respondent’s feelings about a problem that is significant to him/her, or his/her ideas about what would be an appropriate illustration of a solution to this problem.

the most consistent with the pattern, reflecting his/ her ideas about the successful resolution of this problem. The two lower images show which illumination options have been classified by the respondent as less suitable for both the first and second states. In each of the four presented photographs, the left pattern corresponds to the result of completing task No. 1: an image in a spontaneous mode on the surface of a light tablet exhibiting any pattern that the respondent wants to create when thinking about the problem that worries him/her at the moment; and the right pattern in each of the four presented photographs corresponds to the result of completing task No. 2: an image in spontaneous mode on the surface of a light tablet exhibiting any pattern that the respondent wants to create when he/she imagines his/her ideal solution to the problem that concerns him/her at the moment.

Practice has shown that it took for all the respondents, without exception, no more than a quarter of an hour to complete all the above listed tasks. This means that, on average, the creation of visual stimuli in this way, which have an unconditionally high level of subjective significance for each respondent, takes at least half the time compared to other types of multimodal game simulation. At the same time, refusals from participation in this type of multimodal game simulation are found two to three times less as against the case with working with kinetic sand.

The validity of our hypothesis was also confirmed by instrumental diagnostics, some results of which are shown in Figure 3 herein. It shows samples of visual stimuli created in the way described above and shows the parameters of psychophysiological reactions to these stimuli of the respondents who created these compositions.

As follows from the example shown in Figure 3 herein, such a change in the recorded indicators has been reported, when the transition from the results of task No. 1 to the results of task No. 2 leads to a significant decrease both in the Baevsky stress index, the heart rate and in the total area under the curve describing galvanic skin response of the respondent to the presented stimulus. At the same time, during the post-test interview, almost all respondents noted that working with the images created by them in the process of completing task No. 2 calmed them down and formed confidence in the successful resolution of their problem. Additional evidence in favor of the fact that the results of completing task No. 2 were combined in respondents with those states, which were more emotionally balanced, was also the fact that the yellow and green colors were most often chosen as suitable illumination, while for task No. 1 prevailed were red and rarely purple.

An important feature for the study of cognitive processes was also the fact that in the course of post-

The result of completing task No. 1: SI=361; HR= 82; area under the GSR curve 70306 conventional units

The result of completing task No. 2: SI=161; HR= 77; area under the GSR curve 6355conventional units

Figure 3. Examples of typical recorded parameters of psychophysiological reactions to presented stimuli. Legend: SI – Baevsky’s stress index as an indicator of the degree of balance between the effect on the heart rhythm produced by the sympathetic and parasympathetic nervous system; HR – heart rate; GSR curve – a graph of changes over time in the value of the galvanic skin response recorded using a polygraph.

test conversations, the respondents described in detail, without difficulty, exactly how they came to this kind of conviction. It is also noteworthy that, without any hesitation, the majority of respondents during such a conversation described in detail the essence of the problems that worried them and the foundations, value orientations, and deep meanings that guided them, when they chose possible ways to solve the above problems. The peculiarity of such openness is that immediately after displaying in a spontaneous mode on the surface of the light tablet the pattern that the respondent wanted to create, when thinking about the problem that was worrying him at the moment, as a rule, our respondents refused to state the problem in question.

Conclusions

The evidence data obtained convincingly testify to the effectiveness of the use of sand patterns on light tablets, created by the respondents as illustrations of the problems, which concern them, and possible ways to solve them, as visual stimuli for the instrumental cognitive science. The creation of such stimuli is less labor-intensive compared to the use of other types of multimodal game simulation for the same purposes. This procedure, to a lesser extent, causes some resistance by the respondents to the forthcoming instrumental diagnostics and may well serve as an operational means for conducting large-scale studies of cognitive processes, during which the subjects will have to make their vital decisions.

This article was prepared based on the results of research carried out at the expense of budgetary funds under the state assignment of the Financial University “Digital communications technologies for the integration of residents of the LPR and DPR into the socio-cultural space of modern Russia” (VTK-GZ-PI-14-23)