Signmatics: An Interactive Digital Based System for Multimodal Learning of Hearing Students in Bulgarian Sign Language

In recent years, the learning of sign language by hearing people as a second language has become an up-to-date issue, generating fruitful discussions and innovative research. In the scientific paradigm of researchers, sign languages, as a significant factor in nurturing social interaction ( Adam and Braithwaite, 2022 ; Kusters, 2020 ; Balkanska, 2013 ; Balkanska and Lozanova, 2021 ), are on an equal footing with spoken languages in terms of their naturalness, vitality, autonomy, complexity and expression ( Fenlon et all., 2007 ; Dimich and Sesum, 2011 ). Learning a language that is realized in a visual modality different from spoken language (M2) is a real challenge for hearing people, which stimulates the scientific ambition to find ways to optimize sign language learning. Traditional teaching models are being transformed and expanded to include a new “digital” level, in which multimodality as a natural feature of human perception, an important and specific fragment of the categorization of the world, continues to reign – a problem that is the subject of constant scientific interest ( Tsankov and Levunlieva, 2024 ; Tsankov and Dermendzhieva, 2024 ; Dermendzhieva and Tsankov, 2023 ; Dermendzhieva and Tsankov, 2022 ).

In light of these changes, visual language researchers are focusing their creative efforts on using operants and algorithms based on advances in digital technologies for teaching and mastering the individual components that make up their complex composite as semiotic systems.

The Bulgarian Sign Language (BGSL) is a legitimate, strictly regulated language, different from the Bulgarian language, and elevated to a pedestal by the members of the cultural community of Bulgarian

• ^*Corresponding author: diana.georgieva@trakia-uni.bg

  
  
  

  © 2025 by the authors. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license ( https://creativecommons.org/licenses/by/4.0/ ).

deaf people. Its teaching and study as E2 (M2) is an integral part of higher education in Bulgaria. The modern language training of specialists in the separate structural units of universities requires special attention in terms of the quality of formation of a wide range of competencies ( Lozanova and Stoyanova, 2022 ; Angelova, 2017 ), the search, development and application of effective methods and technologies operating in a digital format. This becomes especially relevant at the stage of increasing competitiveness in the global educational space, developing international cooperation of the university in connection with the internationalization and the implementation of academic mobility programs.

On the chronogram of time, signs of introducing digital innovations into the practice of teaching and learning visual language were positioned about two decades ago, when a team of specialists created an interactive system called Sign Tutor ( Aran et all., 2009 ). The interactive technology is designed to master basic language knowledge by hearing students learning Turkish Sign Language (TSL) as a second sign system. The generated visual feedback, providing automatic assessment of the produced sign unit, and the source of information about the precision of the performed sign are among the dominant advantages of the system. The implanted integrated circuit for unifying the parameters of the sign, including nonmanual signals – linguistic head movements, give the technology a unique style of functioning.

A major priority in de Villiersе (2014) research is the development and testing of a visually-based system for mastering language content in South African Sign Language (SASL). An intriguing feature of the technology, which surpasses existing sign language learning systems to date, is its capacity to produce detailed and specific feedback to the user. The new algorithm, taking into account the user‘s experience, automatically guides them to the correct model of linguistic behavior. An additionally introduced operation allows the feedback to take the form of a context menu.

The quest to dicovering optimal solutions for early communicative intervention in deaf children ( Ackovska, Kostoska and Gjuroski 2012 ) finds expression in the creation of an interactive e-learning platform in Macedonian Sign Language (MSL). The collection of game modules and panels is oriented towards optimizing language preparation, improving memory parameters and mental capacity in deaf children. They are encouraged to perceive manual symbols reproduced by 3D animations of a child who follows the user‘s choice: dactylem, word, sign denoting a specific object. A mobile application for children SiLearn, which functions as a visual lexicon, was created by another team of authors ( Joy, Kannan and Sreeraj, 2019 ). Language tests of 28 deaf students show a rapid pace of mastering sign vocabulary.

The research initiative of scientific and cultural figures ( Papadimitriou et all., 2025 ) leads to the successfully implemented SL-ReDu project, which focuses on the teaching and learning of Greek Sign Language (GSL) as E2. The high results achieved by 150 students are determined by the module introduced in the system for visual detection of isolated lexical items, and the HRNet framework for detection of the skeleton of the body, hands and face of the communicator in 2D and 3D format.

Another research project ( Vijitkunsawat et all., 2023 ), demonstrating the value of digital solutions, found evidence of a multimodal learning space modeling technology that stimulated the learning of Thai Sign Language (ThSL). In experimental conditions, students were encouraged to independently select lexical items and practice the language material using animation.

A developed product in the form of an avatar complements the digital achievements related to the study of American Sign Language (ASL) and reflected in the scientific literature ( Quandt et all., 2020 ). The graphic model is three-dimensional and acts as an educator teaching language content at a basic level.

In the systematized scientific analyses on a global and especially on a national scale, a free research niche is still being discovered, which awaits to be updated with new data related to the remarkable potential of software programs offering multimodal options for teaching and mastering a semiotic system functioning in a triad of codes: visual, motor and spatial.

This is a convincing argument that served as a framework for organizing and conducting the experimental study, the goal of which is related to testing an interactive digitally based system for mastering linguistic content in Bulgarian Sign Language by students who study it as a second language (E2). The object of the study is the formation of sign competence in hearing students of the scientific specialty of Special Education, receiving academic training in BGSL, and the subject is the optimization of the educational pedagogical practice for teaching and learning the national visual language through a multimodal educational environment, designed and formed through the resources of the interactive computer program Signmatics.

Signmatics is an interactive multimodal computer program for learning Bulgarian Sign Language (BGSL), including multiple communicative modes – gestural, oral, written, tactile. The digital technology, version 1.0, was created in 2009 within the framework of the social project Grant M-Tel by the Union of the Deaf in Bulgaria (UBD). A diverse team of highly qualified specialists, distinguished by their creative charge, dedication, responsibility and innovation in their mission to establish and popularize BGSL in a broad social and educational context, participated in the development and administration of the program.

The program consists of several modules. The first module is a lexicon of 5200 sign units, which are a solid basis for the formation of lexical competence in BGSL. Of particular note is the built-in filter, which provides an additional opportunity to separate signs according to grammatical (parts of speech: signs nouns, signs verbs, signs adjectives, signs adverbs, etc.) and age criteria (preschool age, primary school age, etc.).

The second module is related to options for combining signs into complete wholes – sentences, which implies mastering morphological and syntactic skills – an integral part of sign language competence. The submodule with pre-composed grammatical constructions complements the program design by demonstrating the linguistic capacity of Bulgarian sign language.

The third module includes content that introduces the user to the unique essence of BGSL and the distinctive culture of the Bulgarian deaf people. This is extremely valuable information that acts as a motivator for immersion in a new subject.

The last module is intended for checking or self-checking knowledge. The attractive algorithm that is followed turns the mastery of signs into an intriguing activity that leads to successful results.

Materials and Methods

Design

The study used a quasi-experimental design of non-equivalent groups to evaluate the effectiveness of an interactive digitally-based Signmatics system in an educational environment. The choice of design is practically justified – the researcher has the opportunity to form two comparable groups, one of which receives an educational intervention with digital technology, and the other continues to use conventional methods for studying BGSL.

The independent variable was decided to be the digitally based Signmatics system used during the lecture sessions. The dependent variables defined were the following: perception and expression of the sign parameters (phonology), knowledge and use of lexical sign units (vocabulary), knowledge and use of morphosyntactic information (morphosyntax).

The formulated research question concerns the existence of a significant difference between the academic achievements of students who study BGSL using the digital program Signmatics, compared to those who are taught using conventional methods for mastering linguistic content.

The research question suggests the following hypotheses:

• Н₀₁: The difference between the average levels of operating with the visual-spatial characteristics of signs by students placed in different experimental conditions (using/not using interactive digitally-based technology) will not be distinguished by a high significance value.

• Н₀₂: The difference between the average levels of knowledge and use of lexical sign units by students in different experimental conditions (using/not using interactive digitally based technology) will not be distinguished by a high significance value.

• Н₀₃: The difference between the average levels of knowledge and use of morphosyntactic information by students positioned in different experimental conditions (using/not using interactive digitally-based technology) will not be distinguished by a high significance value.

Sampling method and research ethics

Non-random sampling was used for this study. It was formed by 62 statistical units – students from two courses, trained in the bachelor’s program “Special Education”, a full-time form of training. The students, having an equal start in studying BGSL, were divided into two groups. One group included the students (N=30), who were trained in visual language by applying the digital program (treatment group), while the other group of students (N=32) were mastering sign language competence through a traditional approach (control group).

The preliminary procedure included the preparation of forms certifying the possibility of consent or refusal by the students to participate in the pedagogical experiment.

For the purposes of the study, the researcher conducted a 3-month lecture course on BGSL, with both groups of students being prepared in parallel, but in different pedagogical conditions. The students from the treatment group were offered training in BGSL through a specialized software interactive language program Signmatics, with the capacity for encoding and decoding signing, oral and written messages; with opportunities for perception and expression at the level of vocabulary and grammar, revision and self-testing of knowledge by using auditory, visual, motor and tactile modalities. The characteristics of the program give a multimodal character to the pedagogical interaction.

Instrument and Procedures

The main instrument used in the study was a standardized test aimed to measure receptive and expressive skills in the Bulgarian Sign Language in 3 categories (phonology, vocabulary, morphosyntax), the elements of which constituted the content of the three subtests. The phonological subtest, through which students could present their skills, included 2 tasks: receptive (33 items) and expressive (28 items). The vocabulary subtest contained items through which students could demonstrate knowledge of the semantic component of the sign (56) and precision in its expression (54 items). Receptive and expressive skills in morphosyntax were also assessed in two tasks. For the first task, 18 items were selected (for expressing plurality, classifying verbs for movement and location, modifications of verbs for type, etc.), and for the second one – 15 items.

The assessment of receptive and expressive skills in BGSL was carried out by the researcher who has 13 years of experience in teaching the basics of Bulgarian Sign Language, and by a specialist – a native speaker of the national sign language. The latter belongs to the so-called CODA group (Children of Deaf Adults).

Data Analysis

The following statistical procedures were used: a Shapiro-Wilk test of normality to check the distribution of the data sets in the four linguistic categories; Cronbach‘s alpha procedure for examining the internal consistency of items in the three subscales; Descriptive statistics to classify the data and obtain summarized characteristics for the individual variables in the entire sample; An One-Way ANOVA test to assess statistical differences in the results of the 3 domains registered in the two groups. The adopted significance level is α=0.05. The empirical data were processed with the SPSS 16.0 statistical package.

Results

The internal consistency of the scale was calculated using the Cronbach’s Alpha procedure (Table 1). The item scores in the three subscales showed very good internal consistency (Cronbach’s Alpha=0.846).

Table 1. Internal consistency between the tests

Test	Number of items	Cronbach’s alpha values
Receptive phonology	33	0.803
Expressive phonology	28	0.813
Receptive vocabulary	56	0.807
Expressive vocabulary	54	0.824
Receptive morphosyntax	18	0.821
Expressive morphosyntax	15	0.851

The Shapiro-Wilk test on all domains assumes a value of p>0.05, which is a clear indicator of normality of the distribution.

Table 2 presents the means and standard deviations for the students‘ achievements in the three subtests of the two groups, and Table 3 draws attention to the significance effect of the digital program Signmatics in relation to different Sign language contexts.

Table 2. Means and Standard Deviations for the student’s achievements in the three subtests

Subtests	Group	N	Mean	Standard Deviation
	Control	32	14.62	5.28
Phonology	Treatment	30	14.64	5.29
	Control	32	15.43	4.57
Vocabulary	Treatment	30	17.70	5.42
	Control	32	15.41	4.75
Morphosyntax	Treatment	30	16.84	5.40
Table 3. Differences between students in different Sign language contexts
Sign Language contexts        Group           M			SD	F            Sig.
Phonological	Control	14.62	5.28	0.001        0.971
	Treatment	14.64	5.29
Lexical	Control	15.43	4.57	26.574      0.000
	Treatment	17.70	5.42
Morphosyntactic	Control	15.41	4.75	1.423        0.001
	Treatment	16.84	5.40

• Н₀₁: The first hypothesis raised was that the difference between the average levels of operation with the sign parameters by the students positioned in different experimental conditions would not be distinguished by a high significance value. The registered results did not reveal a statistically significant difference in the performance of the two groups, which means that the application of the interactive digital program Signmatics did not have a significant main effect on the phonological component (F=0.001, p= 0.971). In comparative terms, the treatment group achieved results (M=14.64, SD=5.29), which were similar to those of the control group (M=14.62, SD=5.28). This was an argument for accepting the first null hypothesis.

• Н₀₂: The second hypothesis contained a statement about the absence of a statistically significant difference between the average levels of recognition and reproduction of lexical sign units by students in different conditions of pedagogical interaction. In contrast to the previous linguistic category, statistical procedures identified a significant effect of digital technology on academic achievement in the lexical subtest (F=26.574, p<0.000). The comparative analysis of the arithmetic mean values in the two groups showed a convincing superiority of the treatment group (M=17.70, SD=5.42) over the control group (M= 15.43, SD=4.57) in relation to the studied domain. A prerequisite arose for rejecting the second hypothesis.

• Н₀₃: According to this hypothesis, the difference between the average levels of knowledge and use of morphosyntactic information by students trained in different pedagogical conditions would not be distinguished by a high significance value. By analogy with the lexical domain, the interactive program also demonstrated a significant effect at the morphosyntactic level (F=1.423, p<0.001). The observed difference between the average values in the groups showed a statistically significant performance of students from the treatment group (М=16.84, SD=5.40) compared to the control group (М=15.41, SD= 4.75), which is why this hypothesis was also rejected.

Discussions

The conducted experimental study was motivated by the need to supplement and enrich the scientific and applied research niche in a field that offers digital solutions for multimodal learning in a language that is unique in itself with its multimodal nature. The research focus was directed at identifying the effect of the application of an interactive digitally based system Signmatics in teaching Bulgarian Sign Language (BGSL) to students who study it as a second language (L2). The testing of language skills in their receptive and expressive aspects was carried out in three domains: phonology, vocabulary and morphosyntax. Depending on the pedagogical conditions of the language training in BGSL, the respondents were differentiated into two groups: a treatment group, in which the mastery of linguistic content was ensured by the Signmatics software program, and a control group, in which the students studied the basics of BGSL using generally accepted methods. Three hypotheses were raised in support of the formulated research question.

The first hypothesis assumed the absence of a statistically significant difference in the phonological skills of students situated in different experimental conditions. The students from the treatment group demonstrated higher achievements compared to the students forming the control group in the subtests for perception and expression of the spatial-kinetic characteristics of the sign but the difference was reported as statistically insignificant. The registered result was not surprising given the complex structure of the sign, the parameters of which are distinguished by simultaneous realization in the signing space. The performance of the first language task allowed the most accessible and at the same time the most challenging parameters of the visual lexical sign for the students‘ perception to come to the fore. The results formed a differentiated picture, outlining a general trend of precision in the perception of the participants from both groups in the order of more than 70% on average for each parameter. The highest degree of accuracy was registered in the area of localization and orientation of the palm. The errors made were the least frequent ones: about 40% for both groups regarding palm orientation and below 40% for the localization parameter. These reactions clearly show that localization, followed immediately by palm orientation, are the first visual-spatial segments that are learned by hearing adults ( Schmidt, 1993 ). In the remaining three parameters, and especially in linguistic movement and non-manual markers, the quantitative indicator of accuracy in perception dropped to 53%. Non-manual markers turned out to be a weak link in the students‘ linguistic behavior, which was logical, taking into account earlier studies of the mentioned characteristic ( Beal, 2020 ). An interesting trend was found within the limits of the errors generated. When the sign presented for perception was produced close to the face (as a location), the students showed a tendency to perceive differences in non-manual signals. In contrast, the distant position during the execution of the sign led to the omission of changes, which had a persistent effect among the participants. The most logical explanation for this linguistic situation can be found in the visual attention that students fix primarily on determining the shape of the hands and the movement, while at the same time the face of the communicative partner remains largely outside the field of their observation. In the second task, which was related to the assessment of the expressive aspect, the movement and configuration of the hand also turned out to be problematic areas of implementation. The highest error distribution indicator is in the movement parameter (28%), closely followed by the shape of the hand (25%), non-manual markers (23%) and the orientation of the palm (20%). The lowest error rate is registered in the in the parameter localization (14%). The errors found are very similar to those registered in the first task, with a small change in the non-manual markers, which in terms of accuracy of execution overtake the configuration of the hand.

The analysis of the data showed that the phonological component was the only one in which students receiving the Signmatics educational intervention did not achieve statistical improvement compared to their colleagues from the control group. There is a deep reason for the similar results obtained in the two phonological tasks. Most likely, the interference of spoken language manifests its effect, especially when it comes to non-manual signals. Expressive facial features accompany verbal languages in the course of communication, but do not convey linguistic data in the way that these elements perform a linguistic function in visual languages. Therefore, students mastering visual language as a second sign system make enormous efforts to decompose the broad category of “facial expression” into specific segments of linguistic information.

The findings in this domain are a reference to a conclusion related to the use of the interactive computer technology together with traditional forms of sign language instruction. Placing special emphasis by the teacher on the specific features of the five parameters of the visual-spatial sign, explanations of the way in which the sign is motivated, as well as the development of tests to verify phonological knowledge, are among the effective strategies for forming phonological competence in students mastering BGSL as E2.

The second hypothesis assumed that the difference in the perception and production of sign lexical items by students placed in different experimental situations would not have statistical significance. Vocabulary is a fundamental aspect in the acquisition of sign language. Mastering a foreign language, which is realized in a new modality for native speakers of the spoken language (M2E2), requires a major lexical volume. In this domain, a significant contrast was registered in the academic performance of students from the two groups, in favor of the treatment one, which created a condition for rejecting the null hypothesis.

Within the first task, designed to identify receptive skills, out of the 56 lexical units set, representing different “parts of speech”: signs nouns, signs verbs, signs modifiers, a high percentage of correctly recognized signs (95.5%) was noted among students drawing lexical knowledge from the interactive program, and 80.7% among those whose lectures were held in an environment not equipped with digital means. Both results are visibly high, but the statistically significant preponderance belongs to the treatment group. To some extent, the high values can be explained by the lack of minimal pairs of sign units in the constructed lexical task, which strategy was used to assess the phonological component. The general tendency for maximally accurate perception can be due to the different human models of representation of gesture signs that are included in the interactive program. During the training, the students had the opportunity to observe and imitate all the specialists who have a different manner of presenting lexical signs, which increased their interest in enriching their lexical cdmpetence.

A statistically significant advantage in the lexical capacity of the students participating in experimental training in BGSL was registered when performing the second task, the content of which was aimed at establishing their expressive skills. In addition to the quantitative aspect (87.3% ^ 73.7%), the reproduction of signs was distinguished by accuracy, which was more typical for the representatives of the treatment group. The strong expression of this qualitative characteristic is due to the uniform amplitude and the constant pace of “articulation” of signs by the performers of sign production recorded on the digital device. In the traditional setting, the teacher, in his or her effort to convey knowledge about the reproduction of the signs and subject to the specifics of “live” performance, can change the amplitude and pace of expression, and this probably led to exaggerated articulation in students, on the one hand, or to a lack of accuracy in performance, on the other.

The results of the specific empirical study are indicative of the role of digital technology in building lexical erudition, as well as the degree of mastery of a lexical database. Its advantages are reflected in the students‘ performance in both language tests and strengthen the effect on their assessment of correct production. The ability to choose a communicative option allows them to fully immerse themselves in the language situation, and the presence of feedback is their corrective for the language actions they have taken.

The third hypothesis postulated the absence of a statistically significant difference between the academic achievements in morphosyntax of the students from the two groups. The mastery of the morphosyntactic rules of the BGSL was assessed in two tasks. The performance of the participants in the treatment group was clearly influenced by operating with the software program, as evidenced by the higher results recorded in the first language task for the perception of morphosyntactic structures (71.57% ^ 58.69%). The interactive software motivates students to perceive linguistic material that functions through rules completely different from oral language. The multimodal component of the program provides students with an alternative format for mastering morphosyntactic concepts. In the second task for measuring expressive skills, the students from the treatment group again had greater success in arranging lexical units in sign syntactic chains. The quantitative expression of correctly reproduced morphosyntactic constructions is about 72.69%. The accuracy rate decreased sharply in their colleagues (59.79%). Omission of signs was observed almost three times more often in the control group. There were also cases in which participants stopped after reading the sentence and showed all nonverbal signals of confusion before proceeding to produce the wrong variant.

It should be noted that this result was lower than the result in the lexical probe, which means that the performance of one perceptual task does not predict the specificity in the implementation of another. At the same time, achievements in the impressive component within one linguistic category can serve as a basis for predicting the status of the expressive component.

Conclusions

The conducted experimental study was focused on identifying the effect of the use of digital technology on the mastery of linguistic content in BGSL by students studying it as a second language. Attention was concentrated on assessing their academic achievements in three linguistic categories: phonology, vocabulary knowledge, morphosyntax by administering a language test decomposed into three subtests for diagnostic verification of receptive and expressive skills.

The results of the implementation at the vocabulary level showed a remarkable change in the group of students placed in a digital educational environment. They dominated in terms of accuracy, demonstrating their capabilities in both modal modes – receptive and expressive.

The findings revealed that the digital application positively affects the speed of learning and consolidation of morphosyntactic concepts. The statistically significant improvement belonged again to the students from the treatment group. Although the result in this domain was lower than the lexical achievements, it is a clear indicator that the interactive program Signmatics can be a promising, evidencebased, additional strategy in the student audience where the basics of BGSL are studied.

The mastery of the phonological component was similarly represented in both groups of students, which is a reference to other factors determining success in this area. One of them is the new modality within which sign language is implemented. The different linguistic situation requires the acquisition of a new motor skill and its application for the creation of an entirely new phonological component with a set of other articulatory organs that do not overlap with those involved in the phonological structures of the spoken language. This inspires the idea of achieving synchronization between digital and traditional educational technologies, which find an intersection in multimodality.

Despite its solid traditional foundation, BGSL teaching has been modified over time through the integration of modern and creative elements, such as digital technologies. This combination of conventionality and creativity can lead to significant success in mastering a unique linguistic code that has a clearly defined role.

Acknowledgements

This study is financed by the European Union-NextGenerationEU, in the frames of the National Recovery and Resilience Plan of the Republic of Bulgaria, first pillar “Innovative Bulgaria”, through the Bulgarian Ministry of Education and Science (MES), Project No BG-RRP-2.004-0006-C02 “Development of research and innovation at Trakia University in service of health and sustainable well-being”, subproject “Digital technologies and artificial intelligence for multimodal learning – a transgressive educational perspective for pedagogical specialists” No Н001-2023.47/23.01.2024.

Conflict of interests

The author declares no conflict of interest.