Methods and Tools for Identifying Human Resource Lesions in Emergency Based on Multimodal Analysis and Deep Learning

Emergencies of natural, technological, and military origin require rapid and accurate assessment of victims' conditions to support effective rescue and medical response. Traditional visual examination methods are often limited by stress, time pressure, and incomplete information, leading to delayed or inaccurate decisions. This study proposes a multimodal deep learning approach for automated identification of human resource lesions in emergency scenarios. The developed framework integrates visual, audio, and text/sensory data using convolutional neural networks, Transformer-based models, and a Transformer Cross-Attention fusion mechanism. The proposed architecture enables effective extraction and integration of heterogeneous features for lesion classification, severity estimation, and automated medical triage. Experimental evaluation was conducted on multimodal datasets containing injury images, audio recordings, and symptom descriptions. The model was trained using a combined loss function and evaluated with classification, regression, and triage metrics. The results demonstrate high system performance, achieving a macro-F1 score of 0.87, validation accuracy of 86–87%, and triage accuracy above 90%, including 95% for the RED category. The regression model for severity prediction achieved an R² value of 0.92, while modality importance analysis confirmed the dominant contribution of visual information. The experiments also showed stable model convergence and strong generalisation ability without significant overfitting. The proposed multimodal framework confirms the effectiveness of deep learning and cross-attention mechanisms for automated lesion identification and emergency medical triage. The developed approach can be applied in decision-support systems for rescue operations, emergency medicine, and intelligent VR/AR training simulators.