Autism Spectrum Disorder Screening on Home Videos Using Deep Learning

According to the National Institute of Mental Health (NIMH)[1], the prevalence of ASD in the United States is estimated to be 1 in 54 children. Diagnosis of ASD is complex, and no particular medical or genetic test is available for its diagnosis. Instead, it is diagnosed based on a comprehensive evaluation of a person's behaviour, communication, and developmental history. The diagnostic criteria for ASD are outlined in the Diagnostic and Statistical Manual of Mental Disorders, 5th edition (DSM-5) [2], published by the American Psychiatric Association (APA). Other commonly used tools for the assessment of ASD include the ADOS [3] and the Childhood Autism Rating Scale (CARS)[4]. Early diagnosis and intervention are crucial for improving outcomes in individuals with ASD, making accurate diagnosis critical.

The significant contributions of this research are

• •    A dataset has been created from publicly available YouTube videos and labelled manually based on the video's description.

• •    The videos in the dataset are then converted into skeletal ones, which reduces the computational complexity as well as protects the privacy of the children in the videos

• •    A CNN-LSTM model is proposed in which the CNN extracts the spatial features from each frame while the LSTM captures the temporal features from consecutive frames.

• •    The proposed model achieved a testing accuracy of 84.95% with 89% precision, recall and F1-score.

2.    Literature Review

In recent years, various ML approaches on different modalities like eye tracking, facial images, medical signals like Electroencephalogram (EEG), video, etc, are being explored for ASD diagnosis. Mahmoudi et al. (2019)[5] developed an ML algorithm that analyzed EEG data to classify children with ASD and typically developing children. The results showed that the algorithm could accurately distinguish between the two groups, demonstrating the potential of EEG data for improving ASD diagnosis.

Another promising ML approach is the use of eye-tracking data, which is a reliable indicator of ASD. Zhao et al. [6] investigated eye-tracking data from conversations in person. For the dataset, 19 ASD and 20 TD children were interviewed with an eye tracker set up on their heads. These children showed no hyperactivity disorder or schizophrenia. The conversations obtained were divided into four sessions with some questions/conservation types to examine the behavioral gaze, nodding, initiating conversation, etc. The aim was to observe how children talk comfortably about things that interest them. It was observed that ASD individuals had a more extended interview duration than TD participants. Various ML classifiers were implemented on visual fixation features and interview duration as input. This approach must still be confirmed on a larger sample under varying disorder severity and balanced sex ratio. Guillon et al. (2014) [7] also used eye-tracking data to develop an ML algorithm that accurately classified children with ASD and typically developing children.

Xie et al.[8] has used a Two-stream End-to-End Deep Learning Network to study the visual attention of ASD and TD subjects. To create the dataset, 20 ASD and 19 TD Subjects were gathered based on IQ, race, age and gender. They were given images randomly from the OSIE database [9] for 3 sec each, and in this duration, a Human Fixation Map(HFM) representing their attention pattern on various pixels based on time was generated. In their model, VGGNet, which takes Natural Image and HFM as input, is used for object recognition separately. After which, the output for the two was concatenated and given to ASDNet for autism prediction. This study lacks clarity on the integration process of information from different parts of the image. Liaqat et al. [10] collected data on children freely viewing natural images (scan-path) for identifying autism using Machine Learning and Deep Learning. Scan paths consist of the fixation points’ 2-D coordinates (where the subject's eyes have focused) and their duration generated by STAR-FC (Wloka et al. [11]), a generative model of synthetic saccade patterns. This Image-based approach gave an accuracy of 55.31% and 61.62% on the test and validation datasets, respectively.

Abbas et al. [12] collected data from parent questionnaires and home videos for building the model. The video of the subject is recorded for a short duration in a natural environment. This video is then reviewed by an expert and filled with questionnaires based on the video. The output of the two models was combined to give the screening results. Both datasets are recorded in a highly clinical environment using AdI-R and ADOS protocols; the testing data is in a non-clinical environment. Also, the model gives an “Inconclusive” result in case it finds the input data challenging to avoid any wrong result like false-positive or false negative. Tariq et al. [13] used video raters who rated the video dataset for 30 different behavioural features like eye contact, passing of smile, etc., using 30 different questions. These raters were students or working professionals who didn't have any expertise in the autism domain. Then, they trained eight different ML models on these ratings. The top-performing classifier was logistic regression. With only five features, it showed an accuracy of 88.9%.

Hosseini et al. [14] have used facial feature extraction and image classification to detect ASD. Here, MobileNet was used, and it was observed that the classification results improved in terms of speed. Still, false positives, nonavailability of metadata about images like their medical history, etc. and verification of images remain a hindrance. Khosla et al. [15] aimed to identify autistic children using facial expressions. This study considered a dataset from Kaggle containing 1468 images of each autistic and non-autistic group. The duplicate image was removed using the MD5 Hash Algorithm (Wang et al. [16]). Near-exact images are generated using the pHash Algorithm. The limitation of this study is the generalization Gap in validation and test accuracy. Lu and Perkowski [17] investigated ASD detection in minority groups by combining the images of minority groups with the Kaggle dataset.

Liao et al. [18] employed a small subset of ABIDE [19] data, which was collected by two universities, the California Institute of Technology and the University of Michigan, solely to accomplish the categorization by using the community structure of the brain. Herein, the study has calculated the correlation coefficients between two ROIs (Regions of interest) using Pearson correlation, and the community structure was discovered using the QCut detection method. Liu et al. [20] aim to address the problem of poor reproducibility and generalizability in neuroimaging investigations for possible biomarkers. They used ABIDE data to conduct an attentional connection analysis. They initially built separate brain networks for every person, then used the CC200 atlas to extract connection properties between ROIs.

Kojovic et al. [21] investigated the usability of nonverbal social interaction in ASD diagnosis. The dataset consists of videos of a child interacting with an adult, where the child was the test subject who could be ASD or TD. A distinct preprocessing used in this study was extracting skeletal key points with a black background for the human figures in the video. This reduced the dimensionality of the dataset. After this, CNN-LSTM architecture was deployed to recognise action, and VGG-16 was used for feature extraction. This study was able to reach a maximum accuracy of 80.9%. The proposed model is inspired by this study and aimed at the behavioural analysis of the subject for autism diagnosis.

3.    Materials and Methods 3.1.    CNN-LSTM

Convolutional Neural Network (CNN) [22] is a well-known machine learning model that uses convolution for pixel-related analysis. It works to break down the complex image data into a simpler form that still has all the essential data but is easier to process. Meanwhile, Long Short-Term Memory (LSTM) networks are a type of Recurrent Neural Network (RNN) that are suitable for sequence analysis problems [23]. RNNs are known for their ability to keep past information while processing the next step. This helps learn the current context, but when there is a time lag of 5-10 time steps, RNN fails to learn the context, resulting in the vanishing gradient problem [24]. LSTM was designed to address these issues [25] using an LSTM unit consisting of three activation functions known as gates [26], as they control the entire flow of information inside an LSTM unit. The Forget gate is responsible for choosing only the relevant information from the previous step, while the Input gate does the processing of current information and learns it. Lastly, the Output gate has the task of passing the updated information to the next timestamp. A useful and important variation of LSTM is the Bi-LSTM, consisting of two LSTMs (forward and backwards), increasing the amount of information available to the model. This leads to improvement in the context understanding of the model [27].

Combining these two architectures has helped to achieve a hybrid model that can be used for sequence prediction problems with spatial inputs, such as images or videos. The CNN in CNN-LSTM structure works upon spatial feature extraction of the input data, while LSTM helps to recognise sequential patterns. Thus, the CNN-LSTM combination creates a spatially and temporally deep model. These models have much potential, as much data is available and processed in visual formats.

• 3.2.    Dataset

The dataset used in this study is self-curated. 128 Videos were collected from YouTube, 66 of which belong to the ASD class and 63 belong to Typically Developing (TD). The videos were collected considering the factors the ML model could use to make predictions. The selected videos capture the social interaction of the subject with a guardian when assigned a task. Since the data was sorted out from public platforms, the surroundings and assigned activity differ from subject to subject. The videos are labelled as ASD and TD based on the description of the videos. The skeletal information was extracted from the clips using OpenPose to eliminate the variation in the surroundings. Figure 1 shows a snap from a video in the dataset and the corresponding skeletal structure created using OpenPose.

(a)                                                                     (b)

Fig. 1. (a) Snap from the video of a medical expert interacting with the subject through some games[28] (b) Snap from the video converted to skeletal structure using 2D OpenPose.

The videos consist of daily vlogs by parents/guardians performing some tasks with the subject or a medical expert interacting with children using some games for autism diagnosis or regular checks. Various tasks or activities include reading a picture book, painting, playing with a ball, soap bubbles, toys or games like building blocks, puzzles, etc. The goal of the activity is to engage the subject, study their attention towards tasks and objects around and social interaction with the guardian. Thus, the collected data captures the children's interest in objects and guardians. The dataset was preprocessed to remove unnecessary portions. Each video is further split into 5-second durations, and thus, 469 videos are created and labelled manually. The dataset is further divided into training and testing data in the ratio of 80:20. The dataset statistics are given in Table 1.

Table 1. Dataset Statistics

S No.	Criteria	ASD	TD	Total
1	No. of videos collected from YouTube	65	63	128
2	No. of videos after 5-second split	316	153	469
3	No of videos in the train dataset	254	121	375
4	No of videos in the test dataset	61	32	93

• 3.3.    Proposed Methodology

• 3.4    Proposed CNN-LSTM Model

4.    Implementation

In the proposed methodology, the videos containing ASD and TD children performing some activity with an adult are collected from YouTube. The relevant parts of each video are cropped out, and the skeletal information of the persons in the video is extracted using OpenPose. Since the deep learning model needs a large amount of data, each video is further split into 5-second video segments. The preprocessed videos were converted into frames using OpenCV [29] for feeding the data to the proposed CNN-LSTM model. The proposed methodology is outlined in Figure 2.

Fig. 2. Proposed methodology

Due to a lack of benchmark video datasets for ASD diagnosis, the required videos were collected from YouTube, which was then passed through 2-D OpenPose technology to extract information on the skeletal structure of each individual. The continuous stream of skeletal key points was extracted from the video, capturing the gestures and interests of the individual in the task. The standard protocol for diagnosing autism [3, 30] mentions that the autistic individual tends to show less interest in the surroundings and lack attention in the activities with a guardian or medical expert. Hence, feeding the acquired data to the CNN and LSTM captures the kinesics of the subject to classify autistic and non-autistic individuals. The pretrained CNN models like VGG16 [31], InceptionV3[32], Xception[33], MobileNet[34] are selected as the CNN. Preprocessed frames are fed to the pre-trained CNN model with “ImageNet” weights for spatial feature extraction, and hence, the fully connected layers at the top are not included. The output tensor is further reduced by adding one 2D convolution layer with 64 3x3 filters and then flattened to get the final feature vector. These feature vectors corresponding to each frame in the 5-second video are passed to LSTM to capture the sequential patterns in the video. Since the data is sequential and LSTM can learn long-term dependencies, it is suitable for video analysis. The proposed CNN-LSTM model is shown in Figure 3.

Fig. 3. Proposed CNN-LSTM model

The proposed model uses several computer vision and deep learning libraries in Google Colab. The OpenCV library reads the input video file and extracts frames from it. Keras, a high-level deep learning library, is used to load the pre-trained CNN models and define the LSTM and BiLSTM model architectures. NumPy is used for array manipulation and storage of feature vectors. The h5py library efficiently stores feature vectors in .h5 file format. The TensorFlow library is used as the backend for Keras.

The proposed model uses pre-trained CNNs for feature extraction from the input video frames. The pre-trained CNN models selected are VGG16, InceptionV3, Xception and MobileNet with “ImageNet” weights, excluding the top fully connected layers. These pre-trained CNN models are loaded using the Keras library. The dimensions of output tensors of the CNN models are further reduced by adding a 2D convolution layer of 64 filters of size 3 X 3. The dimensions of the output tensor and the feature vector are shown in Table 2.

Table 2. Feature Vector Dimension

Sl.No	Pre-Trained CNN Model	Output Tensor Dimension	Tensor Dimension after reduction using 2D Conv.	Feature Vector Dimension
1.	VGG16	7 X 7 X 1024	5 X 5 X 64	1600
2.	MobileNet	7 X 7 X 1024	5 X 5 X 64	1600
3.	Xception	10 X 10 X 2048	8 X 8 X 64	4096
4.	InceptionV3	8 X 8 X 2048	6 X 6 X 64	2304

The resulting feature vectors are saved as NumPy arrays in two .h5 files - training and testing data. The training and testing dataset sizes are given in Table 1. The .h5 file format is a hierarchical data format that supports efficient storage of large multidimensional datasets. To store the training and testing data in .h5 files, the h5py library is used.

The feature vectors corresponding to each frame of a 5-second video are grouped and fed into the LSTM or BiLSTM layer with 512 units for learning the temporal patterns in the input video frames. The LSTM layer is followed by two fully connected layers (1024 neurons and 50 neurons) with ReLu and sigmoid activation, respectively. A dropout Layer with a 0.5 dropout rate is added between the two dense layers. Lastly, the output classification layer with two neurons and Softmax activation is added for classification. The model is compiled using mean_squared_error loss and Adam optimizer with a learning rate 0.001. The model is trained for 100 epochs with a batch size of 50. The model is evaluated on various combinations of pre-trained CNN models and LSTM or BiLSTM. All the parameters are kept the same for both LSTM and BiLSTM layers.

5.    Results and Discussion

The proposed model is evaluated by experimenting with different combinations of pre-trained CNN models and LSTM or BiLSTM. The various metrics used for the performance evaluation of the models are Accuracy, precision, recall and F1 Score. The performance of the models on these metrics is shown in Table 3.

Table 3. Model Performance

Sl.No	Model	Accuracy (%)	Precision (%)	Recall (%)	F1 Score (%)
1.	VGG16 + LSTM	83.87	84	84	84
2.	MobileNet + LSTM	83.87	84	84	84
3.	InceptionV3 + LSTM	68.82	69	69	69
4.	Xception + LSTM	81.72	82	82	82
5.	VGG16 + BiLSTM	83.87	84	84	84
6.	MobileNet + BiLSTM	84.95	85	85	85
7.	InceptionV3 + BiLSTM	67.74	68	68	68
8.	Xception + BiLSTM	81.72	82	82	82

The best accuracy is obtained with MobileNet and the BiLSTM model. VGG16 and Xception have given the same accuracy with LSTM and BiLSTM. Almost similar results have been obtained on LSTM and BiLSTM throughout the models, as shown in Table 3. InceptionV3 has demonstrated the least accuracy among all the models.

Since the final dataset is imbalanced due to the short video length of the TD class, to check whether the model is biased towards the majority class, a confusion matrix has been drawn and is shown in Figure 5. The confusion matrix demonstrates that the model is not biased towards the majority class. False Negatives obtained on Xception and InceptionV3 are more than the other two models. This means that these models will result in more diagnoses claiming that autistic subjects are ASD-negative and, hence, unsuitable for real-world deployment. MobileNet and VGG16 are giving remarkable performance with both LSTM and BiLSTM.

(a) VGG16+LSTM

(b) VGG16+BiLSTM

(c) Xception+LSTM

predicted label

(g) MobileNet +LSTM

(e) InceptionV3+LSTM

(d) Xception+BiLSTM

(h) MobileNet + BiLSTM

Fig. 4. Confusion Matrix

• 5.1    Comparison with state-of-the-art models

6.    Conclusion

There is not much research on video analysis for ASD detection. The only research found in the literature is the work done by Kojovic et al. [21], which is similar to this study. However, the video dataset for their model was prepared in a controlled environment for hours. The expert performed activities with the subject based on ADOS assessment. The task given to each participant was cohesively designed to capture the focus and social interaction of the subject. Moreover, the dataset is created from children from a particular demographic region. In the proposed study, the data is collected from public sources, which are not structured and contain videos created on children from different demographics worldwide. Thus, the dataset used in this proposed study is more challenging, and the model trained on this dataset is expected to be more robust. The proposed model’s performance is compared with the model proposed by Kojovic et al. [21] and is shown in Table 4.

Table 4. Comparison with state-of-the-art

Sl.N o	Reference	Dataset			Model	Accuracy (%)	Precision (%)	Recall (%)	F1 Score (%)
		Method of Dataset Creation	Total duration of all the videos	Number of 5-sec segments
1	Kojovic et al., [21]	Created in a Controlled environment	47.782 hours	34340	VGG16 + LSTM	80.9	78.4	85.4	81.8
2	Proposed Methodology	Created from YouTube videos	30 minutes	469	Mobile Net+BiL STM	84.95	85	85	85

This paper proposes a CNN-LSTM model that is efficient and robust for ASD diagnosis using simple Home videos. The dataset is created by collecting YouTube videos and contains noise and inconsistency. The acquired dataset aimed to capture the social interaction between the subject and the guardian. Since the model is trained on a small and noisy dataset, it is robust towards the subject, environment, and activity performed. The videos are first converted to skeletal key points to reduce the data complexity and protect the privacy of the subjects. VGG16, InceptionV3, Xception and MobileNet are used for obtaining features, and their output is then passed into the LSTM/BiLSTM. LSTM and BiLSTM have shown the same performance throughout the models. MobileNet with BiLSTM gave the best accuracy of 84.95%. Studying the confusion matrix shows that InceptionV3 and Xception are unsuitable for real-world application as the False negatives obtained on them are high and may lead to undesirable misdiagnosis. However, the results show the potential of video-based assessment in ASD prediction. A software tool based on the proposed model may help parents perform an initial ASD screening cost-effectively.

Conflict of Interest

The authors declare no conflict of interest.