A Brief Review on Different Driver's Drowsiness Detection Techniques

Published Online June 2020 in MECS DOI: 10.5815/ijigsp.2020.03.05

Nowadays road accidents have become a huge matter of concern. Every day a lot of people are dying because of road accidents. A matter of fact is that a huge percentage of accidents are caused because of inattentive driving. It is found that a reason for inattentive driving is driver fatigue and drowsiness. A lot of statistics reported about road accidents that happened because of driver fatigue and drowsiness.

Drowsiness is a term which can be defined as a feeling of being sleepy. Due to drowsiness, a driver can fall asleep while driving. The various techniques proposed in the literature for the detection of drowsiness. Driver face monitoring system [26] is one of them. Driver face monitoring system includes imaging and intelligence software part along with the involvement of different hardware part. Imaging and intelligence software, i.e. decision making can be implemented in computers or on smartphones. Computers can be desktop computers, can be a laptop or any processing unit such as ASIC, FPGA etc. In this paper, the driver face monitoring system implemented with the above-mentioned techniques are referred to as desktop-based approaches. On the other hand, methods implemented in the smartphones are referred to as smartphone-based approaches. In general, in this paper our review will be based on two categories: smartphone-based approaches and desktop-based approaches.

The main objective of this research paper is to review different driver drowsiness detection techniques in detail so that people can easily decide which detection techniques are better and also to help in making decision on drowsiness accurately as this review is based on the recent techniques.

We have discussed the desktop-based approach by dividing it into two parts: (1) hardware (2) software. Hardware part includes main board, human-machine interface, etc. On the other hand, software part carries much importance as it includes image processing techniques, symptom parameter extraction, and the most important part, decision making. The image processing technique is most important for drowsiness detection as it focuses on face detection, eye detection, mouth detection, feature extraction, and face tracking. For decision making, there are methods such as threshold-based method, knowledge-based method, methods based on probability theory, statistical method. Based on these methods, the decision regarding whether the driver is drowsy or not being taken.

We divide the rest of the paper as follows: Section “Terminology” gives primary intuition to understand the methodology. Section “Motivation, Challenges and Limitations” provides the cause of selecting the topic. Section “Smartphone-based” provides a detailed description of major contributions of others in the field of detecting drowsiness in real-time using smartphone applications. Section “Desktop-based” provides a detailed description of major contributions of others in the field of detecting drowsiness in real-time using desktop application and it is divided into some main approaches. Section “A Comparative Study Between Desktop-based and Smartphone-Based approaches” presents the major difference between the two approaches and describe which one performs better.

• II.    Terminology

• A.    Drowsiness

Drowsiness [25] is a term that can be stated as a process which occurs because of lack of sleep. Drowsiness causes a person to fall in sleep quietly or frequently.

• B.    Driver Face Monitoring System

Driver face monitoring system detects driver fatigue and drowsiness on the basis of face and facial components. Here the system first detects face and facial components and then extracts symptoms from them and then based on those symptoms it detects drowsiness and fatigue.

• C.    Face Detection

There are two methods for detection of face and they are Feature based method and Learning based method. In case of feature-based methods, face can be detected from an image on the basis of simple features, face rotation etc.

An approach in feature-based method is projection and it can be applied in binary or gray level images. Another approach is on the basis of skin color where skin color is determined using probability distribution in color space.

In case of learning-based methods, face detection is done with the use of number of training samples. Learning based method is better than feature based but it has computational complexity as it uses Haar-like features.

• D.    Eye Detection

Eye detection is needed for symptom extraction. There are 3 eye detection methods such as (1) Methods based on imaging in IR spectrum, (2) feature based and (3) other methods.

Methods based on imaging in IR spectrum are relatively quick and accurate for eye detection as physiological and optical properties of eye are used. Here two band pass filters one of which passes IR light of 850 nm and other one passes IR light of 950 nm are used.

In case of feature-based method, there are two methods one is based on binarization and other one is based on projection. Binarization has higher error rate.

There are some other methods to detect eye. One of which is based on face model where Sobel Edge Detection was used to separate eyebrow area.

• E.    Mouth Detection

Some driver hypo vigilance systems detect mouth based on red color features of lips but it can only work properly in suitable light conditions.

• F.    Feature Extraction

Extraction of face components like eyes, nose, mouth, iris, skin from an input human image is called feature extraction. Feature extraction is the prior step for face tracking, facial expression recognition or face recognition. It is important to detect facial components for feature extraction of face components.

• G.    Face Tracking

Face tracking is a process that locates the presence of a face in an image or video. Face tracking mainly works by comparing between old face feature and new face feature. It is used in many real-time technologies.

• H.    Decision Making

Based on extracted feature calculation and symptoms, it is decided whether the person is drowsy or not. If the above processes work successfully then gives more accurate decision making.

Fig. 1. Overview of Driver Drowsiness Detection

• III.    Motivation, Challenges And Limitations

Road accidents have now become a great concern in Bangladesh. Any non-driving activity that distracts the driver, causes driver not to pay full attention to their driving. Recent surveys show that at least 2,297 people were killed and 5,480 injured in road accidents in the last six months, a sharp rise in the death toll compared to the same period last year [50]. A certain amount of these accidents was caused by driver fatigue and drowsiness. So, our motivation here is to reduce the rate of road accidents caused by driver fatigue and drowsiness.

Challenges : Real-time implementation and low error rate is the challenging part here.

Limitations : The main limitation here is the accuracy. Most accurate detection of drowsiness and fatigue is the region of concern here.

Fig. 2. Road Accident Rate in Bangladesh [50]

• IV.    Smartphone-Based Approaches

Nowadays, in practice, the driver face monitoring system has been developed in the form of application for Android-based smartphone devices. Such kind of systems is developed as a driver safety monitoring system. Measuring safety-related data requires no extra monetary expenditure or equipment in the smartphone. Safety monitoring process involves the fusion of attributes gathered from different sensors including video, electrocardiography, photo plethysmography, temperature, and a three-axis accelerometer, that are assigned as input variables to an inference analysis framework. The most recent approaches for detecting driver drowsiness in smart-phone are summarized in Table 1. For summarization, we focused on the goal of the paper, hardware or software used there, detection techniques, goal, and accuracy.

In Table 1, we can see that most of the detection techniques have more than 90% accuracy and it varies from hardware to hardware. Most common drowsiness detection techniques are pattern recognition techniques such as neural network, support vector machine (SVM), support vector regression (SVR), Percentage of Eyelid Closure (PERCLOS), Eyelid closure degree (ECD), Haar-Cascade Classifier, Artificial Vision. In real-time systems, their accuracy is not good enough for perfectly detecting drowsiness. Hence with the efficient and better hardware platform, their accuracy might increase gradually.

Table 1. Smartphone Based Approaches

Ref.	Category	Used System	Accuracy	Detection Technique	Goal
[1]	Inattentive driving behaviors	iPhone, inertial sensors and GPS	Overall precision of 82% at 92% of recall	Computer vision and pattern recognition techniques	Alerting inattentive drivers
[2]	Drowsy driving detection system	Samsung Galaxy S3	90%	Neural network, PERCLOS	Sober-Drive prototype
[3]	Driver vigilance	EEG sensor		EEG system and SVR	Driver’s vigilance in real time
[4]	Driver behavior information	Samsung Galaxy S3, Korean	Male- 87.5% Female-70%	ECD, EEE, SVR	Driver drowsiness detection
[5]	Abnormal driving behavior monitoring	Sensors. Huawei Honor3C, ZTE U809, SAMSUNG Nexus3, SAMSUNG Nexus4 and HTC sprint	95.36%	SVM	Improving awareness of driving habits to prevent accidents
[6]	Driving maneuvers	Sense Fleet	90%	Fuzzy system	Detect risky driving events
[7]	Aggressive driving behavior	Android smartphone	83 %	SVM	Classify the safe drivers and risky drivers
[8]	Aggressive driving behavior	Android smartphone		MLA: Artificial Neural Networks, SVM, Random Forest (RF), Bayesian Network (BN)	Separate the safe drivers and risky drivers

[9]	Real-time driver behavior information	Android smartphone	93.37%	PERCLOS, Artificial vision,	Alert driver about the presence of drowsiness
[10]	Drowsiness Detection during Driving	Android smartphone with ALS, gyroscope sensor	93%	Haar-Cascade Classifier, LBPH face recognizer algorithm	Sleepiness detection during driving
[11]	Real-time abnormal driving behaviors identification	Android smartphone	95.36% with SVMl,96.88% with NN	Support Vector Machine (SVM) and Neuron Networks (NN)	Identify specific types of abnormal driving behaviors

• V.    Desktop-Based Approaches

We divided the desktop-based approach into two main parts, e.g. Hardware part and software part. Hardware part also subdivided into two parts e.g. i) hardware used for processing and ii) hardware used for imaging techniques. Further, we will go in details of the imaging techniques. In figure [3], you may find the flow diagram of our discussion about desktop-based approaches.

Fig. 3. Overview of Desktop-Based Approach

• A.    Hardware Part

Hardware platform may include a main board, one or more processors and human-machine interface (HMI). It should be an embedded system and inexpensive in realtime driver drowsiness detection systems.

Table 2. Processing Techniques

Ref.	Hardware	Advantages	Limitations
[12]	FPGA (Field Programmable Gate Array)	Faster than conventional microprocessors and flexible in programming.	Slower than ASIC
[13]	Raspberry Pi	Low cost and power consumption	Not compatible with X86 operating systems
[14]	OMAP (Open multimedia applications platform )	Has onboard face detection module that can be used for camera focus control	Lacks the high-resolution with high resolution images

Table 3. Imaging Techniques

Ref.	Camera	Advantages	Limitations
[15]	VGA	Video conferencing and still used in applicable handheld gadgets	Low resolution
[16]	IP CCTV (not the analogue CCTV)	High resolution, can cover a much wider area than an analog CCTV camera	Infirm security system
[17]	Webcam	Produce video in multi-megapixel resolutions, and few can run at high frame rates such as the PlayStation Eye	Continuously connected to web for an indefinite time.
[13]	PI Camera	Able to make use of the graphics processing capability of the Broadcom CPU.	Limited memory

In Table 2, we have listed the hardware used for processing techniques, i.e. to process the images and to detect whether the driver is drowsy or not. There you also find the advantages and disadvantages of each hardware device. From the above hardware, Raspberry Pi is the most popular as it is of low cost, small in size and consumes less power compared to other devices.

In Table 3, we have listed the camera used for imaging techniques, i.e. to capture the images of the driver during driving. There you also find the advantages and disadvantages of each device.

• B.    Software Part

Software part is the most significant part of the drowsiness detection system and can be divided into three main parts: i) Image processing techniques, ii) Symptom parameter extraction and iii) Decision making.

• B.1.    Image Processing Techniques:

In general, image processing forms of image import, analysis, manipulation, and image output. The main goals of image processing techniques include processing, detection, and tracking of faces, eyes, and mouth, and feature extraction of facial components. Image processing techniques are the main part of driver drowsiness detection system. We should focus on these techniques to build an accurate real-time system.

We divided the Image processing techniques used for driver drowsiness detection into five different parts: i) face detection, ii) eye detection, iii) mouth/yawn detection, iv) feature extraction and v) face tracking.

• B.1.1.    Face Detection

In Table 4, we have listed the face detection techniques along with the advantages and limitations. The learningbased technique is more robust than feature-based technique. But both of them usually fail in night light and some real-time stages.

Table 4. Face Detection Techniques

Ref.	Techniques	Advantage	Limitation
[18]	Learning-based (using Viola-Jones method)	Good Robustness	Ineffective to detect titled faces and sensitive to lighting conditions
[19]	Feature-based (in HSV color space)	Average Robustness and The chromaticity is decoupled from the intensity	Non-removable singularities
[20, 21]	Feature-based (in RGB color space)	Removes the brightness information from the RGB signal	Performance varies with skin color and low robustness.
[22]	Feature-based (in YCbCr color space)	Luminance independent. Make color space attractive for skin color segmentation.	Very low robustness and doesn’t give proper information of skin.

• B.1.2.    Eye Detection

Eyes and eye region are the most vital part for drowsiness detection. Most of the drowsiness detection system detects drowsiness by comparing the condition of eyes and eyelid movement. Mainly eye detection depends on head position. The most popular eye detection techniques are investigated and listed in Table 5.

Table 5. Eye Detection Techniques

Ref.	Techniques	Advantages	Limitations
[23]	Support Vector Machine (SVM)	Increase the overall robustness of the system and uses the kernel trick.	The head position does not deviate a lot when fully awake. Need long training time on large data sets.
[24]	Haar Classifiers	Execution speed and detection accuracy are high.	Complexity is definitely increasing, less robustness to different lighting conditions
[25]	Vision-Based Intelligent Algorithm (Convolutional Neural Networks)	Feasible to train easier to integrate and time complexity is O(n)	Need to provide both quantitative and qualitative result, difficult to determine the window size
[2.6]	Fuzzy expert system	Enhances decision quality, solve realtime problems efficiently	Difficult to build and maintain, require a large amount of time to train.

B.1.3. Mouth/Yawn detection

We listed the mouth detection techniques in Table 5, along with advantages and limitations. LDA, Haar-like Features, Fuzzy C-Means Clustering are the most used techniques to detect mouth or yawn. Among them, some systems detect mouth based on color features of lips but they can only work properly in suitable light conditions and color image.

Table 6. Mouth/Yawn Detection Techniques

Ref.	Techniques	Advantages	Limitations
[27]	Latent Dirichlet Allocation (LDA)	Probabilistic model and gives categories for free in any dataset, very high accuracy	Topics are soft-clusters, much information needed.

[28]	Haar-like Features	High execution speed and works even if the driver turns his face	Incredible complexity, accuracy depends on different lighting conditions.
[29]	Improved Fuzzy C-Means clustering technique	Works robustly at night time because of the IR illuminator being used	Performance decreases during daytime especially in bright days, fails to detect when the head is rotated

• B.1.4.    Feature Extraction

Facial features are extracted for tracking faces and computing level of drowsiness. Based on extracted features, the decision is taken whether it is drowsy or not. Most common and used techniques for feature extraction are Haar Classifier, Speeded-up Robust Feature (SURF), Maximally Stable Extremal Region (MSER), Min-Eigen. A summarization of the feature extraction techniques is listed in Table 7.

Table 7. Feature Extraction Techniques

Ref.	Techniques	Advantages	Limitations
[30]	Haar Classifier	High calculation speed, invariant to rotation	Costly, improper detection in case of scaling and illumination
[31]	SURF (Speeded up robust feature)	Robust, invariant to scale changes	Not good at handling view of point change, sensitive to illumination
[32]	MSER (Maximally stable extremal feature)	Reliable and stable, invariant to affine transformation	Slow
[33, 34]	Min-eigen	Invariant to rotation	Doesn’t support scaling and illumination changes

• B.1.5.    Face Tracking

Most of the time face is tracked by tracking facial feature points. In real-time drowsiness detection face tracking is an important part. CAMShift, Kanade-Lucas-Tomasi (KLT), Kalman-Filter techniques are mostly used to track face.

Table 8. Face Tracking Techniques

Ref.	Techniques	Advantages	Limitations
[35, 36]	CAMShift	Uses incessantly adaptive probability distributions, robust and non-parametric	Fails in tracking objects in complex situations like surveillance applications
[37, 38, 39]	KLT(Kanade-Lucas-Tomasi)	Faster than other and cost efficient	Can’t track face properly in some cases like head tilting continuously
[40, 41]	Kalman-filter	Decreases the dimensionality error, robust and estimates the dynamic changes of a state.	In case of nonlinear function need to use extended Kalman-filter

• B.2.    Symptom Parameter Extraction

To detect drowsiness, the extraction of the face and other components such as eye, mouth, etc. are very important. After feature extraction, another important rule is to extract the symptom from the extracted features. Detection of drowsiness is the biggest challenge in this context. From each feature, the syndrome is detected based on different parameters. We have listed the symptom parameter in Table 9.

Table 9. Symptom Parameter Extraction

Ref.	Region from Face	Parameter	Decision to take
[43, 44, 45, 46]	Eye	PERCLOS	If blink rate is much higher than normal state then its drowsiness or if less than then its distraction
[47, 48]	Mouth	The ratio of width to height of the mouth	Is low when the mouth is closed and the ratio is high when the mouth is open
[44, 45, 46, 49]	Head	Head nodding	Head is gradually bent at the time of drowsiness and head nodding resulted from dozing

• B.3.    Decision Making

Various algorithms have been proposed in the literature. Following, we will describe different methods for making the decision.

• B.3.1.    Threshold-Based Method

To detect the driver drowsiness, a threshold is being applied on PERCLOS. Here at first driver’s face is being identified and after that appropriate threshold is chosen. In general, the threshold is chosen based on the psychological characteristics of the driver.

• B.3.2.    Knowledge-Based Approach

In this approach, decision making highly depends on the knowledge of the expert. The most common knowledge in the methods is of if-then rules and fuzzy expert system. In fuzzy rules, an expert’s knowledge is used within the value range of 0 to 1.

• B.3.3.    Methods Based on Probability Theory

This kind of methods based on different probability theories, such as Bayesian Network and Dempster Shafer theory. Bayesian Network is used to detect driver drowsiness as it can predict future states on the basis of past and present information. As the computational capacity of Bayesian Network is large, Naive Bayes is also used. In [42], a nave DBN (Deep Belief Networks) is used to recognize facial expressions due to its high processing speed. Dempster-Shafer theory combines the probability theory and fuzzy theory. By combining the two theories, it works better than the Bayesian Network. The main problem of Dempster-Shafer theory is the high computational complexity.

• B.3.4.    Statistical Method

Neural Network (NN) is the most common method in this category. A combinational NN which consists of two sections determines the drowsiness level in this system, i.e. i) unsupervised section and ii) supervised section. In the unsupervised section, the input layer and the hidden layer are included. Output layer which performs classification process is included in the supervised section.

• VI.    A Comperative Study Between Desktop-Based And Smartphone-Based Approaches

With the growing technology of Android-based smartphone, an application developed in the smartphone is increased day by day. The consequence of that a large number of applications implemented on the smartphone. Driver face monitoring system is one of them. Now the question is, does it perform better than desktop-based approach. In this section, we try to find a meaningful answer to that.

Both desktop - based and smartphone-based have very good detection accuracy but smartphone -based has more drawbacks compared to desktop-based. As in case of smartphone-based, computational complexity becomes a much bigger concern, detection accuracy varies from smartphone to smartphone as camera quality and hardware capacity aren't the same. All of these are the reasons which deduce that desktop- based approach is better than smartphone-based approach as desktop-based approach overcomes these problems.

In Table 10, we have listed the comparison between the desktop-based and smartphone-based based on five different features.

Table 10. Comparison Between Desktop-based and Smartphone-based Approaches

Feature	Desktop-based	Smartphone-based
H/ w (Imaging Technique)	P i camera [13]	Android/IOS based Smartphones      front camera [1, 2]
H/ w (Processing Technique)	Raspberry pi [13]	Smartphones sensors, Smartphones accelerometers [1, 2, 5,]
Detection Technique	Learning-based (Neural network) [13, 18]	Knowledge-based (fuzzy expert system) [6]
Accuracy	More than 90% [51]	Approximately 90% [1, 2, 6, 9]
Advantages	Gives more robustness and accuracy rate in case of real-time [51]	Low cost and availability is easier. Monitor driver behavior using SenseFleet. [2 6, 9]
Disadvantages	Costlier          than smartphone-based approach. Accuracy varies due to  face  formations orientations    glasses and illuminations. [51]	Accuracy varies from smartphone         to smartphone   due   to hardware configurations (camera, gpu, ram etc. ) and sensors. Computational complexity arises and slow down the system in real-time      detection. [2,9]

• VII.    Conclusion

In this paper, we have discussed various techniques of driver face monitoring system, which is used to determine drowsiness. Each of the popular methods is discussed based on the advantages and limitations. Most of them are robust and the accuracy rate is high enough and depends on the hardware system or embedded system. Design and development of the drowsiness detection system are not only reasonable but also very necessary things in our everyday life. It is necessary to develop commercial systems with enough accuracy for real-time detection.

We have also discussed about different techniques related to face, eye and mouth detection, feature extraction etc. In case of face detection, viola jones method (learning based) is preferred as it is much popular. For eye detection, haar-classifier technique is suggested as it has better execution speed and accuracy. Haar classifier technique also gives better performance in case of mouth detection, so it might be preferred. Using kalman-filter might results in better execution of tracking of face as it decreases dimensionality error. Of all the mentioned techniques for feature extraction, SURF and MSER both techniques give more advantage over others.

We have mentioned the whole driver face monitoring system into desktop and smartphone-based approaches. In real-time, desktop-based approach gives the more robustness and accuracy rate of drowsiness detection. On the other hand, the smartphone-based approach also gives much accuracy rate and robustness but it varies from smartphone to smartphone. Nowadays smartphone-based approach is used a lot because of availability and low cost of android mobiles. But in the real-time, the detection rate is not as accurate as of the desktop-based approach. The most challenging problems of driver drowsiness detection systems are related to detection in night light. It can be improved by scientist and then the detection systems will be easily performed. Then the accident rate will reduce gradually day by day.

Acknowledgment

We are incredibly thankful to our supervisor Dr. Nusrat Sharmin who helped and motivated a lot in whole process. We are also thankful to the reviewers who recommended us some good comments to rewrite paper in proper way .