Driver Drowsiness Detection System

Drowsiness during driving poses a major risk to everyone on the road, not just the drowsy driver. According to National Highway Traffic Safety Administration (NHTSA) research on fatal car accidents in Washington in 2019 which resulted in 700 fatalities, 2% of all such incidents were caused by drowsy driving [1]. In India, it is responsible for 21% of fatal road accidents as per Med India report [2]. Unfortunately, 60% of Indian drivers admit to falling asleep behind the wheel, with young drivers between the ages of 18 and 29 being most at risk. Drowsy driving has effects on other road users, including pedestrians and other passengers, in addition to the driver. This might have serious repercussions and puts the lives of innocent people in danger. Although the death toll has been highlighted, the actual effects of drowsy driving may go far beyond the current estimates. The importance of this research derives from the fact that driver drowsiness is a more common cause of accidents than other common causes like rough roads, equipment failures, or bad weather. The highway car crash statistics for the past 25 years in the United States are plotted in Fig.1 [3].

Fig.1. Total Number of Car Crashes Reported Annually by the United States National Highway Traffic Safety Administration (NHTSA)

Drowsiness is a common human condition, and even slight exhaustion from driving can cause drowsy behavior or brief periods of microsleep while driving. The phenomenon of muscle memory further contributes to this issue, as extended periods of driving may prompt the body to crave relaxation, resulting in a drowsy sensation. By harnessing existing car components' potential and collaborating with car manufacturers, we can pave the way for a more secure driving experience for all individuals.

Numerous investigations into various procedures, frameworks, and theories have been performed to identify driver’s drowsiness. To address this persistent and in-depth problem, this paper proposes a novel method by using the machine learning concept. The technology uses a rearview mirror camera and a steering wheel pressure sensor to detect mental and physical drowsiness. The Region of Interest (ROI) is used for improved precision. The system includes facial recognition and steering wheel pressure monitoring. The Perclos algorithm determines drowsiness, while grip pressure and steering wheel angle identify bodily drowsiness. This improves driver and passenger safety.

The strategy that has been proposed involves working directly with producers and manufacturers, emphasizing the significant potential in this area. The main research objective is to use already-existing parts, such as the rearview mirror and steering wheel, to create novel algorithms and techniques for identifying driver drowsiness. The focus is on tactics that, despite differences between automobile manufacturers, may be applied consistently and with little adjustment across these common components. The main challenge in implementing this work into action was to figure out how to accurately and efficiently identify drowsiness in real-time. This was overcome by employing a reinforcement learning model, which can recognize drowsiness from a stream of real-time photos taken of the driver.

Further, this paper is organized as follows- Section 2 highlights the literature work that has been done in this field. Section 3 enlightens the proposed framework of architecture used to implement this novel approach whereas its experimental setup as well as results is shown in Section 4. Eventually, Section 5 concludes the whole work in this paper.

2.    Literature Review

Many researchers in the literature have focused on detecting the state of drowsiness by using different techniques. A few insights from the literature have been discussed in this section. Focusing on the issue of driver safety on the road, Hashemi et. al [4] introduced a cutting-edge method for drowsy driver detection. Convolutional Neural Networks (CNN) are utilized in this system to detect the driver's drowsy state as it relates to the two objectives of real-time application, namely high accuracy and fastness. However, they have not shown its comprehensive validation in diverse real-life scenarios. Another study is provided by J. Singh [5] on a device that warns drivers who are drowsy while the person is driving. Their study made an effort to solve the problem by developing a module called “drowsy eyes detection”. This system lacks the feasibility of taking other factors like cost effectiveness. A driver drowsiness detection system using eye blink, mouth opening, and mouth closure counts was proposed by P.P. Patel et. Al [6]. An alert sound is produced to alert the driver when his eyes are closed for a lengthy period. Furthermore, if the driver is seen dozing off more frequently than a few times, the driver of the car is notified through email. The Convolution Neural Networks (CNN)

and emotion analysis were used by H.V. Chand et. al [7] to present a unique multi-level distributional model for detecting driver drowsiness.

Their suggested approach analyses the driver's emotional state to pinpoint the elements that motivate various driving behaviors. However, how emotional analysis has been done is not provided in the paper. R. Ayachi et. al [8] proposed anobject identification model-based driver drowsiness indication system. After the eye and mouth states were detected using an efficient Det model, the percentage of eye closure (PERCLOS) and frequency of mouth yawning (FOM) were utilized to determine the state of exhaustion. A behavioral method was suggested by M.E. Walizad et. al [9] in their study to develop a driver drowsiness detection system that would alert drivers before an accident happens. This research employs a trained convolutional neural network to determine if the driver's eyes are open or closed. A CNN model for categorizing eye states was proposed by I. Jahan et. al [10], who tested it using the three CNN models (VGG16, VGG19, and 4D). P.K. Singh et. al [11] used a mobile front camera rather than a webcam to detect the driver's drowsiness. Different methods, including support vector machines (SVM), random forests (RF), convolutional neural networks (CNN), and artificial neural networks (ANN), have been used to analyze driver drowsiness. A model was put up by M. Rode et al. [12] that involves a webcam or camera being used to continuously watch the driver. Additionally, they extracted the driver's face from the camera's continuous image frames using OpenCV. The eyes are the primary region of interest (ROI) for this model since they are crucial for detecting drowsiness. Wearable electroencephalographic (EEG) based on CNN technology was proposed by M. Zhu et. al [13] as a way to detect driver drowsiness in vehicles. The three components of the presented system are wearable EEG data collecting, vehicle driver drowsiness detection, and the early warning mechanism. A method to spot drowsy driving was proposed by Y. Chaudhary et. al [14] utilizing facial and eye expressions. They have used machine learning and deep learning methods to predict a driver's emotional state, which will increase traffic safety. The authors efficiently identify the driver's state using OpenCV and the Haar-based cascade classifier to detect eye movement. Convolutional neural networks and attention (CNN-Attention) structures were used by T. Xu et. al [15] in their proposed unified framework, termed E-Key, to detect driving drowsiness while also performing personal identification (PI). On the NvidiaJetson Nano embedded device, E. Civik et. al [16] suggested a revolutionary method that makes use of deep learning to effectively identify driver fatigue in real time. The suggested system analyzes the driver's eye and mouth regions using the deep learning architecture, specifically CNNs, to categorize four different circumstances.

The literature shows that contrary to our proposed system's advantage, the majority of the referenced systems do not consider the initial driving experience to be a critical training time for drowsiness detection. Furthermore, none of the referenced articles specifically address the 2-way confirmation technique, which entails comparing the results of the facial recognition model with the data acquired by the steering wheel monitoring system.

3.    Proposed Framework of Architecture

Globally, the primary cause of accidents is driver drowsiness. Thus, to more precisely detect drowsiness, the work in this paper offers new methods and systems that may be built right into cars using cutting-edge learning techniques and technology. The goal is to provide a complete and consistent system for detecting driver drowsiness by taking into account variables including the driver’s heart rate, body temperature, grip on the steering wheel, and facial characteristics. This study focuses on recommending cutting-edge methods that can be integrated directly into systems by the majority of manufacturers. Two distinct systems are proposed for detecting different types of drowsiness: physical and mental.

A practical approach for dealing with physical drowsiness which affects a driver’s reaction to stimuli, involves placing a pressure sensor, such as a dynamometer, on the steering wheel behind the grip. This sensor can pick up the pressure that the driver applies during braking and turning. In these circumstances, a drop in pressure below a threshold is a sign of driver inattention. A camera installed inside a rearview mirror can be used to identify signs of mental drowsiness like eye contractions and yawning. The drowsy faces of specific drivers are then correctly identified using reinforcement learning algorithms once the photographs have been recorded. This method achieves more accurate drowsiness detection because it integrates the video feed with pressure sensor data.

Results from both physical and mental drowsiness detection techniques are taken into account by a comprehensive drowsiness detection system. To ensure greater accuracy in detecting drowsy driving, the system will only sound an alarm when drowsiness is detected by both systems. There is a concept of ROI (Region of Interest) in the literature. ROI [17] is an area specifically determined by algorithms in a stream of images, concentrating on the area where this proposed algorithm operates behind the screen. This ROI includes the driver's eyes and mouth in the context of detecting drowsiness. The main purpose of the ROI is to remove unnecessary elements from the image input, such as moles and background noise, leaving only the essential facial features. The eyes and the mouth can be clearly seen through the ROI stream, which allows them to input this information into a pre-trained Machine Learning (ML) model that uses features like widening the eyes and yawning to determine if a person is drowsy or not. Fig.2 shows the region of interest.

Fig.2. Region of Interest

The drowsiness value is continuously increased as the ML classifier locates drowsiness. The alert is initiated when this value goes above a predetermined threshold. After locating the ROI and starting the picture stream feed, though, there are some difficulties. Based on physical characteristics like eye closure and mouth opening, the machine learning (ML) algorithm used in this model is pre-trained to detect drowsiness. The architecture of the proposed system is mentioned in Fig.3. This architecture includes two parts of the detection system. The first part is through facial recognition and the other one is through pressure being applied on the steering wheel.

• 3.1    Description of the Detection System

• 3.2    Modular Division

The facial recognition model uses reinforcement learning as its foundation. The assumption is that the driver won't feel drowsy during the first five minutes of the car journey. Then the model receives training during the first five minutes, which prepares it to recognize the driver's drowsiness later on. The output of the model is next compared to the information gathered by the steering wheel monitoring system. The alarm will sound to warn the driver of their drowsy condition if both results match up and exceed a certain threshold.

The 2-way confirmation method offers convincing proof of the driver's condition of drowsiness. The pressure system and the continuously learning reinforcement model integrate well, making them simple, to include in everyday vehicles without covering the driver's vision or affecting the performance in non-drowsy conditions. With this combination, drowsiness detection is more accurate, and only alarms are set off when there is strong evidence of drowsiness. This is beneficial to the driver and guarantees the security of other passengers. The process is predicated on the idea that the driver won't become drowsy for about five minutes after starting the car. Then the model receives training during this vital period, enabling it to pick up on and adjust to the unique facial characteristics of each person. It can take into consideration the diversity of persons and their facial features, resulting in a unique and accurate drowsiness detection system for every car journey. To match the unique facial characteristics of the person seated, the model is trained from scratch with each car start rather than depending on previously taught models.

The Detection device has a pressure monitoring mechanism built into the steering wheel grip that continuously measures grip pressure. Additionally, an angle detector measures the turn of the steering wheel to analyze how the driver is controlling the vehicle at critical times, such as while making a turn on the highway. The steering wheel handling plays a crucial impact on the driver's ability to control the vehicle, which is directly related to how they perform when driving. Thus, in addition to recognizing facial attributes, monitoring grip changes can provide vital proof of driving drowsiness. The steering wheel model is shown in Fig.6.

After talking about these two drowsiness detection system components, it is evident that when used together, they offer additional support for confirming the driver's drowsiness. As was previously stressed, warning the driver without certain evidence of drowsiness puts the driver, fellow passengers, and other road users at danger. Thus, four possible combinations have been generated by taking into account the output of both systems: False-False, True-False, False-True and True True. The sound alerts when both modules yield an accurate result, showing that the driver is drowsy to assure accurate outcomes. This detection system's accuracy is greatly improved because both modules monitor the driver's state in real-time.

Fig.3. Proposed Architecture

Fig.4. Eye and Mouth Detection Model

Fig.5. Perclos Algorithm

Fig. 6. Steering Wheel Model

Therefore, if the driver yawns by stretching their mouth for a specific amount of time, closes their eyes for longer than the threshold, and doesn't have a firm grip on the steering wheel, especially during turns, the combination of these factors should be considered to be strong evidence to warn the driver of their actions.

4.    Experimental Setup and Results Analysis

Few system configurations are required to check the feasibility of the proposed Drowsiness detection system. To run this program, there are three necessities i.e. operating system, an integrated development environment and a language to develop the system. Thus, the proposed detection system is checked by using Windows v7/v8/v10 x64/x32, Python as a high-level language, and JetBrainsPyCharm community edition (latest version) as IDE. Hardware requirements include Processor: 64-bit, quad-core, 2.5 GHz minimum per core, RAM: 4 GB or more, HDD: 20 GB of available space or more, Display: Dual XGA (1024 x 768) or higher resolution monitors and Camera: A detachable webcam.

• 4.1    Testing

Every module is tested to find faults and problems, whether they are semantic or syntax-based. It involves running over the entire program to look for any errors in the logic, the code or any segmentation faults that might occur in edge circumstances. Different testing methods are used for different kinds of error. Multiple tests are necessary because one test may miss some bugs. Unit testing, integration testing, functional testing and system testing are a few of the several forms of testing.

• a)    Unit testing : It entails creating numerous test cases for the code that cover a variety of edge circumstances. After that, the program uses these test cases to further verify the logic. If the result of the code for the test input matches the predicted result, it can be considered that the logic of the code is sound; if not, then there will be need to update the algorithms. It is a structural testing process that depends on understanding the construction of its input and output expectations.

• b)    Integration Testing: Testing every module that has been combined and is being executed as a single program is known as integration testing. Every module must undergo integration testing after unit testing to ensure that the output of one module is consistent with the input required by another module. Every module is compatible with the others after integration testing is complete, but test cases still need to be created to verify the program's working logic.

• c)    Functional Testing: It is a type of testing that verifies the functionality of the application whether it is working as per the requirement specification.

• d)    System testing: It is carried out to ensure that every aspect of the system is operating as intended and that the inputs are producing the desired results. It examines the combined operation and logic of every module.

• 4.2    Test Report

Following these tests, a test report is produced and subsequent procedures for resolving faults and mistakes are carried out.

The program, system, and modules are all tested individually and collectively. A variety of tests yield a variety of outcomes, some right and others wrong. Below is a compilation of those few tests for face recognition. Table 1 shows the test report of unit testing whereas test reports of functional testing and integration testing are tabulated in Tables 2 and 3, respectively.

Table 1. Test Report of Unit Testing

Name of the test	Face detection
Sample input	Stream of image of the face of a person
Expected output	Eyes and mouth detection
Output	Eyes and mouth detected
Remarks	Successful

Table 2. Test Report of Functional Testing

Name of the test	Drowsiness detection
Sample input	Stream of image of face of a person
Expected output	State of drowsiness of the person
Output	Drowsiness state of the person
Remarks	Successful

Table 3. Test Report of Integration Testing

Name of the test	System alert
Sample input	Stream of image of the face of a person
Expected output	Alarm when drowsiness is detected
Output	Alarm played on drowsiness detection
Remarks	Successful

• 4.3    Results

2.0*C

ROI can be seen in the Fig.7 around the face in which the eyes are open, thus, no red border around the image (the red border around the image indicates drowsiness detected). But in Fig.7, the score is greater than 2 (which is the threshold value of time in seconds), thus, the border of the image is red. It means that the drowsiness has been detected. As previously noted, the percentage of eye closure or the PERCLOS algorithm is used to identify drowsiness. It measures the closure of the eyes by drawing six points around the eye's diameter by using the formula given in equation (1) specified below:

(A+B)

ear =---

Where, ear is ear aspect ratio. A, B, and C are the distances between two opposing vertical points and two opposing horizontal points, respectively, that are drawn on the eye. Fig.7 depicts the model detection when the eyes are opened whereas Fig.8 shows the model when eyes are closed.

Fig.7. Model Detecting When Eyes are Opened

Fig.8 Model Detecting When Eyes are Closed

The outcomes from Fig.7 and Fig.8 wraps up the findings of the Perclos algorithm-based driver drowsiness detection system. The pressure system being applied to the steering wheel, which also senses the angular velocity of the steering wheel, is another outcome shown with the help of figures. Fig.9 pictures the angular velocity and steering wheel grip comparison during the non-drowsiness state whereas the drowsiness state is shown in Fig. 10.

The two individuals depicted in the added graphs are turning at the same time. The angular velocity of the steering wheel exhibits about the same values. However, in Fig.9, it can be seen that the driver's grasp tightens as the angular velocity of the steering wheel increases to better control the vehicle. In Fig. 10, the grip on the steering wheel is practically constant no matter how sharply the wheel is twisted, indicating a lack of focus and grip throughout a turn. Thus, it shows what happens when a person is sleep-deprived and consequently drowsy. This module, which is based on the pressure system in the steering wheel, produces a correct response if, despite several spikes in angular velocity, the pressure on the wheel remains low. The alarm won't sound until both of these modules declare the result to be accurate and identify the driver as being sleepy. Even if the proposed module is producing effective outcomes, there are a few things that might alter the outcomes. First, the driver's position and angle, followed by the size of their eyes. Sunglasses may potentially have an impact in the same way.

Fig.9. Angular Velocity and Steering Wheel Grip Comparison Showing Non-Drowsiness

Fig.10. Angular Velocity and Steering Wheel Grip Comparison Showing Drowsiness

5.    Conclusion and Future Work

In comparison to earlier studies and research, the current study discovered a method for more accurately calculating a driver's level of drowsiness It investigated several techniques for determining whether a driver is drowsy before combining the findings. The first method involved identifying the driver's eyes and mouth from a stream of images continuously taken by the driver's camera. Drowsiness is determined when the mouth is opened for a longer period than the threshold and the eyes are closed for a longer period than the threshold. The second technique used a pressure-detecting device built into the steering wheel to track the driver's grip as they controlled the vehicle in a turn. It has an additional feature of an alarm that will sound only if both of these modules declare the result to be accurate. Since, this proposed framework of detection system provides a better solution, it still suffers from a few limitations including the cost of vehicle components and complete vehicles will raise. In that case, mass production could help restrict the cost increase to a minimum. Another limitation is that environmental factors like sunlight or dim nighttime light might alter the visibility of facial features captured by cameras. Furthermore, the steering wheel’s grip on the car may loosen during turns, possibly leading to the erroneous detection of drowsiness. The work in this paper can be further extended to a large scope by using cloud computing. Every car's position and the driver's current condition will be stored in a cloud-based system. The other drivers may be alerted to the drowsiness state inside the specified perimeter of the current location using the information that has been stored. It may cause other drivers to become concerned about the other motorists on the same road, which may help them avoid collisions with that specific vehicle and may also notify the local authorities to assist the accused driver. To protect their clients and ensure the security of their vehicles, service providers like Uber and Ola can also utilize it to monitor the health of their drivers when they are running their vehicles at night. Furthermore, it encourages further researchers to concentrate on and enhance these areas.

Acknowledgment

Amit Kumar, Bhupendra Kumar Pathak, Kaustubh Mishra, and Rajiv Kumar conceived, wrote and improved the paper and their contributions are proportional to the order their names are listed. Kaustubh Mishra conceived and designed the experiments; Bhupendra Kumar Pathak and Amit Kumar performed the experiments; Bhupendra Kumar Pathak and Rajiv Kumar analyzed the results. No Funding has been provided for the given research.