Driver Fatigue Estimation Using Image Processing Technique

Published Online June 2016 in MECS

Boyraz.P.,Leicester,Hansen J.H.L [3], proposed a method of sensing vehicle response to measure uncertainty in steering wheel. The drawback of this approach is, it is limited to vehicle type and driver condition. Mabbottt et.al[2], proposed a method were driver response is monitored requesting the driver to send feedback continuously. The drawback of this approach is driver gets tiresome and feels annoying.

Fig.1. Types of Driver Fatigue Detection System

All the above discussed approaches are intrusive system of driver fatigue detection system. Few limitations of Intrusive methods are system is complex, cannot be placed easily, causes disturbance, poor performance, non-reliable, and produces noise. The solution is Non-Intrusive system.

The basic approach in non-intrusive system is analysis of face. The first symptom of fatigue appears in eye and than in mouth. Lot of research has been done to analyze the facial features to estimate driver fatigue based on eye blink rate and yawning.

Vijayalaxmi,et.al.,2012 proposed algorithm using Neural Network approach for eye detection is divided into two stages- training of neural network and detection. The Neural network is trained using GTAV and VITS database images. The success rate of the algorithm depends on training, more the number of images used for training more is the success rate.

The neural network is trained using two sets of imagesone set containing the object to be detected and other set consists of non-object images such as background, nose, eyebrows etc,. The success rate is 98% tested on GTAV database [23]. The algorithm is tested on images of different face orientation and works successfully on straight images but fails to detect both eyes if the face is oriented 900. It detects at least one eye if the face is oriented by 300,450. The reason for failure of the algorithm on few images is improper ROI which leads to wrong identification of eyebrows as eyes.

In 1998, researchers proposed non-intrusive driver fatigue system based on PERCLOS. Richard Grace, et al., used CCD camera, PC/104 processor, the method worked well for eye closure but problem with warning system [25].

Neeta Parmar [4], proposed driver drowsiness detection system based on eye blink rate, the success rate is 80%.

The drawbacks of the system are poor illumination, unable to track eyes of person wearing spectacles, fails if one or both eyes are closed.

Martin Gallagher [5], proposed driver alert system for road safety, the problem with the system is it takes 8secs to process each frame which is not desirable in real time implementation. Nidhi Sharma, et.al,[6], proposed driver drowsiness detection system using fuzzy logic. The drawback of the system is, it works only on still gray scale images and fails to track eyes of person wearing glasses.

In 2010, researchers made use of PERCLOS to detect driver fatigue, which detects eyes open, semi open, close. But time taken was more and required high grade PC to process the frames. In 2011, an author used Haar features to detect driver fatigue based on eye blink, the method works on rotated images with a success rate of 96% but fails to warn the driver and testing is done on computer.

The challenges for developing a driver fatigue detection systems are:

• •    System should be non-intrusive

• •    Selection of proper hardware

• •    Developing proper algorithm to detect and track

face and eyes

• •    Perfect warning system

• •    Selecting proper programming language which can

be easily interfaced to Embedded System

• •    Time taken to process the frame and alert the driver should be less

In this paper, a new approach is mentioned to estimate driver fatigue based on PERCLOS (Proportion/Percentage of time in a minute the eye is 80% closed). The rest of the paper is organized as Section 2 explains proposed algorithm for Fatigue estimation, Section 3 shows tested results under different illumination condition. Conclusions are drawn in Section 4.

• II.    Related Work- High Performance and Low Power Dfd

Driver fatigue is detected based on face analysis and calculating PERCLOS based on eyes. If the PERCLOS value exceeds 80% than driver is alerted to stop the vehicle. The proposed algorithm for fatigue detection using image processing is shown in Fig 2.

• A.    Image Capture

Logitech USB camera is used to capture the images at the rate of 30fps of size 480x640. The images are captured under different illumination condition for training Haar Cascade Classifiers and also for testing. The images are taken for different face orientation, distance and face occluded with glasses is shown in Fig 3.

• B.    Haar Cascade Classifier

To meet the challenges listed above, four different algorithms are tested in the laboratory for efficient detecting and tracking face and eyes. The methods developed are Template Matching, Skin Segmentation, Neural Networks and Haar Cascade Classifier. After comparing all the four methods, it is observed that Haar Cascade Classifier achieves the highest success rate of 99.2%.

Firstly Haar Cascade Classifier is trained for detecting face and eyes. To train two set of images are required i.e Positive images(images which contain object to be detected i.e face and eyes), Negative images (images which does not contain objects like scenery,pin,etc.).

In this research, total 4000 negative images and 3000 positive images from GTAV,Mathworks Video, Face expression and VITS database are used. As Viola and Jones suggested that Haar Cascade Classifier produces upto 95% detection rate if it is trained properly. In order to improve the efficiency of Haar Classifier, the positive images used for training are selected based on skin segmentation i.e, since positive images consists of objects to be detected, hence based on skin segmentation only face area is used for training by removing all other parts.

After training Haar Cascade Classifier, .xml file is generated which is loaded to detect and track face and eyes from captured images. Fig 4 shows, the results of success detection of Face and Eyes for different images.

Fig.2. HPLP Driver Fatigue Detection System

Straight image                Head Rotated face image

Image with Spectacles

Closed eye image

Fig.3. Images captured under different Illumination

C. Eye Open/Close

After successful detection and tracking of Face and Eyes, next step is to find whether eyes are closed or open. The steps followed are

The original image size is 480x640, the size of the face is 222x224 and the sizes of cropped eyes are 147x64.

Face cropped image

Eyes cropped

Fig.5. Eyes cropped

Image enhancement is the pre-processing step in image processing. There are so many methods for image enhancement, the approach used is Homomorphic filtering. The reason behind this is, most of driver fatigue detection system failed due to illumination effect. The image is a combination of illumination and reflectance, illumination is due to low frequency components and reflectance is due to high frequency component. Homomorphic filtering normalizes brightness across an image and increases contrast.

Grayscale image

Fig.6. Enhanced Eye image

Enhanced eye image

It is a process of converting grayscale image to binary image. The binary image consists of only two gray levels, depending upon the threshold value

g(x,y)=0, if f(x,y) > T =1, if f(x,y) < T where ‘0’ represents Black and ‘1’ represents White. In this the object i.e eye balls are assigned white pixel value and background is assigned with black pixel. Adaptive Thresholding is used in this algorithm, as it handles illumination problem more prominently compared to Global and Local Thresholding.

Enhanced Eye image

Binary Eye image

Fig.7. Binarized Eye image

Binary image with noise

Fig.8. Binarized Eye image without noise

Binary image with noise removed

Contours are useful tool for shape analysis and object detection and recognition. For better results, the contours are applied on binary images where background should be black and white pixel represents object. Contour pixels are small subset of the total number of pixels representing a pattern. Amount of computation is greatly reduced. In this research Contours are used to find the white pixels and mark the contours.

Open Eye image       Binary image         Contours

Closed Eye image

Binary image         Contours

Fig.9. Open/Closed Eye images with contours

After marking contours, the number of white pixels representing contours is counted. If the value exceeds some fixed value then eyes are open otherwise eyes are closed. Since open eyes shows more white pixels compared to closed eyes.

Pixels      of Eye

Binarized Eye

Image

Contours

Fig.10. Eye Open/Close images

Open

Close

PERCLOS is an established parameter to detect the level of drowsiness. The PERCLOS (the percentage of time that an eye is closed in a given period) score is measured to decide whether the driver is at drowsy state or not. On an average human blinks once every 5 seconds (12

blinks per Minute).

Frame rate: 30fps

In 60 Seconds: Total 1800 frames

Time Taken to read each frame: 0.0333Secs

Number of frames with closed eyes: 15

Total time showing closed eyes: 0.4995 Secs PERCLOS= (0.4995*100)/60

= 0.8325*100=83.25%

• III.    Results

The complete set up is made in the laboratory and proposed algorithm is tested on four (GTAV, Face Expression, MathWorks Video and VITS) database images and live testing is also performed on 45 different persons under different illumination conditions. The setup is shown in figure below.

The results of the proposed algorithm are shown in figure, under different illumination conditions different head positions and for persons with and without glasses. The images used for testing the proposed algorithm are tabulated in Table 1.

Table 1. Images used for Testing the System

S.N.	Database	Total No. of Images
1	Face Expression	148
2	GTAV	46
3	MathWorks Video	126
4	VITS	200
5	Live	45(15 Women, 15 Men, 15 persons wearing glasses

The overall success rate of the proposed system is calculated using equation (1) given below

0 veralI Detection (%)

TP+TN

TP+TN+FP+FN

Where

TP= True Positive: No. of frames that are correctly detected eye blinks.

FN= False Negative: No. of frames that show eye blinks but the program is not detected.

FP= False Positive: No. of frames that are reported as eye blinks but they are not.

TN=True Negative: No. of frames that are correctly reported as no blinks.

The system is tested on Video captured through Logitech USB camera and the details are shown in Table 2.

Table 2. Captured Video details for Testing

Frame Rate	30fps
Frame Size	480x640
Duration of the Video	65Sec
Total Frames	1950

TP=6, FP=2, FN=2, TN=1890

Overall Detection = 97.7%

The proposed system is also Live Tested using Logitech USB camera and the details are shown in Table 3.

Table 3. Live Video details for Testing

Frame Rate	30fps
Frame Size	480x640
Duration of the Video	85Sec
Total Frames	2550

TP=27,FP=4,FN=2,TN=2517

Overall Detection = 99.7%

The simulation results of proposed Driver Fatigue Detection System tested in the laboratory are shown in Figure 11.

Simulation result of Closed Eye

Fig.11. Simulation Results

Simulation result of Open Eye

• IV.    Conclusion

In this paper a new system is proposed to estimate driver fatigue using image processing technique. Driver fatigue is estimated based on eye blink rate and is calculated using PERCLOS method. If the eyes are closed for 15 frames or if PERCLOS is >80% than the driver is entering into drowsiness and driver is alerted to stop the vehicle. The face and eye are detected using Haar Cascade Classifier. Eye Open/Close is estimated using contours and counting the number of white pixels.

The entire set up is made in the laboratory and tested on GTAV, Mathworks Video, Face expression and VITS Database images. The system is tested on a video and the overall detection rate is 97.7%. The system is also tested Live and the overall detection rate is 99.7%. The developed algorithm works for different illumination, head rotation and persons wearing glasses. The time taken by the algorithm to estimate driver fatigue is <5ms.