Hand gesture interface based on skin detection technique for automotive infotainment system

Published Online February 2018 in MECS

In recent years the automotive domain has developed significantly with the growth in technology and rising user demands. The users demand for embedding multimedia, internet and computing features has evolved into a totally new family of product called Infotainment System. The in-vehicle infotainment system (IVI) are popular in advanced automobiles and may have integrated features like automotive navigation systems, radio, media player, Carputers, in-car internet, Wi-Fi, USB and Bluetooth connectivity [1-2]. In fact there is a range of infotainment devices available in the market designed for various automotive models. Typically, the interface systems in these devices have a driver centric design features. It is necessary that the driver should not be distracted while dealing with the large number of functions of the infotainment device [3-6]. Thus, designing an interface system for these devices in automotive environment is a challenging task. Moreover, it is required to design an effective solution offering optimum features for driver and enhancing the accessibility features for the passengers [7].

Traditionally, these devices are operated through simple dashboards knobs and dials. Presently, the advanced versions of infotainment devices have controls on steering wheel, touch screen interface and remote control device for interaction [8-10]. Aiming for further expansion towards touch less interaction, a lot of research and development is reported for utilizing gesture interactions as a complete solution in automotive environment [11-12].

In automotive domain the research proposals have emphasized on design of gesture interface considering the driver as its prominent user [13-17]. The major issues encountered in this scenario are as follows:

• 1.    Extra load compelled on the driver to learn various gesture commands for interaction

• 2.    Menu based nested user interface system require larger gesture database, cause distractions and are time consuming

• 3.    Dealing with varying illumination

On the other hand the rear seat passenger uses remote control or communicates with driver to interact with infotainment system. This may cause distraction to the driver. Also, the remote control interface is not hand-free interactions and should be accessible. Ultimately, as an enhancing technology the computer vision based gesture interface is a novel possibility in this scenario and would improve the naturalness of user interface. The major advantage of such system is that they provide hand-free interactions.

A typical vision based hand gesture application consists of camera interfaced to a computing system having control features. Thus, developing an effective algorithm is essential part in the system design [18]. In such system the image acquisition is the first stage followed by pre-processing, segmentation, feature extraction and gesture classification stages. The flow of typical gesture interface system is shown in following Fig.1.

Image Acquisition

Hand Segmentation

Feature Extraction

Hand       Hand

Posture       motion

I

The pixel based technique is the simplest method. It involves skin detection by defining threshold values. The edge based segmentation involves boundary identification between image regions. In this technique there is no need for the region boundaries to be closed. The advantage of this technique is that it deals better with gaps. The major disadvantage is that it provides comparatively inefficient information. The region base segmentation method involves identifying objects having similar characteristics. In this technique it is required that boundaries should be closed. It is robust than edge segmentation method.

The feature extraction is the next stage and can be implemented through static or dynamic gesture recognition techniques. The static gesture recognition technique involves extracting information from stable hand postures. The popular static hand gestures comprise of finger count or hand gestures without moving the hand. In dynamic gesture technique the hand movements are classified as input commands [19]. Accordingly, various hand gestures are classified through algorithm. Further, each gesture is associated with a particular command to trigger an event.

In the present work, after a rigorous study it has been decided to develop a hybrid system consisting of touch based interface for driver and hand gesture based interface for rear seat passenger. The issues that are focused in the design are:

• 1.    Light variations

• 2.    Continuous changing background

• 3.    Interior design

• 4.    Hindrance caused during vehicle movement

In the proposed setup a traditional webcam is used which is mounted on the head rest area of the front seat. The arrangement made in automotive environment for the rear seat passenger is shown in the following Fig.2.

Event

Gesture

Classification

Fig.1. Typical Flow of Gesture Interface System

The image acquired from the camera is applied to the segmentation stage. It consists of skin classification algorithm for separating hand region from the image. The segmentation method can be classified into Pixel based, Edge based and Region based techniques. The classification of segmentation techniques is shown in the following table 1.

Table 1. Classification of segmentation Techniques

Pixel based	Edge Based	Region Based
Explicit Defined	Watershed Region	Region Splitting
Histogram Based	Canny Edge Detector	Region Merging
Gaussian model	Wavelets
Neural Network	Laplacian of Gaussian
Bayes Classifier

Recognition Area

Fig.2. Setup for Rear Seat Passenger

The advantages of the proposed setup are as follows:

• 1.    Fixed plane background

• 2.    Optimum distance of camera and object of recognition.

• 3.    Camera view is precisely set for recognizing hand area

• 4.    Camera equipped with light source

The image acquisition is done through webcam which is interfaced to a computer system. In many advanced automotives the computer is used for providing trendy features on dash board. These computer systems are often termed as Carputers. The use of computer enhances the connectivity and computing features for an infotainment product.

In the present work a “Hi-Vision” algorithm for segmenting hand region is developed and implemented in MATLB10a. The Hi-Vision algorithm is a hybrid technique designed using effective color combinations with decision rule. The performance of Hi-Vision classifier is tested over images having varied light orientation and color background. The identified hand region is then processed to extract features like center of hand and tip of hand region. The computed tip is utilized as pointer for classifying various gestures. A Graphic User Interface (GUI) system is developed to implement user interface for automotive environment. Thus, the user can interact with media player through GUI in multimode way with a provision of touch or gesture mode of interaction.

The results of this research show that it is possible to achieve reliable gesture interface in automotive environment. The main features of the developed system are fewer and effective gesture sets, user independence, and drivers control for providing access to user.

• II.    Related Work

In this section a brief review of the popular color models, classification techniques, evaluation techniques and combination of color space used for skin detection are discussed.

The skin detection domain finds a wide area of application including surveillance, medical field, home automation, robotics and automotive domain. However, the skin color recognition remains a challenging task over several decades. This is because many objects in the real world have skin tone color. The basic method for skin detection is to define decision boundary for a color space component. The efficiency of skin detection algorithm can improve or degrade the overall response of a system. Thus, the skin detection is a vital phase and it is the preliminary stage in gesture recognition system. The skin color technique has many advantages; it is simple powerful, fast, and has proved to be useful in face detection, hand gesture detection, localization and tracking systems.

The skin recognition methods are typically influenced by the following factors:

• 1.    Dynamic light

• 2.    Camera characteristics

• 3.    Background colors, surface orientation

• 4.  Skin tone variation across individual

• 5.  Blur images

The fundamental element in skin identification technique lies in the concept that skin surface reflects the light in a different way as compared to the other surface. This is because the human skin is created by the combination of blood and melanin which gives it a restricted range of hue. The skin color detection is done by classifying skin pixel and non skin pixel. It is seen that the accuracy of skin classification depends on the selection of color spaces as well as skin color model.

The desired properties for an efficient skin color classifier are as follows:

• 1.    Response should be real time

• 2.    Optimum distinction between skin and non skin classes

• 3.    Coverage of all different skin type ex. Blackish, yellowish, brownish, whitish.

• 4.    Good result under different light conditions

• 5.    No false skin detection

Researchers have proposed skin detection using various color models and fusion of color models [20-23]. Thus, the color space technique is widely used and preferred technique for detecting skin region.

• A. Color Models

The primary step for skin detection employing color information involves utilizing a suitable color model. A color model is a specification of a coordinate system and its subspace where each color is represented by a single point. The skin classifier is designed by defining threshold for recognizing skin region using color space. There are various color spaces for detecting the skin region and they are classified as (1) the basic color space: RGB, normalized RGB and CIE XYZ, (2) the perceptual color spaces: HSI, HSV, HSL, TSL, (3) the orthogonal color spaces: YCbCr, YIQ, YUV, YES and (4) the perceptual uniform color spaces: CIE lab and CIE LUV. Different skin detection method use different color spaces. Fundamentally, it is expected that a good skin color model should have quick classification response and simple computations to match the real time response. It is seen that the RGB, YCbCr, YIQ, YUV and HSV color space provide optimum skin classifier [24-27]. The prominent color spaces used for skin detection are discussed in the following section.

I Basic Color Spaces for Skin Region Identification

The RGB is a default color space. It is the most commonly used color space for displaying, printing, storing and representing digital image. The data captured by camera has RGB format. Also, the other color spaces can be obtained from linear or nonlinear transformation of RGB format. The advantage of RGB Color model is that it does not require transformation, thus it is simple and fast technique. However, the major disadvantage of this classifier is that the chrominance and luminance components are mixed. In RGB the luminance factor is a linear combination of the R, G, and B values. Thus, variation in luminance of skin region affects all the R, G, and B components. Many researchers have proposed RGB color space specifically and in combination with other color space for skin detection. The RGB color space is widely used with histogram and Gaussian mixture model techniques for skin detection. The normalized RGB is obtained from the RGB values by simple normalization procedure. The sum of the normalized component is unity r+g+b=1, by normalizing luminance is separated from chrominance. The Normalized RGB has better performance than RGB for variations in light conditions. It is also a candidate for skin region detection.

Marwa Jmal et al.[28] applied the pixel based skin color classification technique using RGB model for face recognition. The classification model showed 87% hit rate with high accuracy and speed. Abdellatif Hajraoui et al.[29] employed RGB and normalized RGB models for detecting facial characteristics. The combination of these color models provides satisfying results in terms of True Positive (TP) and False Positive (FP) rates. Chiunhsiun Lin et al. [30] have used the RGB Color space since it fast technique and does not require any transformation for skin detection. They have also proposed that the developed system can be employed efficiently for determining the mature condition of vegetables and fruits. Ghazali Osman et al. [31] proposed a skin color classifier using RGB ratio model. The model explicitly defines the skin region in an image. The performance of skin classifier was measured based on true positive (TF) and false positive (FP) indicator. The experimental results showed that the RGB ratio model outperformed all the other models in term of detection rate. The RGB ratio model is able to reduce FP detection that caused by reddish objects color as well as be able to detect darkened skin and skin covered by shadow. Chelsia Amy Doukim et al. [32] made a comparative study of three color spaces Normalized RGB, Modified Normalized RGB and YCbCr utilized for skin detection. It was found that the chrominance component form the modified normalized RGB color space provided highest detection rate.

II Perceptual Color Spaces for Skin Region Identification

The perceptual features of color such as hue, saturation and intensity can be defined using the perceptual color spaces HSI, HSV, HSL and TSL. The HSV color space is popular and widely used in image analysis and computer vision. The humans perceive HSV color space better than the RGB system. Thus, they are useful to capture typical human perception for color. The main advantage of this color space is that it separates the chromatic part. The performance of HSV color model is superior for ambient light, high intensity white light and different surface orientations. The disadvantages of this color space is that it gives discontinuity in numeric values of Hue around red, it is sensitive to RGB deviation and the fact that Hue is undefined for a saturation of 0. The HSV color space is widely used with histogram operations for skin detection. It provides an effective solution for the parametric skin color models. Thus, it can be good candidate for skin detection.

Vandana. S. Bhat et al. [33] have utilized the HSV color space for face detection with 82% detection rate. The H component of HSV color space was used for classifying skin and non skin pixels. Dhia A. Alzubaydi et al. [34] employed the HSV color space for face detection with real time performance. The hit ratio observed was 93100%. Iyad Aldasouqi et al. [35] have used the HSV skin classifier with median noise removal technique for detecting face in image. The images for test were taken randomly from internet. The images had different backgrounds. The recognition rate was 94.66%.

• III    The orthogonal color spaces for Skin Region Identification

The family includes YCbCr, YUV and YIQ. The YCbCr is a digital color system, while YUV and YIQ are analog spaces. YCbCr is a commonly used color space in digital video domain. It is used in image and video compression standards like JPEG, MPEG1, MPEG2 and MPEG4. The main advantage of this model is that it explicitly separates luminance and chrominance component. Research has shown that skin color is clustered in a small region of the chrominance plane. The basic disadvantage of this color space is that it depends on RGB primaries. YCbCr is the most popular color space used for skin detection. It provides simple skin detection rules and fast classification of pixels.

Zhang Qiu-yu et al. [36] have employed the YCbCr color space for hand gesture detection with skin recognition and K-mean clustering method efficiently. The classifier is used to separate skin region from the non skin. The hand segmentation is done by clustering values of chrominance using k-means clustering algorithm followed by morphological process. The method was tested for complex background and varying illumination dataset. Bahare Jalilian et al. [37] have applied YCbCr color space model with single Gaussian model, Bayes rule and morphological operation for detecting face and hand shape. The method has been tested over 700 posture images of sign language.

• IV    Perceptual uniform color spaces for skin region identification

The CIE La*b* color space is perceptually uniform color space that describes all the colors visible to the human eye. The main advantage of this color space it that it not device dependent. The disadvantage is the color model depends on RGB primaries. The CIE La*b* and CIE-LUV are among popular skin detection candidate. This color space is widely used for tone and color correction required in photo enhancement and color reproduction systems.

Amanpreet Kaur et al. [38] have compared the performance of YCbCr and CIELab color space. The analysis shows that the YCbCr has influence of device dependency and CIELab to be better candidate for skin detection. Manju, Deepika et al. [39] have utilized the CIELab color space for hand gesture detection. The system has provided real time response with MATLAB software. The comparative summary of prominent color spaces used for skin region identification is presented in the following table 2.

Table 2. Color spaces for skin classifier

Color space	Advantage	Disadvantages
RGB	•    Fast computation since transformation is not require •    Output is sharper	•    Sensitive to the illumination intensities •    Device dependent
NRGB	•    Works well in many light conditions •    The Skin color distribution build up in r-g plane	•    High computational cost •    Output is unclear image •    In efficient for real time application •    Device dependent
HSV	•    Separates the chromatic part •    Good with histogram operations •    H-S is effective in characterizing skin color •    It is effective for the parametric skin color model	•    Sensitive to value RGB value deviation •    Un-stable hue because of the angles •    Device dependent
YCbCr	•    Separated chrominance and luminance •    Skin color Connect well in CbCr plane	•    Depends on the RGB primaries •    Output not sharp •    Device dependent
CIELab	• Device Independent	• Depends on the RGB primaries

B. Classification Techniques

The popular approaches for hand detection are classified into explicit color space threshold technique, non-parametric skin and parametric skin modeling methods [40-45]. The explicit skin classifier is widely used where high computing speed and less memory is required. In this technique the skin classifier is composed by defining explicitly the decision boundary of skin color cluster for different color space.

The Non-parametric skin modeling methods estimate skin color distribution from the training data. In this method lookup table or histogram is used to represent distribution of skin tones in color space. These approaches are usually employed when a huge amount of training data can be obtained. The histogram techniques are robust to image variations such as limited geometric transformations and object class variability. The advantage of non parametric methods is its fast training process. The drawback of this technique is requirement of large storage space.

The parametric skin color modeling estimates skin color distribution from the training data by making inferences about the parameters of an explicit model of the distribution. The popular methods include Single Gaussian, Mixture of Gaussian and Elliptical Boundary methods. The software techniques involve mathematical algorithms for skin classification. These techniques provide good reliable performance with complex background. The drawback of this system is requirement of long time of learning. The database samples, variation in scale, translation and rotation led to a misclassification in the process [46]. A review of skin color classification techniques is shown in the following table 3.

Table 3. Review of skin color classification techniques

Skin color classification
Explicitly Define skin region	Non Parametric Method	Parametric Method	Software Computing
Basic color RGB, normalized RGB	Standard lookup table	Single Gaussian	Neural networks
Perceptual color spaces HSI, HSV, HSL, TSL	Bayes Classifier	Mixture of Gaussian	Fuzzy systems
Orthogonal color spaces YCbCr, YIQ, YUV, YES	Self organizing Map	Elliptical boundary Model	Generic algorithm for recognition
Perceptual uniform color spaces CIE lab and CIE LUV

The Non-parametric methods are not affected by color space representation while the explicit methods and parametric modeling techniques are affected by the choice of color space. In the explicit technique the color selecting good color space determines the performance of the system. The comparative summary of various classification techniques is presented in the table 4.

Table 4. Comparative summary of classification technique

Technique	Advantages	Disadvantages
Explicit	•    Simple approach and requires less memory •    Due to simplicity this method is widely used •    Training not required	•    Difficulty in selection of color space •    Selection of color space determines the performance of system
NonParametric	•    Better    suited    for constructing   classifier for large target data set. •    Fast training •    Not affected by choice of color space	• More storage is required
Parametric	•    Better    suited    for constructing   classifier for limited training •    Able   to   interpolate training data	• Slow in training

Thus selection of color space plays important role in skin segmentation technique. The parametric techniques like Gaussian mixture model involve use of efficient color space for modeling skin color [47-49]. However the GMM involve complex computations for large data sets and poor performance for single color value.

The explicit skin classifier becomes a prominent candidate having advantages of better performance in the following situations:

• 1.    Static background

• 2.    Easy and fast computation response

• 3.    Optimum memory usage

• 4.    Fixed gesture recognition architecture

The color space thresholds used for skin classifier is discussed in the following section.

• C.    Thresholds for Explicit skin classifier

The explicit classifier defines skin and non-skin colors using a decision boundary rule. This range is selected experimentally after analyzing a variety of human hand for skin color. The pixels in the range of classifier are considered as human skin and are marked as 1 and remaining non skin is marked 0. Thus, with the conventional explicit skin classifier skin can be detected without using much computational work. The main issue with explicit classifier is the false positive results where the skin and non skin color are same. The RGB skin color classifier [50-51] under uniform daylight illumination is:

R>95 and G>40 and B> 20 and

Max {RGB}-min {RGB}>15 and |R-G|>15 and R>G and R>B         (1)

Using normalize RGB color space reduces brightness dependence. Thus, normalized RGB (rgb) can be effective candidate in reducing the effects of the lighting condition and ethnicity. Also, the skin clusters in rgb space have relatively lower variance. Thus, rgb is a good candidate for skin detection but it has a higher computational cost than the basic RGB which is important while considering real time applications.

In the HSV color space the H and S components provide the necessary information about the skin color. The skin classifier for HSV [52] is given below.

V ≥ 40, 0.2 < S < 0.6, 0 < H < 25         (2)

The intuitiveness of Hue-Saturation based color space component and explicit discrimination between Luminance and chrominance properties make this color space popular candidate for skin segmentation.

The popular YCbCr condition [53] for skin detection is

However, it is seen that a single color space classifier may fail if the scope of input skin color goes beyond the defined boundary of decision rule for the skin region. This may be due to the dissimilar response of color spaces for variation in illumination, background and device response characteristics. It is possible that the classifier may detect too many non-skin pixels. Thus to achieve optimum response the combination of color space can be used [54-57].

• D.    Evaluation Technique

The evaluation techniques that are used for skin region detection region can be classified into quantitative and qualitative methods. The qualitative method is natural process, and its goal is to understand the behavior in the response. The evaluation process involves visual interpretation of the response for a given skin classifier [58]. The quantitative evaluation technique involves observation through statistical, mathematical or computation technique. The statistical confusion matrix for evaluating skin segmentation is shown in the following table 5.

Table 5. Confusion Matrix Truth table

	A	Not A
Test A	TP	FP	P (Total Positive)
Test not A	FN	TN	N (Total Negative)
	Correct	Incorrect

Thus to evaluate segmentation the TP, TN, FP, FN, accuracy and precision metrics are used [59-60]. The true positive (TP) indicates the number of pixels correctly identified as skin also referred as hit, True negative (TN) is number of non-skin pixel correctly identified as nonskin, False positive (FP) indicates incorrect segmentation of background pixels and False negative (FN) indicates skin pixel incorrectly identified as non-skin.

The accuracy or recognition rate is the percentage of testing set correctly classified. However the accuracy alone cannot be used as measure for evaluating the performance classifier. In cases where TP < FP the accuracy will always be increasing thus, misleading the evaluation. Thus recall and precision parameters are also used for evaluating classifier.

The Recall indicates the true positive rate. Recall can be thought of as a measure of classifiers completeness. A low recall indicates many false negatives. The precision indicates the Positive predictive. The precision value signifies the exactness of classifier. It is a measure of how noisy is the classifier. A low precision can also indicate a large number of false positive. Hence, the accuracy, recall and precision should have high value for a good classifier. The accuracy, Recall and Precision are defined in below equations:

Accuracy =

TP+TN TP+TN+FP+FN

X 100

RecallTPR =

TP

TP+FN

TP

Precision =        X 100

TP+FP

The true positive rate (TPR) it is same as recall or sensitivity, detection rate and false negative rate. Sensitivity measures the ability of a test to detect the condition when the condition is present.

The Specificity measures ability of a test to correctly exclude the condition when the condition is absent. It refers the proportion of negatives in a binary classification test which are correctly identified. The expression for TPR, FPR and Specificity is given in below equations [61].

TPR =

TP

TP+FN

X 100

FP

FPR =

FP+TN

X 100

Specificity =

TN

TN+FP

It is expected that system should provide high TP, TPR and Low FN, FPR. The term false positive ratio (FPR), also known as the false alarm ratio, usually refers to the expectancy of the false positive ratio. The FPR is a random variable ranging between 0-1. The false acceptance rate (FAR) is the percentage of identification instances in which false acceptance occurs, where skin is detected for non skin object. The expression for FAR and Success rate (SR) is given in the following equation:

FAR =

FP

TP+FP

Table 6. Combination of Color Space used for skin region identification

Author	Color space	Success rate	False negative	Precision
Fan Hai Xiang. Et al. 2013 [44]	RGB-YU'V	-	-	-
Koh Klevan. Etal 2014[56]	Hnew, Snew and g	Average value 96%	-	-
Mohd Zamri Osman.et al. 2015 [54]	YCbCr-SV	84.5	6.9	91.49
R. Vijayanandh et al. 2011 [58]	RGB, YCbCr and CIEL*a*b	-	-	-
Zaher Hamid Al-Tairi et al.2014 [22]	RGB & YUV	95.11	4.8	-

TPR

SR =

TPR+FAR

• E.    Combination of color spaces for skin region identification

The comparative analysis of popular color spaces used for skin region identification shows that there are certain disadvantages if they are considered separately. Thus for optimum response, the combination of color spaces can be utilized for better results.

The combination of color components for optimum skin recognition for different ethnics had been proposed by Koh Klevan et al. [62]. The combination of a*, b* and g, a combination of U , V and g, a combination of H, r, g and b, a combination of Hnew, Snew and g, and a combination of Hnew, Snew, H and g. The combination technique responds better for images under various imaging conditions however the system detects non skin pixels for complex background. Sayantan Thakur et al. [63] have used combination of RGB-HS-YCbCr technique for face detection. The skin regions are extracted from bounding rules. The system provided good detection rate for near-frontal faces. A hybrid approach to human skin region detection has been proposed by R. Vijayanandh et al. [64]. The combination of RGB, YCbCr and CIELab is used for skin region detection. The hybrid algorithm could separate hair, eye, wood and skin from image. Komal Seth et al. [65] proposed a hybrid hand detection algorithm based on hand motion for complex background. The human skin color can be reliable cue for detecting hand in complex background if combined with motion cue. The motion cue is obtained using GMM and the skin color cue is obtained using look up table generated from YCbCr color space.

The Combination of color spaces for skin region identification used by various researchers have been presented in following table 6.

The color spaces like YIQ, YUV, YES, R/G ratio and combination of color spaces have also been proposed for skin detection.

In conclusion, major research work for explicit skin color recognition is done considering RGB, HSV and YCbCr color space. There is a wide scope for implementing these color spaces for skin detection application in an automotive environment.

• III. System Description

The present work is focused on designing a gesture interface system for automotive environment. The system is specifically designed for the rear seat passenger to interact with media player application of the infotainment system. The hand localization is done based on explicit skin classifier technique. The following factors are considered in the design of prototype system:

• 1.    Camera placement for optimum focal length

• 2.  Dimensions of rendering screen

• 3.  Visibility of hand region

• 4.    Gesture selection

• 5.    Response of selected algorithm for optimum speed and accuracy

The system is developed in two stages. The first stage consists of designing a Hi-Vision skin classifier for hand segmentation. The second stage consists of developing graphic user interface with multimode interface features. In the first stage prototype lab setup is developed. The lab setup has a provision of adjustable camera interfaced to infotainment system. This arrangement is made to generate images with various light intensities, color background and orientation. Thus, a classified database of images is developed. The setup is also used to measure the range and span for the proposed system. The arrangement made for lab experiments is shown in the below Fig.3.

LUX Meter

Horizontal Scale

detection a new robust “Hi-Vision” (Hi-Vi) classifier is developed and implemented in MATLAB software. The performance of Hi-Vision classifier is tested over large set of images having varied light orientation and color background.

The hand gesture detection algorithm is developed and implemented using MATLAB 10a using Hi-Vision skin classifier. The first step in hand segmentation consists of acquiring the image in RGB format. The systematic flow diagram for hand segmentation is shown in Fig. 4.

Read input as Image File

Preprocessing Contrast Enhancement

The webcam Intex 306 WC has been used as image sensing device. The camera has following features:

• 1.  Focus distance: 4cm to infinity

• 2.  Frame rate: 30fps

• 3.  Image sensor: 1/7’’ CMOS sensor

• 4.  Resolution: 30 Mega Pixel

• 5.  Power consumption: 160 mW typical

• 6.    Image Control:  Brightness, Contrast, Hue,

• 7.    Automatic sensitive light source

• 8.    USB interface

  The cable used for camera interface is 4 meter long. The camera is interfaced to laptop system having a Core 2 Duo processor and 4 GB RAM. The model LX 101A/LX-102A HTC instruments, LUX meter is used to measure light intensity. The sound level is measured using LT SL-4010 digital sound meter. The algorithm is developed and implemented in MATLAB 10a environment. The arrangement of LUX meter is done to measure light intensity. This provision is made for analyzing the following characteristics:

Saturation, Gamma, White Balance

• 1.    Range: The region between the limits within which the gesture movement is to be measured.

• 2.    Resolution: The minimum detectable change of some variable in a measurement system. The minimum distance for hand movement that is recognized as gesture input.

• 3.    Response time: The response of system for gesture command.

The lab setup is utilized to study the performance of various color models for classification of skin region . This study is useful in understanding the behavior of color models and combinations of color models for identifying skin region for varied luminosity and skin color. The skin boundaries are derived through experimental results. The experimentation is carrier out for classified set of database. The qualitative and quantitative analysis is done on the database images. Thus, based on the response of color models for skin

Hand Segmentation

Hi-Vi

Morphological Operation

Fig.4. Flow Diagram for Hand Segmentation

The acquired image is applied to the preprocessing stage for enhancing contrast and intensity parameters of image. The enhanced color image is processed for skin classification with RGB, HSV and YCbCr color space. Here, the skin candidate is identified as white pixel and remaining non skin area as black. In the next stage the morphological operations are applied to produce a filtered response. The number of identified skin pixels for the three color spaces are used to define the explicit combination rule for RGB, YCbCr and HSV color space. This Hi-Vi classifier is used for determining the skin and non skin pixels. The Hi-Vi classifier is utilized for segmenting hand region since it provides comparatively superior skin classification. Thus, the hand region is segmented and then the center of hand region is calculated. The center of hand is further processed to compute the tip of hand region. The tip of hand is used as pointer for interaction.

In the second stage a user friendly GUI is developed with the Hi-Vi classifier for hand gesture detection. The GUI design consists of options menu, stop, start and pause for user interface. The snapshot of GUI design is shown in the following Fig.5.

The snapshot of GUI interface in automotive is shown in Fig.6.

Fig.6. Snap Shot Of GUI Interface for Automotive

The “Menu” button is used for selecting media file, the start, stop and pause options are used as attributes of media player. These buttons can be accessed through touch interface. The “Start Camera” option provides access to gesture mode for the passenger. After having the control the passenger can browse the media file through horizontal swipe gesture and volume control can be done by vertical swipe gestures. The led interface is used as feedback response for hand detection in ROI. The led glows when skin is detected in the region of interest.

The finger tip is used as pointer and its movement is traced for designing various gesture commands. The pointer movement is traced in the region defined for performing gesture interaction. The classified hand gestures are further used as gesture commands for accessing infotainment system. The touch interface has an event driven frame.

The following algorithm is used for gesture interface:

• 1.    Initialize the parameters.

• 2.    Apply processing tools for contrast & brightness enhancement.

• 3.    Define skin classifier for RGB, YCbCr and HSV color space.

• 4.    Apply morphology filtering on the bin formats.

• 5.    Measure number of detected skin pixels

• 6.    Apply explicit combination rule for RGB, YCbCr and HSV color space Compute center of gravity and tip of palm.

• 7.    Track tip movement for vertical or horizontal motion

• i]    If tip movement is vertical and lower trigger point and upper trigger point are detected then increment volume and vice versa

• ii]    If tip movement is horizontal and LTP and UTP are detected the next media file

• iii]    Similarly for UTP and LTP movement play previous media file.

• 8.    Scan for start/stop/pause/Mute event.

• 9.    Terminate the process.

The JAVA module is used for accessing sound port.

The following are the steps for accessing volume control.

• 1.    Initialize javax sound file.

• 2.    Select audio port address

• i]    Loop over system mixer to find speaker port.

• 3.    Get and open the speaker ports line.

• 4.    Loop over the line control’s to find volume control.

• 5.    Set the volume value according to the user request.

The following are the steps for accessing media file using MATLAB after recognizing the gesture commands:

• 1.    Initialize the counter for number of files in directory

• 2.    Define the path for media file

• 3.    Load the media file pointed by counter and play filename=(fileNames{counter})

• 4.    Increment the counter

filelist=dir(fullfile('Path*.mp3'));

URL=[filename]

If the counter equals length of media files then Reset the counter.

The light condition plays a significant role in vision based recognition system. It is clear that in poor light intensity the captured image will not have optimum information. In the present work the Intex 306-WC webcam is used. The camera has its own light source. Commonly the camera light source has intensity in the range of 250 LUX in dark [66]. Thus, it is required to analyze the cause of external light intensity over the camera view considering the region of interest (ROI). To study this response the experimental setup shown in the following Fig.7. was used.

Fig.7. Arrangement for measuring Intensity

The experiment is performed with camera position fixed at 500mm and intensity is measured over horizontal scale. The external light intensity is varied for various sets of observation. It is observed that the light intensity is greater at the center and fades when moved away from it. The response is plotted in Fig.8.

The optimum position for mounting camera is the basic requirement for capturing image.. The webcams typically have minimum illumination specified for capturing image in the range of 0.5 to 2 LUX in dark. The optimum position of camera is decided through experimentation. In this experiment the external light source is kept at constant intensity and light intensity is measured at center by varying camera position on vertical scale. The experiment was performed in dark room. The Fig.9. shows the relation of light intensity at various camera positions.

Fig.8. Light Intensity Vs Distance

Fig.9. Camera Position Vs Intensity

The camera position is proportional to the measured intensity. Thus, for optimum performance the intensity at ROI must be above 10 LUX. The camera should be placed between 250 to 500 mm for achieving optimum response [67-68].

The qualitative analysis of skin classifiers shows that a particular skin classifier performs well for certain condition. The RGB skin classifier performs well even for the colorful hand images. But it fails to detect skin having similar color in the background condition. The HSV Color space has very good response for the images having skin color background. The YCbCr color space has performs well in varying illumination and also in detecting different skin tone. It shows poor response in case of the background having similar color. The response Hi-Vi classifier is tested by simulating the radiance noise in lab environment. The response of classifier is tested for more than 1000 images from the data base and the images developed form lab setup. The Hi-Vi classifier proves to give satisfactory response for variation in illumination, images with background and various skin tones. The response of RGB, HSV and YCbCr color space skin classifiers with Hi-Vi model was compared. The analysis of skin classifier for different color background was done using lab setup. The response for images taken in lab is shown is in Fig.10.

Fig.10. Response of classifier on lab images

The hand images were captured for different color background. The ground truth (GT) value for skin region was measured by masking the original hand in the image. It is observed that the system could detect skin well for all the backgrounds that were used. However for the colors which are in contrast with skin color like green, dark blue, gray, black the system shows high TPR, accuracy and low FRP. The quantitative evaluation is done on the basis of the ground truth value. The skin pixels classified for Hi-Vision classifier are tabulated in the following table 7.

Table 7. Response of Hi-Vision skin classifier

Accuracy	TPR	FPR	FAR	DR
98.7045	95.856	0.767	0.041	0.999
92.16479	81.728	4.986	0.182	0.997
99.81091	99.311	0.109	0.006	0.999
90.04203	62.882	5.750	0.371	0.994
93.38578	84.121	4.177	0.158	0.998

It is seen from figure 10 that the response of Hi-Vi classifier is very near to the measured ground truth values. The detection rate achieved is in the range of 99%. This classifier provides high true positive ratio and low false positive ratio. The accuracy of system is in the range of 90 % to 99%. The accuracy, TPR, FPR, FAR and DR are calculated for the Hi-Vi classifier.

To study the sound volume variation for gesture interface the vertical hand movement was analyzed. The Fig.11. shows linear relation between volume level and pointer movement.

I

0Н----------1---------1----------1----------1----------1---------1----------1----------1----------1---------1----------1----------1----------1—।

0        5       10       15       20       25       30       35

Hand Displacement

Fig.11. Displacement Vs volume

In this part of experiment the tip of segmented palm is traced as pointer. The movement of vertical pointer causes stepwise increment or decrement in volume. The pointer is exactly mapped to volume attribute. The variation in sound level was measured using LT SL-4010 digital sound meter. The variation in sound level was tabulated for increasing volume in ten steps from 0 to 10. The resolution measured for the gesture interface for volume control is 3 units.

To study the gesture response for browsing file horizontal hand movement was analyzed. In this part of experiment the pointer movement is traced for horizontal hand movement in the gesture recognition area. Hand movement from left to right position is detected as one swipe movement. This swipe gesture movement is interfaced as gesture command for changing next or previous song file. The system has real time response for next and previous song selection.

The prototype setup is develop and implemented with gesture interface in automotive domain. The selection of gesture inventory is for the present application is done considering the practical issues in automotives. In automotives the dynamic hand gestures appear to be much obvious solution than static gestures. The basic advantage of dynamic method is that the position of the hands and fingers is not important in recognition process. Considering the automotive domain and compact interior area, the micro dynamic gestures can be used for optimum performance. A micro-gesture is a gesture that is created from a defined user action that uses small variations in configuration or micro motions. Hence the gesture actions can be achieved with significantly less energy than typical touch motion or sensor- enabled gestures. This has great benefits for user as it allows effortless interactions to occur in rapid succession which directly supports greater fidelity and productivity within natural user interface. Some popular micro gestures are finger slide, pinch, taps, and sweep.

The micro gestures can deal in a better way with the hindrance caused due to automotive environment. Thus, in the present work instead of using incremental swipe gesture for volume control and browsing media files micro swipe gesture are used. The developed microgesture will increase volume by one level for every complete swipe gesture. This will minimize false input. Also one horizontal swipe will change the file attribute.

The set of possible gesture commands that can be generated from the developed setup for the targeted application is summarized in the following Table 8.

Table 8. Gesture set for the developed prototype system

Gesture	Possible Application
Vertical up- right side	Volume increment by one level
Vertical down-right side	Volume decrement by one level
Right to left	Play next file
Left to Right	Play previous file
Vertical up-left side	Play
Vertical down-left side	Pause

The response for classifiers for the images taken in automotive is shown is in Fig.12.

Original     RGB

HSV

YCbCr      Hi-Vi

Я » м^и^^я

1 1(1 ■■

a)

(b)

(c)

(d)

(e)

Fig.12. (a) Original image. (b) RGB classification. (c) HSV classification. (d) YCbCr classification. (e) Response of Proposed method

The system response is tested in lab for all the practical issues considering the automotive environment. The response Hi-Vi classifier is tested by simulating the radiance noise in lab environment. The system showed exact response for the gray, dark blue and black backgrounds. The media file could be accessed and new song could be played by just swipe gesture. Horizontal swipe movement was used to access media file and vertical swipe movement was used to control volume. The pilot system was tested in automotive environment.

• V.    Conclusion

In the present work a brief study of color models used for extracting skin region, various gesture classification techniques and gesture inventories is presented. The analysis of the response of color classifiers for images with varying light conditions, backgrounds and skin tones is presented. The observations were taken on the developed proto type lab setup. It is observed that color classifiers used for skin identification have certain disadvantages if they are considered separately. In the present work a combination of RGB, HSV and YCbCr color space is used to define the Hi-Vi skin classifier. This hybrid classifier provides superior response as compared to other color spaces.

The paper presents a touch less solution for interfacing with infotainment system in automotive environment. In the present design the gesture interface is proposed for the passenger seats. The design of gesture based media access control for the passengers in automotive domain is a novel solution in minimizing driver efforts. The developed interactions are real time and easy to use. The advantages of developed system are immunity to light variations, user independence, driver’s control for providing access to users and easy gesture set. Although static gestures like close palm, open palm, finger count can be use with the same setup. The dynamic micro gestures have been used here in particular. The basic advantage of using the swipe gesture to access media player attributes is immunity of its response to the hindrance while passing over speed breakers, and during sudden start or stop . Also using the swipe gesture it is possible to achieve exact recognition rate even with adjustable seat.

In future, other attributes of infotainment system can be considered in the designing a prominent interface for automotive.