Real time Bangla vehicle plate recognition towards the need of efficient model - a comprehensive study

Published Online December 2018 in MECS

Population is the big challenge in many countries, i.e. Bangladesh, India where road traffic and traffic-related crime is increasing day by day such as stealing, hit and run, abduction, despoilment, trafficking, murder etc. Those cases remain unresolved for lacking evidence i.e. identification of vehicle as recognition of vehicle plate is still an unsolved problem in computer vision, pattern recognition and image processing research domains. Automatic Number Plate Recognition (ANPR) refers to recognize vehicle and verify number plate by using various License Plate Recognition (LPR) methods. In these methods, image is considered as input, then process characters recognition on license plate using various recognition techniques. Automatic Number Plate Recognition (ANPR) can be used in toll gate automatic charging, parking charging, and crime on road identification, traffic monitoring etc. Countries like Italy, Korea, and Vietnam recently focus on vehicle plate recognition, however, countries still far way behind far behind on this research, although recent research still need robust results based on the respective environmental experimentation. This research performs comprehensive reviews on various recognition approaches in very details in various aspects, i.e. methods, frameworks and experimental results on various datasets.

In general view, overall recognition process of vehicle number plate recognition can be stated as four stages [1], i.e. first stage is to accrue full car images or videos with the camera with modeling of camera parameters i.e. camera resolutions, sutter speed, lights. The second stage is to locate vehicle plate region from image or video based on plate boundary features, color features, present of characters features and various edge detections. Third stage is the character segmentation based on the color combination and characters position. The final stage is to extract characters recognition using template matching [13], [16, 17], neural network [10,11], [21,22, 23] or fuzzy classifiers [20] and alike techniques. First two stages is the image processing step and evaluation depends on success rate and error rate. Previous approaches have some advantages and disadvantages in various experimental situations. In this research, attention is paid in both accuracy and time complexity besides proposed research focused on real-time recognition techniques. Some methods are faster than others [6, 7]

• [24]    some have the more overall accurate result than other techniques [6] [12] [24]. The maximum technique first converts the image in grayscale to remove color confusion [8] [14] [16,17]. The real-time recognition system is absence in many techniques applied in previous study. Research in Bangla vehicle license plate detection is still quite low over against English vehicle plate. Several plate detection techniques have been proposed for Bangla vehicle plate i.e. multilayer feedforward neural network, artificial neural network, chain code neural network.

The main purpose of this comprehensive study is to provide details of existing methods so far used in automatic vehicle plate recognition with various surrounding’s challenges and comprehensive experimental evaluation for the validation of the existing methods and frameworks. Section II represents core background study, Section III presents review based on methods, Section IV illustrates review based on frameworks, Section V demonstrates review based on experimental results in various aspects comprehensively, Section VI presents observation and discussion consisting of three subsections and finally, section VII presents concluding remarks.

• II.    Related Work

Vehicle plate detection and recognition have a large number of application and importance in computer vision and pattern recognition research domain, more specifically in Intelligent Transportation System (ITS). Resultant output can replace manual input in parking and toll collection in roads having the higher traffic problem. Previous research of vehicle number plate detection and recognition used various methods. Most of the previous related research is divided into four processing steps: enhancement of a digital image which can consider as a pre-processing of the source image, number plate localization or detect the number plate region from preprocessed image, apply character segmentation on the extracted number plate and character recognition.

For pre-processing, median filter is used which can apply for eliminating noise, tophat-bothat used in Ref. [25] which can overcome illumination condition or different light condition and deal with night and day mode as well [12]. Gabor filter is more successful than others in extraction as Gabor filter can deal with different camera angle and overcome the problem of camera and vehicle distance [2]. Hough transform also can deal with different rotation angle [14]. For segmentation vertical and horizontal projection is used in most of the previous research and has satisfactory segmentation result [9][24][18][26]. Then, various methods is used for character recognition and optical character recognition [17][19][34][35]. However, existing methods cannot deal with misalignment of characters and different sizes of character. Featured based template matching is done by resampling, however, template matching does not work in real time condition, i.e. in different pan tilted image or rotation of License Plate (LP) [26]. Support vector machine [44], naive Bayesian classifier [30], Neural Network [11], Multi-layered Perceptron [24], and Learning Vector Quantization [11] are as Supervised Learning for vehicle plate recognition. However, these methods did not perform well in gray image function and lack of real time processing capability in the previous research. Among these methods, mathematical morphological tophat–bothat transform receives more attention due to perform well in gray image function and neural network due to adaptability with different scenario.

• A.    Mathematical Morphological Tophat–Bothat

Transform

If the grayscale of the target image increased, the background grayscale should decrease. Previous research found that mathematical morphological tophat–bothat transform can perform well based on the gray image function [41]. The main idea of mathematical morphological image processing is to make use of a structural element to probe into an image so as to understand its structure characteristics by parts correlation. The basic mathematical morphological operations are erosion and dilation and the others can be achieved by these operations [41].

• B.    Back Propagation Neural Network

BP (Back Propagation) Neural Network is a supervised neural network which can be combination of three layers including the input layer, hidden layer and output layer. For training the Back Propagation Neural Network (NN), learning rate is used between normally as 0.09. 540 input nodes are used in Back Propagation Neural Network (BP NN) for features of each character, for hidden layer 36 nodes are used and for the output layer, another 36 nodes are used [11].

The main purpose of this comprehensive review is to analyze existing research in various aspects, i.e. analysis on existing methods, analysis on existing frameworks and critical review on existing experimental results in various views, i.e. results, datasets, relevancy with real time experimental evaluations.

• III.    Review based on Methods

Template matching is an image processing technique which works by matching an image portion with the existing templates [4] [6] [13] [16] [17]. In previous research, template matching method matches the input images characters with stored images and gives output. The overall process is simple and straightforward, however, template matching algorithm process insignificant pixels. In addition, the method cannot deal with rotation change of License Plate as well. By using improved feature extraction method non-important pixel process problem of the current method can be resolved [5] [11] [15] [16]. Besides, Hough transform method can handle rotation change problem of template matching method can resolve by detecting parallel line segments [14][22]. In addition, as some car plate may be dirty, in that case, segmentation of the characters with feedback self-learning and hybrid binarization techniques is used to regulate the parameters [9].

Since some countries have specific color in number plate, previous research worked using color feature by locating the specific color [7]. However, in illumination variation

Fig.1. Existing methods from previous research

Table 1. Existing methods with advantages and disadvantages

Methods	Advantages	Disadvantages
Shape    based Detection[4] [6]	Easy, straightforward, speedy	Sensitive to noisy edge
Adjacent Pixel Property [5]	Flexible to plate rotation. Simple and straight technique.	Cannot deal with joined or broken characters.
Color feature based detection [7]	Perform well for Malformed license plates and able to sight prone	Poor performance for illumination variation and sensitive to noise
Hotelling transform[9]	Can extract salient feature. Recognition process is fast.	Overall   process   is computationally expensive  and  not robust.
Feed Forward Neural Network[10] [11] [21][24]	Can    deal    with ambiguous characters.	Computationally expensive
Template matching13] [16] [17]	Easy, straightforward and speedy.	Process unnecessary pixels, does not perform well with ambiguous characters, thickness changes, image noise, rotation problem
Horizontal and vertical projections [18] [26]	Characters extraction is independent of their own space	Needs prior knowledge and projection value can be affected by image noise
Neural Network [23]	Provide good color knowledge and intensity represents texture information.	Computationally complex   and   not robust

RGB color format is. Every car license plate position is fixed on the car body and color combination is also fixed. There are only two color combinations i.e. one is background color, another is text color and unique color combination in plate area. Using color features for vehicle plate recognition, noise is a big issue for accurate results [5] [7]. By using median filter as nonlinear digital filtering, noise can be resolved [12]. Besides, color feature and fuzzy aggregation is used for extraction, connected component and blob coloring applied for segmentation, self-organizing character recognition applied for final recognition to ensure more accurate result in various condition and scene[20] [21]. However, these methods most of the cases, did provide expected results in real time scenarios. Neural network acts as more efficient approach to resolve the real-time processes problems where Backpropagation neural network method is used to recognize vehicle number plate [24]. In addition, problems of template matching and OCR, i.e. ambiguity of characters (B and 8, D and O, Z and 2 a, O and 0) also can overcome by using multi-layered perceptron Artificial Neural Network (ANN) and FeedForward Back Propagation Neural Network (NN). Fig.1. illustrates existing methods and Table 1 illustrates existing methods with advantages and disadvantages.

• IV.    Review Based On Framework

This section is comprehensively divided into two subsections, i.e. segmentation of vehicle plate characters and recognition of vehicle plate characters shown in Fig. 2.

Fig.2. Existing frameworks in various aspects from previous research

• 1)    Segmentation of Vehicle Plate Character

Feature extraction based framework focused on highest pixel density by plotting horizontal projection in the lower part of the image which helps to get a rough location of the license plate region and then control the contrast to get the characters [26]. However, overall feature extraction based framework is not applicable in real time scenario as number plate can be anywhere of the picture.

Hybrid frameworks, i.e. Hough transform and contour algorithm localizes the number plate to get closed boundaries of the objects and find two interacted parallel lines which are transformed to Hough coordinate to be considered as a plate-candidate [22]. However, there may be images that have other objects such as wear glasses, headlights, decorated things and so on. These objects may also have the shape of interacted 2-parallel lines, and therefore, are also falsely detected as plate-candidates. In addition, plate candidate verification is another disadvantage of using Hough transform and contour algorithm together as hybrid framework. In this scenario, usage of Gabor filter localization is more successful than other frameworks where the resultant of convolution with Gabor filters, is considered as license plate detector [2]. Probable plate region is considered from high values of the image which is the high value of the considered input image. To segment the regions, binary conversion of image is first performed followed by applying thresholding algorithm. To merge nearby regions morphological dilation operator is applied to that binary image. After that, eight-connected blob coloring extraction was done to find plate regions followed by thresholding and vertical projection to obtain the segmented character. Similarly, previous research applied vertical and horizontal projection for the segmentation of characters [18] [26] [27]. Ref. [18] detected boundaries by scanning the image from the middle of the image from number plate with an assumption of minimum character in 6. Ref. [26] used the idea about the boundary of the upper and lower portion of license plate, project the pixel density horizontally which is row-wise in the license plate. Then, other areas will be discarded for next step. After that, to get the starting point and ending point for each character, pixel density is projected vertically, where the scenario considered that some number plate may have a dash or dot, that problem is solved by Eq. (1),

C = ( O) >  T (1)

B

Where C denotes area criteria, O denotes area of the object, B denotes total area of the box and T denotes threshold.

• 2)    Recognition of Vehicle Plate Character

For character recognition, template-based pattern matching method is applied to recognize characters by resampling the characters according to the estimation of character size where the character height is similar to the height of the license plate and the license plate is resampled according to height [33]. After that, matching is checked by pattern recognition and resample the license plate. However, pattern matching cannot deal with real time situation where rotation and different pan-tilt images may exist.

Optical Character Recognition (OCR) is applied to recognize the character from the license plate where all white pixels of the license plates are transformed into black and black pixels to white[17][35]. The plate background becomes black and the text becomes white. Line separation process is applied to get the individual lines in the separated text. The line split adds each pixel's value in a row where consideration of no character is found if the addition result of the row is zero and if the addition result of the row is larger than zero, means there are text in the row. If the total of the sum is larger than zero that denotes the beginning of the line and after this when the first total sum will be zero denotes the end of the line. The starting and ending values of the line is used to crop the first line. To separate the second line in the text the same process applied again. However, OCR has some problem for misplacement of characters.

Artificial Neural Network (ANN) is also used to recognize the characters where ANN mostly useful to overcome the ambiguity of characters [10] [11] [12]. Multi-Layered Perceptron (MLP) ANN model was also used for character classification which requires three layers to process recognition. Firstly, the input layer which incorporates the information which is used to make the decision. Secondly, the hidden layer which assists the network to compute additional sophisticated associations and finally, the output layer which includes the resulting decision. For training the ANN, feed forward back propagation algorithm was used for efficient performance. The performance of the ANN measures using mean square error (MSE). In addition, Learning vector Quantization (LVQ NN) was also used previously which combines competitive learning with supervision [11]. Using these ANNs success rate is remarkable where processing time is lower than other techniques.

• V. Review based on Experimental Results

This research performs comprehensive reviews on previous research validation in terms with accuracy levels, time required for detection for different techniques of license plate, character segmentation and character recognition and various datasets with processing time show in Table1, Table, and Table 2.

• A.    Validation based on Accuracy VS Methods

For rotation invariance, accuracy of number plate images is 95% [32]. Tophat used for pre-processing under a difficult condition [11] [25], accuracy of recognition under difficult condition using top-hat is almost 99%, but without the top-hat algorithm, it’s nearly 50% [25]. So, top-hat has a great role in pre-processing of images in the difficult condition like night mode and rainy condition. Vertical edge extraction or vertical projection is used to extract the number plate from the captured image which the 90% accuracy for localization [34].

Neural networks provides recognition rate of 98% to 99.23% which is remarkable and applied on different resolution images with various position angle where accuracy increased by the iteration of convolution [10][27][28][29]. Besides, Backpropagation Neural Network using various training set increases accuracy significantly [10] [32].

Table 2. Experimental Results based on Accuracy on Various Datasets

Methods	Datasets	Accuracy
Template matching [19]	2340 images of Italian plates in Different weather and illumination conditions	Total procedure accuracy is 91.1% and OCR success rate is 98.6%
Template matching [13]	Image captured from 4-5 meters away and taken under various illumination conditions.	Plate       region extraction accuracy is       97.6%,for characters segmentation 96% and for character recognition 98.8%. Overall performance    is 92.57%
Template matching [37]	610 images captured in various illumination conditions, different angle of view and dirty plates	95.24%
Template matching [38]	180 pairs of images, Multi-plates at a time with  occlusion  and different sizes	94.4% accuracy for License      plate extraction     and 95.7% for OCR
Neural   Network [29]	6000 Bangla vehicle plate images	Character segmentation -98% and      character recognition accuracy -98%
Multi-Layer Perceptron neural network,     back propagation algorithm [31]	700    images    of Vietnam vehicles	Average 97.43%
HSV color space, Vertical      edge extraction, Connected component analysis [36]	Applied on two types of number plateyellow, non-yellow.	For yellow number plate 95.55% and For non-yellow number plate 98.94%
Generalized Symmetry Transform(GST) [40]	330 images in Various viewing Directions of Korean number plates	93.6%
Vertical and horizontal projections [39]	805    Images    of Vietnamese  Number plates   in   different rotation and lighting conditions	LPE accuracy-98.8% OCR accuracy-97.5% and Total success rate 92.9%

Classifiers like Bayesian Classifier provide 94.4% and 87.8% accuracy for the k-NN classifier [30]. Featured based recognition has lower accuracy of 78% in daytime and 70% accuracy in night time which is lower than CNN and other algorithms [26].Template matching have satisfactory percentage of accuracy, i.e. 97.6% success rate for the extraction of number plate, 96% accuracy for character segmentation and 98.8% accuracy for character recognition [3] [13] [16] [25] [33]. Optical Character Recognition (OCR) provides accuracy of 97% to recognize the character of the license plate [19] [34] [35]. Although, existing research mentioned above provides higher accuracy, most of experimental results depend on various situations, as there is an absence of standard, i.e. rural condition, inefficient in noisy scenario etc. Fig.3. shows accuracy among various methods from existing research and Table 2 shows experimental results based on accuracy in accordance with methods and datasets.

• B.    Validation based on Processing Time VS Methods with Various Datasets

Existing research using Template matching based recognition provides processing time of 75 ms where various illumination conditions are considered [13, 38]. However, connected component based vehicle plate recognition provides processing time of 47 ms and 92 ms for variation of colors dependent vehicle plates [36]. Generalized Symmetry Transform (GST) provides accuracy of 1.3 seconds in various viewing directions of vehicle plates [40]. Fig.4. shows processing among various methods from existing research and Table 3 illustrates experimental results in terms with processing time in accordance with methods and datasets.

Table 3. Experimental results based on pprocessing ttime VS methods with various datasets

Methods	Processing Time	Datasets
Template matching [13]	75 ms	Image captured from 4-5 meters away and taken under various illumination conditions.
Template matching [38]	75 ms	180 pairs  of images, Multi-plates at a time with   occlusion    and different sizes
HSV     color space, Vertical edge extraction, Connected component analysis [36]	For     yellow color   License Plate- 92ms , For non-yellow License   Plate 47ms	Applied on two types of number plate-yellow, nonyellow.
Generalized Symmetry Transform(GST) [40]	1.3 sec	330 Images in Various viewing Directions of Korean number plates
Vertical and horizontal projections [39]	0.65 s for License Plate Extraction and 0.1s for OCR	805      Images      of Vietnamese     Number plates in different rotation and lighting conditions

Fig.3. Accuracy comparison chart among various methods in existing methods

Fig.5. Challenges of Bangla vehicle plate recognition from previous research

• C.    Validation based on Relevancy with Real Time Application

Relevancy with real time is vital issue to validate existing methods used for vehicle number plate recognition. Most of previous research related to segmentation or recognition of characters from vehicle plate recognition applied in real time scenario, i.e. template matching [38], Multi-Layer Perceptron neural network, back propagation algorithm [31]. However, methods like Neural Network [29], Generalized Symmetry Transform (GST) [40], Vertical and horizontal projections [39] were not validated under real time scenario. Table 4 represents validation on relevancy with real time scenario of existing research.

Fig.4. Processing time among various methods from existing research.

Table 4. Experimental Results based on Relevancy with Real Time Scenario

Applied in Real Time Scenario	Did Not Apply in Real Time Scenario
Template matching [38], MultiLayer Perceptron neural network, back propagation algorithm [31]	Template matching [19], Neural Network    [29],    Generalized Symmetry Transform(GST) [40], Vertical      and      horizontal projections [39]

• VI. Observation And Discussion

Proposed research illustrates challenges for Bangla vehicle plate recognition from existing methods, frameworks and experimental results into three core categories, i.e. Potential difficulties in vehicle plate recognition, lack of suitable methods in vehicle plate recognition, efficient model for vehicle recognition mentioned in Fig.5.

Potential difficulties are mentioned in Potential difficulties in vehicle plate recognitions section. Lack of suitable methods includes various potential challenges observed in previous methods. After that, based on investigation on potential difficulties in vehicle plate recognition and lack of method in vehicle plate recognition, this research demonstrates the need of efficient model for vehicle plate recognition. Observation in the need of efficient model includes extraction of features for vehicle number plate, segmentation of characters from vehicle and recognition of characters from segmentation of vehicle number plate.

• A)    Potential Difficulties in Vehicle Plate Recognition

License plate recognition is extracting the license plate information from image or videos of a vehicle. Images and videos may be captured with various types of camera like infrared, color or black and white camera. This research demonstrated reviews on some efficient and computationally fast methods and algorithms which can be applied detect Bangla number plate and used for the traffic management in the metropolitan area of large and important cities in Bangladesh or in tollbooths of busy roads. Bangla vehicle number plate consists of different number of characters, but in metro areas vehicle number plates have 11 characters and two hyphens in a fixed position, i.e. first 4 characters mean the metro area name and the next character introduce the vehicle type and rest numerical numbers represents the registration number. There will be a variety of potential difficulties, i.e.

• 1)    Low-resolution images usually when the distance between vehicle and camera is high,

• 2)    Usage of low-quality camera for detection,

• 3)    Images with the blurry condition, particularly

• 4)    Poor lighting,

• 5)    motion blur and

• 6)  Low contrast owing to overexposure,

• 7)  Shadows or reflection.

Two cars from completely different cities will have a different identical variety with different style of the plate. In addition, number plate localization is the next challenge as the real-life scenario may have the complicated background.

• B)    Lack of Suitable Methods in Vehicle Plate

Recognition

Existing methods depend on the different viewing angle or pan-tilt of viewing and distance of placement between the vehicle and camera. The main distinction is within the degree of dependency. Methods like template matching cannot deal with different rotation angle and process non-important pixels. To handle the rotation problem Hough Transform can be followed. However, the problem of using Hough Transform is the higher processing time.

To handle different illumination condition sensible preprocessing strategies like image enhancement need to be applied to take away the lighting effect and make the number plate remarkable or notable. Combining Hough transform and contour algorithm may fail to extract number plate if the image quality is poor as edges are not clear.

• C)    Efficient Model for Vehicle Plate Recognition

Based on the critical and comprehensive reviews above, this research illustrates for the need efficient model where overall vehicle plate recognition for real time scenarios can be composed three phases, i.e. Extraction of vehicle number plate, segmentation of characters from vehicle number plate and finally, recognition of characters from the segmented characters from the previous phase.

• i)    Extraction of Vehicle Number Plate

For number plate extraction from videos, first step should be the extraction of frames and finding out which frame has the vehicle. Based on the situation, there can be problem, i.e. if the vehicle is moving fast by not maintaining the speed limit then vehicle number plate may not clear. In terms of the real-time image may have a complicated background with different trees, houses and shops which initiate the need to extract number plate with morphological analysis or edge efficient detection methods as the number of the edge is more than License plate area. Gabor Transform can be applied for texture analysis of License Plate because Gabor Transform can deal with the limitation of the distance between vehicle and camera or viewing angle. After getting License Plate region by Gabor transform, need to apply thresholding, Morphological analysis and eight connected blob coloring to extract the number plate, although the overall procedure claims to be expensive.

• ii)    Segmentation of Characters from Vehicle Number Plate

After getting extracted License Plate, the next task is to segment the characters of the License Plate which is the extraction of each character. The extracted character will lead to go for the next step recognition. Vertical and horizontal projection can be a reliable approach for segmentation where a threshold value should set for projection which depends on the number of minimum character in License Plate. Besides, Binary processing and Vector quantization may apply for segmentation which may have more processing time due to computational requirements.

• iii)    Recognition of Characters from Segmentation Characters from Vehicle Number Plate

For recognition of characters Optical Character Recognition, Template Matching, Different Classifier, Support Vector Machine (SVM) and Artificial Neural Network (ANN) are applied in previous research. OCR and Template Matching cannot deal problems like ambiguous character. SVM cannot deal problems arises for illumination conditions, orientation of characters and physically damaged License Plates. According to the best computational time and accuracy level, Backpropagation, ANN and Learning Vector Quantization NN is suggested for recognition of characters and train the data set for future detection.

• VII. Conclusion

This research illustrates the challenges for Bangla Vehicle Plate Recognition as different real-time situations as multiple types of plate recognition, ambiguous character recognition, License Plate Recognition from videos, multiple plate recognition at a time, different illumination condition. In this paper, comprehensive review based on the various comparison of license plate recognition is demonstrated, i.e. methods, frameworks and their experimental results including advantages and disadvantages. This research performs critical reviews on experimental validation into five aspects, i.e. methods, datasets, accuracy, processing time and relevancy with real time scenario.Based on the challenges, reviews on various aspects, he recommended model is expected to be efficient to detect the number plate from various distance, perform detection in different conditions and deal with ambiguous characters where the main focus of the proposed model to achieve higher accuracy with least precision in real time. Propose comprehensive reviews are expected to reveal efficient Bengali vehicle plate recognition and significantly contribute in Intelligent Transportation System (ITS) research.