Fruit recognition using color and morphological features fusion

Published Online October 2019 in MECS

Recognition system that is one of computer vision challenges has emerged to achieve near human levels of recognition under a wide variety of conditions. Several challenges overcome to automatic recognize the kind of fruits using the image. Many kinds of fruits are affecting with the variation of images, such as color, texture, and shape.

Various researchers have successfully proposed automatic fruit recognition using computer vision. Some methods for fruit recognition are feature-based recognition, detection-based recognition, and deep learning-based recognition. This study is only considered on the feature-based fruit recognition methods because the variety of fruit type is basically depended on the features of fruits.

However, one type of fruit can be a variation of shape, color, and texture as well as different fruits can have the same shape, color, and texture. Many previous researches based upon color feature and texture feature, or color feature and shape feature, or color feature, shape feature and size feature. However, this paper proposes an efficient fusion of color and morphological features; and classifies the types of fruits by applying five different classifiers.

Section 2 of this paper reviews the previous related papers. Section 3 outlines the methodology of the proposed methods. The implementation and comparative results are shown in section 4. Section 5 concludes the proposed method.

• II.    Related Works

  Li et al. (2017) proposed the green apple recognition method based on the combination of texture and shape features. This recognition method has used the combinations of texture features, shape features, and color features to extract the background and target apple that has picked by the robot in the complex background. The recognition rate and running rate are more superior to other recognition algorithms. However, fruit with slight background occlusion and overlapping fruits cannot efficiently segment. That can be future learning. [1]

Shukla et al. (2016) presented the fruit recognition method for nine different classes of fruits. The first stage of this method is background subtraction and fruit block extraction as pre-processing. Color, shape, and texture features are combined to represent the visual characteristics of fruits. Multiclass SVM and KNN classifiers are used to pass the feature dataset for fruit recognition. The result that extracts by using KNN (for K=2) is better than SVM. This system is only considered the images that exist the simple background. [2]

Hu et al. (2016) submitted the new segmentation method for fruits with plastic packing and a modified minimum distance classifier. The image is segmented with HSV color space to achieve the region of interest (ROI). Color features are extracted by using Hue and Situation from the HSV color space, and texture features are extracted by applying gray level co-occurrence matrix (GLCM). This also proposed the modification method based on minimum distance classifier is considered to recognize the pattern on the feature vectors. The recognition rate is over 91% on the modified classifier. To improve the method and the recognition rate, many fruit types are required and more features need to extract. [3]

Arivazhagan et al. (2010) introduced the fruit recognition strategy by applying an efficient fusion of color and texture features. Before feature extraction, background extraction is processed by using the H and S channel from HSV images. Statistical features are extracted for the color features and co-occurrence features extracted from the Wavelet transformed subbands for the texture feature. This recognizes the types of fruits with the minimum distance classifier for Supermarket Produce data set. When the color and texture information are complementary, the classification process obtains the good results. [4]

Jana et al. (2017) exploited the automatic fruit recognition from color and texture features of natural fruit images. This recognition process firstly works image-level segmentation by using GrabCut, and then the region of interest is defined from the segmented image. 12 color statistical features and 16 texture features with gray-level co-occurrence matrix are extracted to classify 8 classes of fruits. SVM is used to train the features and the accuracy of the proposed system is 83.33%. The segmentation method can correctly work for the various color fruits that capture the natural scene. The authors suggest working the fruits counting process by extending this process for the future scope. [5]

Ilyas et al. (2018) proposed the robust, accurate and flexible method to identify the varieties of fruits by using color, shape, size and texture features. This works with three main processes: acquisition and cropping of images, extraction of features, and classification. The acquisition and cropping process operates by calculating the mean value of the RGB component. For the shape and size features, the area and perimeter of fruit are calculated. The contract, energy, homogeneity, and correlation from gray-level co-occurrence matrix are used for the texture features. K-nearest neighbor algorithm with k=1 is applied as a classifier to train the features extracted from fruit images. The recognition results of the proposed system are up to 97% and the proposed system can also efficiently recognize the types of fruit that exist in different angles and position. To get more accurate results, the authors suggested that to increase the database size. [6]

Dubey et al. (2015) introduced the fruit and vegetable recognition framework. In the framework, background subtraction is firstly processed to detect the foreground fruit and vegetable objects in the images by using the S channel of the HSV color space and K-mean algorithm. Color and texture features are extracted for the fusion approach. Global color histogram (GCH), color coherence vector (CCV), and color difference histogram (CDH) are used for color features and structure element histogram (SEH), local binary pattern (LBP), local ternary pattern (LTP), and completed local binary pattern (CLBP) for texture features. For training and classification, a multiclass support vector machine is used as the classifier. When recognizing the 15 types of fruits and vegetables by fusing color and texture features, the experimental results get more accurate recognition rate. For the future work related to this framework, the authors’ advice is to classify the type of fruits’ diseases from the images. [7]

Naskar et al. (2015) proposed a feature-based fruit recognition technique. This technique used the Log Gabor filter for the texture features, mean value of the hue color for color features, and counting perimeter and area pixel for shape features. This study worked on the fruit data set that comprised of six different types of fruit. They implement the proposed technique on the Matlab Neural Network Toolbox. The confusion matrix is used to calculate the performance of the technique. The proposed technique can precisely recognize the type of fruit. The constraint is that this work is only considered on the white background image. That constraint can work as future work. [8]

Jana et al. (2016) exploited proposes for the solution to solve a viewpoint invariant for fruit intra-class recognition by combining color and texture features and using a Neural Network (NN) classifier. The standard deviation of each RGB color channel is extracted from the color histogram for color features, and the gray-level co-occurrence matrix is calculated with four directions for texture features. To classify the type of fruit, color and texture features are combined as a 19-element feature vector by concatenating 3-element color feature vector and 16-element texture feature vector. The accuracies of the system for each fruit class obtain between 90 % and 100%. Even the angle of view is changed; the results of the system are also accepTable The reorganization process for the subclass of fruit type was suggested for future work by authors. [9]

• III.    Methodology

In this section, we present a system flow of the fruit recognition system. For fruit recognition, there are three main parts: preprocessing, feature extraction, and classifier. The preprocessing and feature extraction are worked for both training images and testing images of the dataset. To recognize the type of fruit, we first extract the features from fruits image dataset with the first two steps. And then, we will use the classifier to recognize the types of fruits. The flow of the system is shown in Figure 1.

Images

Fig.1. System Flow for Fruit Recognition System

• A.    Image Pre-Processing

Fig.2. Preprocess Images

• B.    Feature Extraction

We extract color features and morphological features are extracted from the image to recognize the types of fruit. The fusion of color features and morphological features are proposed in this system to get the high accuracy for fruit type recognition and the texture features are also extracted to compare with the proposed fusion model. Four statistical features, such as mean ( µ ), standard deviation (σ), skewness (s), and kurtosis (k), are extracted based on the color and morphological processing.

Mean statistical features are calculated the average value of each color channel of an image. The mean value of the i-th color channel is defined as µ i and calculates with the following equation on the pixel values of the image. In the equation, M and N are the number of row pixels and column pixels respectively, and P i (j,k) is the (j * k) pixel values in the i-th color channel of the image.

MN µ i = _M ¹ _N ∑ _{j = 1} ∑ _{k = 1} P i ( j , k )

Next statistical feature for standard deviation is calculated the square root value on the variance of each color channel. σ i is defined as the standard deviation of the i-th color channel,

MN

σ i =   _M ¹ _N ∑ _{j = 1} ∑ _{k = 1} ( P i ( j , k ) - µ i )²

Third statistical feature, skewness, measures the asymmetric of the color distribution and defined as s i with the following equation.

MN si = 3 ( 1  ∑M ∑N (Pi(j,k)-µi)3)    (3)

^MN j = 1 k = 1

The last statistical feature is kurtosis that is similar to the concept of skewness. Kurtosis is a shape descriptor and it measures the tailedness of a probability distribution. Kurtosis of each color channel is defined as the ki, ki = µ44                  (4)

σ

• a)    Color Features (CF)

Color images are composed of three channels: Red, Green, and Blue. Each channel is used as the identifier of the image and four statistical features, such as mean ( µ ), standard deviation (σ), skewness (s), and kurtosis (k), are extracted from each channel. Color features are extracted with four statistical operations on the three RGB channels of the image as shown in Figure 3. So, 12 color features (CF) are extracted in this study.

CF = [ µ _r , µ _g , µ _b , σ _r , σ _g , σ _b , s_r , s_g , s_b , k_r , k_g , k_b ] (5)

Fig.3 RGB Color Cannel of Fruit Image

• b)    Morphological Features (MF)

Morphological features are also extracted based on the boundary extracted image by operating with the morphological operation. To extract the object boundary from the image, this study operates with five stages. The boundary extraction steps are shown in Figure 4 and the boundary extracted image is shown in Figure 5.

P ( A^C ) = 1 - P ( A ) (6)

Image

Boundary Extracted Image

Fig.4 Steps for Boundary Extration

Fig.5. RGB Color Cannel of Fruit Image

A Θ B = { z | ( B ) z ⊆ A }            (7)

A ® B = { z | ( B ) z П A / ф }        (8)

For the morphological features, the statistical features, such as mean ( µ ), standard deviation (σ), skewness (s), and kurtosis (k) are used on the boundary extracted images.

MF = [ µ _bi , σ _bi , s _bi , k _bi ]             (9)

• c)    Feature Fusion (FF)

Feature fusion (FF) is calculated by combining 12 color features, and 4 morphological features. The fusion is extracted 16 features to recognize the types of fruits.

FF = [ CF , MF ]          (10)

d) Texture Feature (TF)

The texture is a repeated pattern of information or arrangement of the structure with regular intervals. In a general sense, texture refers to surface characteristics and appearance of an object given by the size, shape, density, arrangement, the proportion of its elementary part. The characteristics of the image region are used as texture features. This study uses the Gray Level Co-occurrence Matrix (GLCM) for the texture features. GLCM is based on the repeated occurrences of gray level configuration in the texture [10]. GLCM matrix is computed four texture features: Contrast (Ct), Correlation (Cn), Energy (Ey) and Homogeneity (Hy) as shown in equation (11) to (14).

M - 1 N - 1

j - 0 k - 0

M -,¹ N ^^{1 ( j -} Ц j )( ^{k -} Ц к ) ^S jk

Cn - xx---------- j-0 к-0          CTjCTк

M - 1 N - 1

j - 0 k - 0

M - 1 N - 1     V

ну -XX j-0 k -0 1+ I j - k I

GLCM has extracted four features along with four directions: horizontal (0 ° ), vertical (90 ° ), left-diagonal (45 ° ), and right-diagonal (135 ° ). So, GLCM has extracted 16 texture features; 4 texture features are calculated with 4 directions.

TF - [ Ct₀ „, Ct₄₅.

Ct ^i 90 °

Ct t 135°,

Cn , Cn

45 °

ⁿ 90 °

, '- ⁿ 135⁾ ,

^Ey 0 ° , ^Ey 45 ° , ^Ey 90 ° , ^Ey 135 ° ,

^Hy 0 ° , ^Hy 45 ° , ^Hy 90 ° , ^Hy 135 ° ^]

C. Classification

Classification is one of the supervised learning approaches that learn from the input data and uses the model to classify the types of fruit. There are many types of classifiers. Among them, this study has utilized six different classifiers that are the most commonly used and better than other classifiers. That six classifiers are Naïve Bayes (NB), Logistic Regression (LR), Support Vector Machine (SVM), Decision Tree (DT), K-Nearest Neighbor (KNN), and Random Forest (RF).

a) Naïve Bayes (NB)

Naïve Bayes is one of the machine learning algorithms based on Bayes’ theorem to classify the class from the features. It is also called Gaussian Naïve Bayes because it works according to Gaussian distribution. It can be used to characterize a group of feature vectors of any number of dimensions with two values: a mean vector and a covariance matrix [11].

• b)    Logistic Regression (LR)

Logistic regression is the most famous multivariable algorithm that is one type of linear classifiers. It analyzes the interaction the labels and the features by using the logistic sigmoid function to predict the probability of event occurrence [14].

• c)    Support Vector Machine (SVM)

The structure of SVM classifier is complicated than other classifiers. SVM works by constructing the hyperplanes in a high dimensional feature that is used for the classifier and regression. SVM is effective for high dimensional spaces. The result of the system shows the low transparency. Accuracy depends on hyperplane selection [16].

• d)    Decision Tree (DT)

The decision tree is a popular machine learning algorithm that can be used for regression and classification problems. It is a faster computation frequency-based algorithm and also easily interprets to the rules. This algorithm solves the problems with tree representation [12]. In the tree representation, each leaf node represents a class label, each non-leaf node represents a feature, and each branch represents a decision rule [13]. The criteria of the decision tree are the same as the criteria of information gain. Decision tree does not need many preprocessing data and assumptions for the data distributions.

• e)    K-Nearest Neighbor (KNN)

K-nearest neighbor is one of the simple classification algorithms based on the similarity measure. KNN is worked as a non-parametric technique for statistical estimation and pattern recognition techniques [17]. This is very useful in predicting the result of regression and classification problems. This classifies the object from the neighbors by assigning the class [18]. For the classification phase, KNN assigned unlabeled query image with a label of its closest match by calculating the Euclidean distances between the query image features values and stored features of the trained images.

• f)    Random Forest (RF)

Random forest is an easy and user flexible classification algorithm. This randomly selects the sample data from the decision tree and it parallelly works for prediction [15]. It also performs by selecting the best prediction results from the voting process as an ensemble model. This is a highly accurate and robust method and it cannot suffer from the overfitting problem because it takes the average of all predictions. The random forest can also handle the missing values with two ways: using median values to replace continuous variables, and computing the proximity-weighted average of missing values.

• IV. Experiments

• A.    Dataset

  For the study, we recognize the five types of fruits: apple, cherry, grape, pear, and tomato. For the five types of fruits, the fruit images are extracted from the fruit360 dataset. This study uses the 25671 total images for five types of fruits. Table 1 lists the number of images for each type of fruits. Figure 6 shows some of the images for each class for the fruit.

Table 1. Number of Fruits for Each Class

Class No.	Fruit	Image Number
1	Apple	7946
2	Cherry	4592
3	Grape	4565
4	Pear	3914
5	Tomato	4654
	Total Images	25671

Class - 1 (Apple)

Class — 3 (Grape)

Class - 4 (Pear)

Fig.6. Sample Images for Each Class

• B.    Implementations and Results

  This study is used MATLAB 2018a to extract the different features from the image dataset and weka 3.9.0 to use the different classifiers. This study extracts the 7 different types of features to implement for feature recognition. The representation of features shows in Table 2.

These features are passed to five different classifiers. We calculate the accuracy of the system with 10-folds cross-validation. The results show in Table 3 by comparing different features on different classifiers. The performance of the proposed fusion method for fruit recognition system is mostly good than other feature extractions and other feature fusion methods.

The performance on the Naïve Bayes classifier is less than other classifiers but the accuracy is over 60%. The classification rates of the proposed method on Logistic Regression and Support Vector Machine are better than the rate of Naïve Bayes but not over 90%. Decision Tree’ result is over 90% but less than K-Nearest Neighbor and Random Forest. In Decision Tree, the recognition rate of the proposed method is the same with color feature-based recognition, color and texture feature-based recognition, and color, texture, and morphological feature-based recognition. The classification rate on Random Forest classifier achieves 99.8948 % accuracy. The proposed fusion method is the best on the Random Forest classifier, but the accuracy is a little less than color feature on K-Nearest Neighbor (KNN).

Table 2. Features Representation

Types of Features	Descriptions of Features	Feature Dimension
Color	Mean, standard deviation, skewness, and kurtosis	12
Texture	Statistical features extracted with GrayLevel Co-occurrence Matrix	16
Morphological	Mean, standard deviation, skewness, and kurtosis for boundary extracted image	4
	Color + Texture	28
	Texture + Morphological	20
Combined Features	Color + Texture + Morphological	32
	Color + Morphological (proposed method)	16

Table 3. Comparative Results with Different Features and Different Classifiers

Classifiers	Features
	CF	TF	MF	CF + TF	TF + MF	CF + TF + MF	CF + MF (proposed method)
NB	57.7422 %	21.8885 %	42.0669 %	48.4866 %	38.4831 %	56.5463 %	60.8274 %
LR	83.3703 %	32.453 %	46.6168 %	83.1678 %	49.7487 %	84.8896 %	85.1622 %
SVM	86.2958 %	31.8219 %	44.6223 %	85.8712 %	46.2234 %	87.0437 %	87.2619 %
DT	91.6715 %	32.4491 %	57.4695 %	91.6715 %	57.2942 %	91.6715 %	91.6715 %
KNN	99.6221 %	23.0182 %	51.3887 %	96.9265 %	44.7859 %	98.2315 %	99.6027 %
RF	99.8247 %	26415 %	58.2759 %	99.5793 %	60.6716 %	99.7935 %	99.8948 %

• V.    Conclusion

In this paper, we propose the feature fusion method for the fruit recognition system. Feature fusion method is fused the color features and morphological features extracted from the fruit image. Color features can precise the type of fruits and morphological features are used for the shape of fruits; and the combination of color and morphological features can more precise five types of fruits: apple, cherry, grape, pear, and tomato. We used different classifiers to recognize the types of fruits based on feature fusion. The recognition accuracy of color and morphological feature fusion method increases other individual features and other fusions on all classifiers. Among six different classifiers, random forest classifier achieves the best outstanding performances in this study that classified the five types of fruits by applying the proposed feature fusion method.

Acknowledgment

We would like to give our thanks and gratitude to people who have contributed towards the development of this manuscript from the planning stage to the finish stage.