Image Processing Method For Embedded Optical Peanut Sorting

Published Online November 2015 in MECS

Quality inspection of agricultural food products needs real time and continues optical operations in which under inspection objects are not stable but they are in motion. Continues motion is necessary because of large quantity of material inspection in limited time. For achieve this task with high accuracy, the optical method of sorting is sophisticated solution. The mechanical sorting does not give required accuracy level as optical method able to give. For such purpose the conveyor belt sorting method or gravity fall sorting method is used. Embedded systems are differs from computers, because of embedded systems are application specific computing. In contrast with traditional method of computer based image processing. As for optical image processing the objects travel small distance the optical standalone embedded image processing board makes disition that object is in acceptance quality or not. Peanut is one of the important agriculture food in world and having largest demand among the all nuts. As they grown in farms with having contain the outer shells and other unwanted material which other than peanut called foreign materials are mixed with peanuts batch. To sort good quality of peanuts out from the unwanted foreign materials and defective peanut needs to take measure the parameters of peanuts [10]. In optical method of sorting parameters can measure by taking an image from dedicated camera sensor and performing image processing operations on image to extract information from camera taken frame [15]. There is possible to develop such a system based on computer and software. But to make system low cost and operation faster, the embedded platform is used to perform image processing [11]. The two basic parameters that can differentiate peanuts based on quality are their colour and sizes. The image processing ability of embedded processors is very limited compare to computer based software analysis [1]. But they can implement for dedicated image processing operations and for lower resolution images. There are various embedded image processing options like FPGA, DSP and ARM RISC processors. FPGA based image processing is generally very difficult and hardware oriented task. The system development cycle of FPGA based image processing is very large. FPGA based sorting generally consider only one of the parameter colour or size. For measure size or height to width ratio the first step is to detect the edges by observing object with edge detection algorithm [9] and then apply further image processing operations. To design entire hardware dedicated to optical peanut sorting based on colour, area and height to width ratio on FPGA is time consuming and extremely difficult task compare to ARM based sorting algorithm development. DSP based image processing is good option[13] but, it having memory limitation for store images, lack of development libraries and higher cost makes it conditional option by frame rate vs. cost. DSP based image processing is capable to execute complex operations because of high speed signal processor hardware [7]. There are complex dual hardware options for implementation for multichannel image processing in real time like DSP and FPGA based combined hardware [12]. The RISC processors like ARM are widely used in embedded systems and they are capable to process large data such as images as a standalone embedded platform [2]. The Raspberry pi is ready hardware based on ARM BCM2835 processor. That can implement and develop any native Linux based applications. The image processing on ARM processor based raspberry pi is best for low cost solution for initial developments. The multiple peanuts must detect sorted in single frame of processing for increases sorting capacity as more number of peanuts sorting in less amount of time. As the processing power is directly depends to sorting capacity. To achieve that purpose the vibrating feeder have arrangement to feed the concurrent multiple peanuts with maintain equal distance between them and passing them on conveyor belt to the inspection zone. The particular channel actuator is activated when unwanted quality peanut runs from that particular channel. Most widely used actuator is pneumatic actuator, due to its fast operation and accurate target separation capability.

• II.    System Architecture

The belt feed sorting optical system having four subsystems. Belt feed vibrator and hopper used for store and equally feed the peanuts on conveyor belt. Second is conveyor belt system having smooth defect less white colored surface moving at constant speed open loop or controlled motor. Third is camera and image processing sub-system having the image capture and process image on embedded platform like ARM, which executes algorithm and measure image parameters for sorting peanuts. The camera used for project is Picam of colored 5 MP resolutions. The fourth system is pneumatics having decoder and driver circuits. The activation pulse is received from image processing board. The number of pneumatics uses in system is depends on channels and image processing capacity of multi peanuts per frame.

• A.    Belt Feed Vibrator and Hopper

The mechanical vibrator feed the peanuts from storing hopper to conveyor for equal distribute peanuts on belt surface. It maintains the flow of peanuts from hopper to belt. This arranges accurate distributed channel peanuts on image frame. So there is easy to measure the parameter of peanuts by using algorithm that applies on frame captured by camera. The maximum number of channels captured in single image is depends on camera spatial resolution. The number of rows scanning by camera is important for define number of channel capacity. The field of view (FOV) of camera is also important parameter for deciding the number of channels. According to standard peanut size the channel width define in between 0.9 cm to 1.2 cm. The distance between camera and conveyor belt is adjusted by perfectly peanuts detection and maximum channels allocation. If the number of pixels per peanut are more then there is easy to algorithm to detect the sharp edges from background. The accuracy of derived data is higher when number of pixels covered by single peanut is higher.

• B.    Optics and Camera System

  The X-ray, visible and infrared band of electromagnetic spectrum is used for detect the under inspection object parameters depends on object type. The separation of peanut needs analyzed the size, color and wide spectrum response properties. Agriculture research for peanut having analyzed that 400nm to 1100nm spectrum response is able to take analyzed the most parameters of peanuts [3]. This band acquires full visible band and some part of infrared band. Some sorting systems directly analyze the spectrum response properties of object. The infrared band response of sensors can able to detect the internal defect of peanuts, whether or not the outside R G B color response is in the acceptance range. The camera in system can able to take large spectrum images having equal sensitivity with ideal constant gain. The peanut respond on wavelength of wide spectrum images more accurate and used for differentiate defective peanuts with high aflatoxin contents. The color sorter use only visible spectrum camera of red, green and blue color pixel sensors. The raspberry pi camera module is used for taking the frames [4]. These frames are in appropriate form and can further process. Optics provides the perfect focus and proper field of view so the image of peanuts from different channels can able to takes sharp and high contrast images. It is necessary to set focus of optics lenses matches with predefine algorithm program parameters; either it will give wrong results. The distance from camera lens to fixed focus peanuts is 45 mm. The algorithm is based on color of peanut, location of peanut inside image and the area covered in image frame. So this information able to makes precise disition of which pneumatics is to be triggered.

• C.    Image Processing System

  The system is independent of large data computation PC and dedicated image processing software for manipulate images. The algorithm is implements on standalone embedded system of raspberry pi having RISC ARM processor of BCM 2835 to process images and detects the foreign material or defective peanuts to differentiate from good quality peanuts. The BCM 2835 ARM architecture processor works at 700 MHz of clock frequency used in raspberry pi to process images and extracting features. The clock frequency is very much important parameter that affects the real-time criteria. It also depends on architecture that how fast the integer and floating point operation with different clock frequencies. The algorithm executes much faster speed if the Raspberry pi set to turbo 1 GHz clock frequency mode. In

  turbo mode the integer data is processed in half of the time then the normal 700 MHz operating mode. The important to note that the life time of ARM BCM2835 is decreases when it programmed for turbo 1GHz frequency mode due to the high dynamic power dissipation. Raspberry pi board having all interfaces necessary for image  processing  and  having  large GPIO and

communication protocols which is very helpful to develop algorithms to be implementing for any peripheral device. The capabilities to process image on raspberry pi can consider as same type capability of other 32-bit ARM processors. The same optical sorting algorithms can also apply to iMX6 series ARM processors as independent image processing platform, but the compilation process and hardware oriented files are different. The raspberry pi board having image and video capturing peripheral of fully compatible camera hardware called PiCam. This is 5 MP of PiCam can directly interface with the raspberry pi hardware. The camera driver is same as most Linux used with called UV4L compatibilities provided to capture 30 frames per second images in real time. To make camera system software compatible to intermediate driver supporting facility with high level functional library such as OpenCV needs linking programs to provides real time communication with resource accessibility. The dedicated programs are used for ease of compatibility and accessing of camera resource. This is analyzed during algorithm development and during programming that frame rate to process image is degrade as more and more complex algorithms implemented. So the frame rate directly proposals to the number of processes apply to manipulate images for purpose of extracting features inside image specially as per case here peanut as object parameters extractions. The one other important parameter analyzed during development process which affects the frame rate is resolution. Resolution of image is in terms of both as spatial resolution and pixel resolution. As spatial resolution increased the system takes more time to process the 8-bit integer data for dedicated each pixel values. For pixel intensity the standard value used is 8-bit image capture. But the OpenCV can able to process up to the 16-bit or 32-bit images for more accurate results. The pixel intensity is not very much affects the frame rate, but It practically importance as nano seconds time is matters. It depends on ARM hardware architecture that can able to process data 8-bit, 16-bit or 32-bit concurrently processed.

• D.    Pneumatics System

Pneumatics acts as important actuating part to apply pressure force to make defective peanuts slightly move towards from its natural gravitational trajectory. The pneumatics system has signals receiving from image processing board of raspberry pi, amplify to it and make the air pressure valve release for very little amount of time. The system having multiple pneumatics as long as channels ,so to eject air from which one of pneumatic is must decide by processor to separate that particular channel defective peanut. The synchronization between falling peanut with image and signal processing system must be necessary. It can calculate by the time analysis starts from image taken by camera to pneumatic air generated. This time cannot always constant. Due to the time variations are depends on number of peanuts in image, the percentage concentration of defective peanuts in batch, the real time operating system type as task priority scheduled and variation in signal time to make pneumatics activate. For generate air pressure the air compressor can be used. The capacity of compressor depends on number of channels decided during the design. The response time of pneumatics is practically high. The newest technology of multiple pneumatics can give less than 5 ms of response.

• III. Image Processing Embedded Hardware

• A.    Raspberry pi

The Raspberry pi is very little ARM platform based small embedded hardware which fully compatible with the native linux applications and compatible camera hardware and camera software drivers [14]. The Raspberry pi model B+ having all feature as for develop image processing algorithms on native Linux platform. It is very low cost ARM architecture based hardware that can able to supports total 6 optional operating systems. The OS of raspbian is specially optimized for particular ARM Broadcom processor platform and support all most all features as any other Linux distributions are provides. The all image processing algorithm is developed on raspbian operating system on raspberry pi. The reason why to select raspberry pi hardware instead to other embedded low cost boards like UDOO, beaglebone black, or panda board is that raspberry pi having full compatible, configurable and programmable camera [5] for process real time image for sorting. The model B+ of raspberry pi having 26 programmable GPIOs and peanut rejection pulse generate to pneumatic driver circuit by them using GPIO accessible libraries from main program. The OpenCV library on raspberry pi can build as cross platform. But due to cross architecture platform the building process is very slow and not all features of OpenCV can accessible on the raspberry pi hardware. The frames from CSI pi camera are taken during the execution of program and takes real time images from camera. The camera driver of UV4L is used to take images from camera in real time to write in specific format in buffers as define in driver configuration file. The 512 MB of RAM provides good support for images to be store in intermediate operations during the executions. The micro SD card can able to store permanent data as well as operating system image. The 8 GB of micro SD card is enough to store the libraries and other necessary set of tools required for image processing with OS image.

• B.    PiCamera

For taking the images in real time the frame rate of camera is very important. The PiCam is fully compatible with raspberry pi with camera serial interface. The PiCam can able to give up to 30 frames per second images capture in video mode with full HD resolution. There are other choices for the USB cameras but for developing real time algorithms the pi cam is suitable choice as per resolution and frame rates. PiCam can compatible with the Linux driver UV4L having make changes for compatible with raspbian and available as open source. To make appropriate changes the parameters of input frame from PiCam the driver configuration file have need to make changes as per the image processing applications requirements. Some important parameters like contrast, brightness, height and width of image can change by making changes in configuration file. The camera is fixed in focus so the peanuts are placed at fixed distance from camera lance for all algorithms of four channels peanut. So the algorithm is capable to detect defective peanut or peanuts in single image contain at least four peanuts. Frame resolution is set to 640 x 480 to take four peanuts to be regularly arranged in single frame.

• IV.    Software and Tools

The Image processing operations required large data to be access. To process large data needs more amount of time. The software tool selection and programming language is highly affects the capability of optimization. Because of each language, compiler and linker generates binary execution file with different ways and methods. Some of the methods are optimized for target processor architecture at certain level. So the selection of software tools is important for hard real time criteria and memory optimization especially for image processing algorithms.

• A.    Raspberry pi with OpenCV

The raspberry pi with OpenCV is compatible with both C/C++ and python language. Compare to both of them the python is relatively very easy to develop algorithms. But python is kind of scripting language have use interpreter to generate binary executable files it is slower in execution than the C/C++ execution files. C/C++ languages uses compiler to generate object files and linker to link files to combine and to generate execution file. For to make time domain optimized algorithm, the C/C++ language is used for programming with image processing. The most recent version of OpenCV has most of all API (Application programming Interface) and functions that can access by pure C++ language. So the class and object philosophy is apply to image as set of data types. The image intermediate results are to be stored using “MAT” class of the OpenCV. For real time images capture in series the “VideoCapture” class is used.

• B.    OpenCV Image Processing Library

Raspberry pi ARM processor supports the OpenCV libraries which were originally optimized on Intel architecture. Most of OpenCV functions are worked fine but some high level functions are not work properly. The time to build OpenCV on raspberry pi takes 8 to 9 hours because of same reason of cross architecture. OpenCV version 2.4.10 is used for developing algorithm. Once

OpenCV successfully build on raspberry pi, we can access into main program. There are some settings of tools and compilers needed to compile main application program files with OpenCV libraries. Complete procedure to install OpenCV and all necessary tools to install OpenCV is available at [6]. To make the PiCam camera compatible with the OpenCV APIs there must need the intermediate libraries, so the frames transferred by camera can understand by OpenCV in appropriate formats. Those libraries works closed with camera drivers and give data to application program files. So the peanut sorting algorithm develops using OpenCV in C/C++ language with intermediate supporting libraries. Raspberry pi has not permanent memory on board, so the intermediate results can newer store permanently in RAM. If we want to store image permanently the OpenCV functions can use to write the image in SD card.

• C.    Software tools for development

Editor

For user interface to write the code must need text editor. For linux the most widely used editor is nano which is very simple and light weighted editor that can access from terminal provides the variety in text color for different code type for easy to see the code during the programming. The nano is fully open source and generally prebuilt with raspberry pi board. There is some other options like jed, emacs, leafpad and vim. Among them leafpad is also prebuilt on raspberry pi. The nano editor with C or C++ file allows the C or C++ keyword specific text color makes user convenient for easy and fast way to identify key words during the programming. The most convenient way to use nano editor is from terminal by filename followed by keyword nano.

Compilers

The compiler used to translate the source file of programming to generate the object file for further processing of generate executable file. For purpose of image processing with the OpenCV the best compiler is GCC which is GNU C compiler having open source accessibility. As the latest OpenCV version is fully compatible with the raspberry pi the C++ language is favorable. The g++ compiler used as favorable for latest version of OpenCV 2.X and after. The g ++ can able to compile both .c and.cpp files but it will all be treated as C++ files.

Cmake

CMake is cross-platform, open-source make system facility for easy compile programs. CMake is used to control the software compilation process using simple platform and compiler independent configuration files. CMake generates native makefiles and workspaces that can be used in the compiler environment of choice. It is used to set pre-processor generation, code generation and provide facility to set environments necessary for compiling and generate binary executable files.

UV4L driver

The driver is available on raspbian for raspberry pi CSI camera from native Linux. It is develop for user space Video4Linux2 - compliant drivers for real or virtual video input devices. While the Read-Write I/O interface is embedded in the core, the UV4L framework comes with an optional extension library, libuv4lext, which enables the Streaming I/O interface as described by the Video4Linux2 specifications. Most of the Video4Linux2 applications only support the Streaming I/O method.

• V. Related Work

For the optical peanut sorting based on RGB color, covered area, contour length and height to width ratio of peanuts the image processing algorithm developed in C++ language with native linux operating platform. This algorithm developed on embedded platform having interfaced with camera to take frames at 30 frames per second and 640x480 of image resolution described with the mjpeg encoding format. The Fig.1 shows the flow chart of algorithm. The program execution starts with taking the image from camera. For continues frame monitoring the OpenCV video capture class is used. This frame is getting converted to OpenCV class object. Then taking clone for the same frame as OpenCV Mat class object variable. For apply threshold Operation the pure binary thresholding operation is used due to the high contrast background. The value of threshold can defer for different background light brightness and incidence light intensity. So the threshold value is selected based on user define input. As image format used is 8 bit the observed value of threshold is in range of 75 to 90. This value is accepted from user in percentage value. The normal room condition percentage threshold value is 30% to 35% applies. The value of threshold is calculated by using equation (1), which is applied on original image and give the destination image as shown in Fig. 1.

• 255,    if I S (x,y) > Th

I D (x,y) =                                         (1)

0, otherwise

Fig.1. Threshold Operation at Th = 80

In next stage this threshold frame apply to edge detection [6] method for finding edges from peanut to its bright background. Canny edge detect is more accurate and generate only one bright connected pixels at the edge. Sobel edge detect is same as canny edge detect except the canny initially apply the Gaussian filter to remove the noise. For the edge detect the mask window for convolution in x and y direction is applied as equation (2) and (3).

Then calculate the gradient magnitude as define in equation (4). The direction is rounded with the possible angle 0 to 135. Then non maximum suppuration is applied as equation (5).

This will results in hysteresis of edges are highlighted with zero background value as shown in Fig.2.

Fig.2. Canny Edge Detect Apply to Threshold Image

These edges are made from the connected components of individual pixels. If there is single pixel is missing out from the shape that make connected, the next operation of image contour detection is unable to detect the peanuts shape. This is the rezone why to select canny edge detect. Based on canny image closed contours are formed by using the contours property of OpenCV library. The current algorithm is define for the four concurrent peanut quality analysis, but this can expand up to 12 to 14 channels by make change in algorithm program. So for this algorithm i=4, the number of i contours are to be analyzed per frame as number of channels defined. For four channels the frame having four connected contour. To convert the edge detected image to contour detecting, OpenCV functions and vector properties are used.

Fig.3. Contours Detect and Finding Contour Centers

data of particular channel contour fair area, contour linear length and height to width ratio is to be calculated.

The third operation on contour image is to fill the contours as opposite to its background color that is pure white to black background. This image can acts as mask on original frame which initially captured by camera. The filling the contour with the highest intensity value and then overlap operation with original source image will generate the image as shown in Fig. 5. The masking operation gives the only filled colored peanut with the complete black background in resultant image.

Fig.4. Algorithm for Optical Peanut Sorting

Contour property of OpenCV is very power full tool to execute the shape and size detect algorithm. This property is very versatile and applies to any random shape in image. Fig. 3 shows that the different random color is applied to enhancing the contour edge and to differentiate all contours with each other. This contour image is applied for three different paths. The first is analytically calculating the each contour center. So based on that can decide the column location of each contour. This algorithm accepts the 640x480 image means the image having the 640 columns. For divide those in four different channels. For detect which contour is lie in which channel the 0 to 160 column is considering as channel 1. The 161 to 320 column number is dedicated to the channel 2. Same way the column number 321 to 480 and 481 to 640 are for channel 3 and channel 4 respectively. The disition is based on the contour centers.

Second path of operation is based on these contours to find the contour length and covered area by analytical pixel count methods, which directly avail by OpenCV APIs. The other important parameter of contour property is ratio of height to width. This can finds by maximum and minimum column and row pixel points. Based on this

Fig.5. Overlapping with Source Image

This image is now convenient to measure each peanut color values. The average color value of each contour color can calculate by adding intensity value for each pixel R G B value and divide total number of pixels covered inside peanut marked contour. But for higher color sorting accuracy, the independent R G B value based calculation is applied to sorting algorithm. Based on the same contour property of OpenCV the area, linear length and height to width ratio parameter measured for each channel. The calculation of these six parameters of each channel peanut can be calibrated from image in calibration mode of operation. The process of calibration is to take the reference frame for measure all parameter values and then consider these data as reference for sorting the peanuts. For implement the process of calibration, the algorithm is capable to mathematically calculate the parameter values. Suppose the calibrated data for peanut color R, G, B value is R C , G C and B C respectively. Consider the area, linear length and height to width ratio of calibrated image peanut parameter value is A C , L C , RA C respectively. Now the user wants to give sorting accuracy according to all these parameter values and percentage tolerance input. The percentage tolerance for each parameter is necessary because the peanut quality definition is not always remaining same. If the percentage input value by user for color parameter is C_P, the size parameter is S_P and height to width ratio parameter is R P . Then the condition of acceptance for any peanut is that all values must falls in user define percentage input and calibrated data range. So the maximum and minimum allowed values for peanut parameters for acceptance condition is define by,

Average red value in peanut,

Rmax= RC + RC[CP/100](6)

Rmin= RC - RC[CP/100](7)

Average green value in peanut,

Gmax=GC + GC [CP/100](8)

Gmin =GC - GC [CP/100](9)

Average blue value in peanut,

Bmax=BC + BC [CP/100](10)

Bmin =BC – BC [CP/100](11)

Peanut contour area,

Amax=AC +AC [SP/100](12)

Amin=AC - AC [SP/100](13)

Linear contour length,

Lmax=LC + LC [SP/100](14)

Lmin= LC - LC [SP/100](15)

Height to width ratio of peanut,

RAmax=RAC + RAC [RP/100](16)

RAmin= RAC - RAC [RP/100](17)

Fig.6. Images at Each Stage of Algorithm

The equation (6) and (7) gives maximum and minimum allowed R value for sorting. The generally analyzed value of calibrated R C is 80 to 85 under normal luminance condition. For calibrated G C value is in between 45 to 60. The calibrated B_C value is 22 to 30 for ordinary peanut. As the comparative size of peanut is increases the peanut area and linear length value is also increases. For ordinary peanut the calibrated area value A_C is 6800 to 8000 pixels for 45mm distance between the camera lens and peanut. The calibrated height to width ratio RA C of ordinary peanut is measure 1.24 to 1.80

values. Based on calibrated data the calculation of maximum and minimum acceptance value is measured using equation (6) to (17). The user gives higher percentage calibration value, so that probability of peanut acceptance is higher. The values of parameters are more near as under inspected peanut is more similar to ordinary calibrated peanut.

• VI. Conclusion

For optical sorting on standalone embedded platform there are varies options of ARM, DSP and FPGA. Among them ARM is low cost and fast developing solution. For multi channel peanut sorting on raspberry pi ARM platform, the algorithm is able to sort the defective peanut based on parameter values of color, size, edge length and area of peanut based on user define criteria. To implement this system the ARM raspberry pi board is used and develop algorithm with native linux raspbian OS. Developed algorithm is able to detect and sort the foreign materials as well as non-accepted quality defective peanuts. In optimized mode the algorithm executes rate of 2 to 3 frames per second. According to rate optical sorting capacity is about 43.2 Kg/hr is achieved.

Acknowledgment

This project consumed huge amount of work, research and dedications. Still, implementation would not have been possible if we did not have a support of many individuals and organizations.The research work is supported by vision embesoft solution for providing all the necessary resources from start to completion of project. Thanks to Mr. Bhavesh Pethani and Prof. Arjav Bavarva for providing the guidance during the project. I also thanks for Mr. Amrut Kothiya and Mr. Vijay Bharda for gives me direction for software implementation and solving troubles. Thanks to university staff and library department. I would like to express my sincere thanks to OpenCV developers and contributors, who make such a thing and contribute free for developers.