Text Deblurring Using OCR Word Confidence

A lot of text information can be extracted from the images of the text documents. Images of the text document can be taken either with the help of a scanner or with the help of a portable camera or smart phone camera. A lot of research is done on the image acquired by portable camera for efficient optical character recognition. But the image acquired using a camera is affected with problem of blur, skew and variation in the lighting conditions. In this paper we are dealing with the problem of blurred textual image that too affected with a uniform motion blur.

Text Deblurring is image restoration problem in which the image of the text document taken form a camera is blurred or degraded due to motion of the camera or the object at the time of image capture then to estimate the Blur kernel and convert the blurred image to a Deblurred image or restored Image using Deconvolution operation. Blur is caused by camera sensor motion which is represented by a blur kernel. Theoretically, the motion blur process is modeled as the convolution of the Image and a blur kernel with additive noise.

д(х,у) = f(x,y) * Н(х,у) + п(х, у)      (1)

Where д (х, у) is the Observed blurred Image f (х,у) is the Non-blurred Original image and Н(х,у) is the blur kernel also known as Point Spread Function and n (х, у) is the additive noise * is the convolution operator in equation (1). Therefore, motion Deblurring tries to estimate the blur kernel and then it performs the Deconvolution operation to recovers Original image also known as latent Image from the blurred image. The Deconvolution operation is the inverse of convolution operation used in the Blurring process.

• A.    Model of Image Degradation and Restoration

The degraded image is ( , ) is obtained by applying the degradation operation over the input image ( ,)

along with the additive noise ( ,)

g (х,у) = Н [f (х,у)] + n (х, у)(2)

The objective of Deblurring [25] is to recover f(х,у') form the degraded observed image ( , ) using the estimated value of . Here can be linear or non-linear. Mostly it is assumed to be linear and satisfies the linear property of homogeneity and superposition. The degradation operator H is also considered to be space invariant or position invariant that is its response at any point depends on the value at that point but not on the position of the point and is defined mathematically as

Н [f (х-а,у-Ъ )]=д(х-а,у-Ъ )       (3)

For all f(x,y) and any a and b. The overall model of degradation and restoration operations is shown in Fig. 1. Here Latent image f'(x,y) in the figure is the restored image after the Deconvolution operation which is the inverse of the Convolution operation which has degraded the Input Image. Deblurring is almost irreversible due to additional noise which gets amplified at the time of Deconvolution operation. So Latent Image f ' (x,y) cannot be the exact same as Input Image f(x,y) it is the restored version of the Input Image.

Fig.1. Overall degradation and restoration Model

Impulse function f(x,y) is expressed as,

a(x,y)

n(x,y)

— b)dadb] +

This expression is called the convolution integral in the continuous variable.

• B.    Discrete Convolution model for Degraded Image

Generally Image can be constructed in two ways either discrete or continuous. The continuous image can be converted to a discrete image by sampling, quantization and coding. The discrete model of the degraded image caused by the blur and the additive noise added can be expressed as,

g(x,y) = z™ ^J (a, b)h(x, y, a, b)+ n(x,y) (9)

Where f(x,y) represents the original image of size m x n and h(x,y) represents the PSF of size p x q. In the above equation n (x, y) is taken as additive noise introduced by the system and is assumed to be zero mean white Gaussian noise.

Using spatial invariant property of PSF, the g (x, y) can be described as,

f (x,y} = /*”L+”f( a, b )5(x-a,y-b )d adb  (4)

Substituting (4) in (2) we get the blurred image g (x, y) as

g (x,y) = Hl J J f (a, b)5(x-

n (x,y)

a ,y - b)da db ] +

As f(a, b) is independent of x and y, using Linearity, the g (x, y) can be expressed as,

g (x,y)

J f(a, b)h(x, a, y, b)dadb

n(x,y)

I

+

g(x,y) = ^!™₌ f !^ i f(a , b)h(x - a,y - b ) +n(x,y) (10)

g(x,y} = f(x,y) * h(x,y) + n(x,y)     (11)

Where * denote the two dimensional linear convolution.

C. Motion Blur Model

Motion blur in the image is caused due to either the motion of the image capturing device or the object to be captured. Image capturing device can be a scanner or portable camera. Motion Blur is governed by two parameter that is the length of motion ( ) and the angle of motion (  ). When the text document to be captured translates with a relative velocity V in respect to the camera, the blur length L in pixels is L = VTexposure where, Texposure is the time duration of the exposure. The expression for motion blur is given as,

Where h(x,a,y,b) = H[5(x - a,y - b)] is called point spread function (PSF) in the optics Since H is spatial invariant

So the expression of H can be rewritten as,

H = [5(x - a, у - b)] = h(x - a,y - b)      (7)

And the blurred image is given as,

(1 L ’ if 0 < M < Lcos6 ;y = L sin в h( , ) = l 0, о therwise When the angle of blur 9 = 0, it is called horizontal motion blur. Point spread function can be represented in discrete as,

ℎ(т,и,L)

⎪⎧     1,       т=0, |П|≤|( 2

=⎪⎪  21 { (1-1)-2 | " -21|}, ifm=0,

_ гL-1 ⎪⎪⎪⎪                                   | ” |≤⌈ -21

⎩       0, elsew ℎ ere

• D.    Noise Model for Gaussian Noise

Image capture and transmission is also affected with the additive noise which is induced to the image. Basically the most common types of noise are impulsive and Gaussian noise, which affect the image at the time of capturing due to the noisy sensors which induce the noise in the image. As the image has to be transmitted through different channel it gets affected by the noise due to channel error. There are many noise models in our study we will be dealing with the Gaussian noise which is the most common in practical applications. Gaussian noise is a random variable and is expressed as,

^•даиззшп ⁽ X ^, У )=    exp(- )       (12)

It is characterized by its variance О"2 . The noisy image 7g (X,У ) is the addition of the original image f(X,У ) with the noise term. It is given as, fc (X,У)=/(X,У)+Tlgaussian (X,У)        (13)

• E.    Deblurring using Blind Deconvolution operation

The problem of Image restoration after the blur is known as Deblurring. In Deblurring the original image /( X , У ) requires the deconvolution of the PSF ℎ( х , У ) with the observed image 9 ( х , У ). In most of the cases the PSF is known prior to the deblurring this type of deblurring is done using the classical well known techniques such as inverse filtering, wiener filtering , least square filtering, recursive Kalman filtering is available. PSF is unknown in various applications and very little information is available about the original image. To recover the original image f ( х , У ) from the observed image 9 ( х , У ) using partial or no information about the blurring process this phenomenon is known as Blind Deconvolution.

Blind Deconvolution algorithm can be broadly classified in two types. In first type PSF is identified first and then utilized to deblur the image using any of the classical Deblurring techniques. In second type algorithm we estimate the PSF and restore the image simultaneously. A number of methods exist to remove the blur from the observed image 9 ( х , У ) using a linear filter. The restored image f ( х , У ) from a given blurred image is given by

/ ( X , У )= д ( X , у )∗ℎ( X , У ) (14)

Where * denotes the Deconvolution which represents the inverse of the convolution. In the frequency domain, this can be expressed as

/ (и,v)=д(и,V)Н(и,V) (15)

Where f ( и ,V) denotes the estimated image in spectral domain. 9 ( и ,v) And Н ( и ,V) are the blurred image and PSF in frequency domain respectively.

Motion Deblurring with single-input image is more complicated than that with two-or-more-input images because multiple blurred images always provide more information in solving the problem. In this paper, we mainly focus on Blind image Deconvolution in which we try to estimate the PSF Н ( X , У ) and then we perform the Deconvolution operation of this PSF with the blurred Image 9 ( х , У ) to get the Original Image f ( х , У ) . As deblurring is ill poised problem is difficult to recover the original image due to additive noise. In the process of motion blur, the Deblurred image loses much high-frequency information. The traditional methods always give undesirable Deblurring results because of the effect of the additive noise on the single image based motion Deblurring.

• II.    Related Work

Deblurring Text image is a problem which is being researched due to its wide application in optical character recognition. We know that the recognition of a blurred text is still a big problem for the most efficient optical character Recognition (OCR) engine. Text Deblurring can increase the efficiency of OCR. Hence various Deblurring techniques have been proposed so far are either based on blur kernel estimation also known as blind Deconvolution method and non-blind Deconvolution method. In blind convolution method no information about the blur kernel is known. In non-blind Deconvolution method we have some knowledge about the blur kernel.

Blur can be estimated by the estimation of the blur kernel or the PSF Н(х,У) . PSF for the motion blur depends on the pixel length of the blur (L) and the angle of the blur(6). Based on these two parameters we can estimate the blur kernel and use this kernel for deblurring process. Blur Kernel estimation is iterative process in which we have to find the blur kernel which can help us to restore the original imagef(X,У). But as we know that after deblurring we cannot get the exact Deblurred image as the original image due to the additive noise п(х,У) which is induced due to the deconvolution process. So we try to get the image that much deblurred so that we can extract the desired information in our case the text information. Various kernel estimation technique have been proposed for the estimation of the motion blur kernel for Deblurring in which Taeg Sang Cho and Sylvain Paris [1] proposed a method to estimate blur caused by the camera shakes using edge analysis and by constructing Randon Transform of the blur Kernel. Shamik Tiwari [2] in his paper compared the entire blind Deconvolution algorithm to estimate the PSF and then estimated motion blur parameter can be used in a standard non blind Deconvolution algorithm. Lu Fang [4] proposed method by decomposing a blur kernel into three individual descriptors trajectory, intensity and point spread function. So that it can be optimized separately. Nimali RajaKaruna [5] proposed a method for Deblurring for visually impaired people it used a 3-axis accelerometers and gyroscopes of the smart phone camera used for image capture to get the motion vector and the heuristics method is developed to determine the optimal motion vector to Deblur. Jing Wang [6] proposed a blind motion Deblurring approach that works by kernel optimization using edge mask is used. An alternative iterative method is introduced to perform kernel optimization under a multiscale scheme. Total-variation based algorithm is proposed to recover the latent image via non-blind Deconvolution. Long Mai [7] addressed the problems of fussing multiple kernels estimated using different methods into a more accurate one that give better Deblurring result. Jinshan Pan [8] proposed Lo-regularized prior based on intensity and gradient for Deblurring using distinct property of the text image.

Deblurring can also be performed using non-blind Deconvolution technique in which we have more than one blurred image of the input. We also have some knowledge about the PSF. So it helps us in the estimation of the blur Kernel which can then be used to get the latent Image. Another solution to Deblurring is through the neural networks in this approach we train a neural network using a back propagation algorithm [13] in which we require both the ideal image as well as the blurred image of the ideal image. We train the neural network with blur image and ideal image pairs to find the relationship between the blur Image to the ideal Image.

Convolution Neural Network (CNN) [20] is also one of the solutions for Deblurring using blind Deconvolution method. CNN is based on the concept of deep learning and is powerful machine learning technique [21]. CNN is trained using large collections of diverse images. From diverse image it can learn rich features representations. We can also use a pre-trained CNN. It is a multi layered structure which can help in extracting different features in each layer. In Deblurring the blurred images are used to input and they are mapped into its corresponding ideal image to learn various features relationship between blurred and ideal image in the training phase. Thus after training lots of diverse blurred image into its ideal image. The CNN can then deblur an input image based on the learning from the trained samples.

• III.    Proposed Approach

In our proposed approach we have followed the blind Deconvolution method for the estimation of the blur kernel or the PSF. As we know that Motion blur is caused by motion of the camera at the time of the image acquisition. So the PSF of motion blur is a function of the length (L) and angle of blur (9).

So we tried an iterative method to estimate the length (L) and angle of blur ( 9 ). For each PSF estimated with a length and the angle iteratively, we check its Optical character Recognition [23] result simultaneously by finding the cumulative sum of the confidence of each word recognized and then calculating the Average Word Confidence (AWC) metric by dividing the cumulative sum of word confidence by number of words.

[£*=oC^ord onnf idence of each wo rd)] n

Where n is the total number of words in the text obtained after OCR result. We keep the value of the length (L ) and angle(9) of the blur kernel along with the corresponding deblurred image OCR Average Word Confidence AWC in a two dimensional array. Then we find the value of the length (L_m) and angle of blur ( 9) corresponding to the maximum Average Word Confidence ( ) result of the OCR. Hence this value of the blur length and angle is the value at which the Deblurring or Deconvolution method gives the optimal text recognition result using Average Word Confidence metrics of OCR. Thus blur kernel or PSF is estimated using the OCR Average Word Confidence metrics iteratively which gives us a criteria on which we can say that this PSF is the optimal for Deconvolution using any filter. Thus Average Word Confidence for the best recognition will be the highest and the length and the angle corresponding to it will be optimized PSF to get the latent or original image with higher text recognition rate than other PSF.

• A.    Estimating Point Stread Function (PSF) for motion blur

Motion blur is caused by the motion of the camera at the time of image capture. Point spread function of the motion blur depends on the length ( ) and angle of blur ( ) . Estimation of PSF is blind Deconvolution problem. In our approach we have tried to find out the length of blur ( ) for various images and we have estimated the length of the blur to be between 0 to 25 pixels based on our practical requirements. This value can be increased or decreased based on the requirement. We have taken the value of the angle for a blur ( ) in between 0 to 180 degree counter clockwise. As the value of is taken in the broad range it can easily estimate the angle in the textual input image.

This means if length of blur L = 9 and angle is 9 =0 degree then the blurred Observed image g (x, y) will a translated image of Original image f(x,y) by 9 pixels. We have iteratively created the blur kernel or PSF for the different value of length ( ) and angle of blur ( ) .

For each estimated PSF we apply Deconvolution operation which is the inverse of convolution operation with the blurred image ( , ) to obtain a latent image ( , ) . From the latent image obtained we perform the OCR operation to find the value of word confidence of each word.

• B.    OCR Evalution of the latent Image

OCR is the image processing technology which helps to recognize the text in the image. OCR operation requires to converts true color or grayscale input images to a binary image which contains only two values that is 0 which represents black pixel or 1 which represent white pixel, before the recognition process. It uses the Otsu's thresholding technique [22] for the conversion. Then segmentation of text and non-text is the next step for the OCR. Obtained text is segmented into line and then individual words. Words are recognized and a text file containing the recognized word is the output of OCR.

In OCR evaluation of the latent Image obtained after the Deconvolution operation of the Blurred Input Image and the estimated PSF. We apply the OCR on the Latent Image and the from the OCR result we find the value of Word Confidence of the recognized words that indicates the confidence level with which the word is recognized.

Word Confidence is normalized value between 0 and 1.

The Word Confidence of Individual word is calculated and then the Average Word Confidence (ЛWC) is calculated which the mathematical average of the all the Word Confidence obtained. Then this Л W C value will be used as a metrics for the estimation of the Optimal PSF for the Deblurring Process.

• C.    OCR Word Confidence Metrics

We have used Word Confidence attribute of the OCR result. Word Confidence is determined based on character level confidence. Character Confidence gives the normalized value of how effectively the character is recognized. Better the Character Confidence of recognition better is the Word Confidence of Recognition. In addition the Word Confidence is affected by the dictionary based verification. If a word is found in the dictionary, it increases the Word Confidence value of that word. The longer the word, the higher will be the confidence value if it is found in the dictionary. For example if a long word of around 15 characters is found in dictionary it is pretty sure that the word is correct and will yield a higher word confidence, while on wrongly detected character a match against the dictionary by mistake is unlikely to occur. Short words like 'add' or 'odd' will both be found in dictionary. Therefore for smaller words there is a probability that we can get the dictionary match. Hence to overcome this problem words with 2 or less characters are not checked against the dictionary. The word confidence is normalized to an interval of 0.00 to 1.00 where 1.00 is the best and 0.00 is the worst word confidence. This OCR word Confidence metrics is used for the evaluation of the Deblurring result for the estimation of the Blur Kernel or the PSF.

Algorithm of the proposed approach

Step 1: Input a blurred imageg (x, y).

Step 2: Repeat step 3 to 8 for length (L ) =1 to 25 pixels

Step 3: Repeat step 4 to 8 for angle ( 9 ) =0 to 180 degree

Step 4: Create a blur kernel of motion blur using length and angle as ( ,  ).

Step 5: Apply Deconvolution operation between g (x , y) and H^x , y) to get the latent image/ (x , y).

Step 6: Apply OCR on latent image f (x,y) obtained in step 5.

Step 7: Find the sum of word confidence of every word in the latent image f (x, y) then find Average word confidence Л WC by dividing sum of word confidence by number of words n.

^^^  E«=o(Word confidence of each wo rdf]

n

Step 8: Store the value of length (L ), angle ( 9 ) and ЛИС obtained in a 2D-array R or a table.

[End of angle for loop]

[End of length for loop]

Step 9: Find the value of length L_m and angle corresponding to largest value of Average word confidence        .

Step 10: Return length and angle     obtained in step 9.

Step 11: END.

Fig.2 Blurred Input Image g (x,y)

Fig.3. Latent Image f(x, y) with AWC= 0.508564115 for PSF with L=17 and 8= 10.

Fig.4. Latent Image f(x, y) with AWC= 0.756988287 for PSF with L=20 and 8= 14.

Fig.5. Latent Image f(x,y) with AWC= 0.621142268 for PSF with L=25 and 8 = 13.

Table 1. Results of the Average Word Confidence Metrics

Top 20 results of Largest Average Word Confidence for Estimated PSF
Estimated PSF for the Blur		Average Word Confidence ( AWC )
Length ( L )	Angle ( e )
20	14	0.756988287
21	14	0.745501578
21	13	0.739337564
19	12	0.734167099
20	13	0.725162864
22	14	0.714256167
19	13	0.713874578
21	12	0.713807285
20	12	0.712606966
22	12	0.706631124
25	14	0.698771179
23	14	0.696232617
21	11	0.691197157
19	11	0.688717008
20	11	0.678997517
25	8	0.67813015
22	11	0.653479457
19	10	0.640951991
24	14	0.635393023
23	11	0.632594824

• IV.  Result Analysis

In the proposed work we have estimated the PSF for the blurred image 9 ( х , У ) using blind deconvolution method iteratively by calculating the Average Word Confidence AWC metrics that help us to estimate the optimal PSF. For every Estimated PSF value we apply the  Deconvolution  operation between the Blurred

Image 9 ( х , У ) and the PSF to get the latent image f ′( х , У ) then result of OCR Average Word Confidence AWC is calculated. For latent Image highest Word Confidence means that its word recognition rate is higher than the others. Hence it can be the optimal PSF for recovering the text information using the OCR efficiently. Based on the Proposed Algorithm we have calculated AWC for different values of Blur Length L and Blur Angle 6 . In Fig.2 the Blurred Input Image 9 ( x , У ) containing the text is shown this is the Input to our Algorithm. The next Fig.3 shows the Output latent Image f’ ( x , У ) after the Deconvolution operation of 9 ( x , У ) and the estimated PSF with Blur Length L =17

And Angle в =10 degree. The individual Word Confidences are 0.39378 and 0.62334 respectively. Hence the AWC evaluates to 0.508564115 this value represents the Confidence for the recognition of text in the image. The individual Word Confidence value is shown in the figures with the help of Yellow boxes. Similarly Fig.4 shows the Latent image f' ( X , У ) obtained for the PSF with Blur Length L =20 and Angle e =14 degree with AWC = 0.756999287. Fig.5 also shows the Latent image f' ( X , У ) obtained for the PSF with Blur

Length L =25 and Angle 6 =13 degree with AWC = 0.621142268.

It is quite evident from the figures that the based on the value of AWC we can estimate the value of Blur Length L and Angle 6 for which the text recognition efficiency will be the highest. The highest value of Average Word Confidence AWC_m = 0.756988287 for the length of blur

= 20 pixels and angle of blur = 14 degree. This is the value for the PSF or the blur kernel estimated using our method which will get the best OCR output for the input blurred image 9 ( x , У ) using Blind Deconvolution method. The highest value of the Average Word Confidence AWC means that the text document is best recognized with that PSF.

We have represented the top 20 values of Average Word Confidence for different values of Blur Length L and Blur Angle 6 of blur in the Table I. The Table shows that the Blur Length L varies from 19 to 25 pixels and Blur Angle 6 varies in the range 8 to 14 degree. The value of Average Word Confidence ( AWC ) varies in the range 0.756988287 to 0.632594824. These values of AWC are acceptable but the highest value will give the best Text Recognition result so the value Blur Length L and Blur Angle 6 corresponding to it gives the optimal PSF to apply Deconvolution operation with the Blurred Input Image to get the Latent Image which can be feed to an Optical Character Recognition Engine. Then the Optical Character Recognition engine will give better text recognition for a Blurred Input image with text, which was one of the biggest problems with the existing system.

• V.    Conclusion and Future Work

This method is applied successful on the Blurred image taken manually and through the internet. We have also tested this method on artificially blurred images to validate our result. This method works successfully for the image having text and it cannot be used for deblurring any type of image it is only meant for the image having text in it. As OCR word confidence is used. If the image does not have any text then this method cannot be used. This method was proposed to overcome a problem that was to deblur a text document image So as to increase the accuracy of Recognition. The existing system was heavily dependent on OCR for text segmentation and recognition. But when the acquired image was blurred Image then most of the Efficient OCR engines performance was not satisfactory. In this paper we have worked on the issue of Motion blur which is the most common type of blur. The Blur kernels or the PSF were estimated and evaluated with the Average Word Confidence value. Based on AWC we have estimated the optimal PSF for the Input Image with text.

The method used is based on the estimation of PSF which is an iterative process and another solution to the Deblurring text is through Artificial Neural Network. Training a Neural Network is a time taking process and requires a lot of computation. As a part of future work we would like combine our approach and a Neural Network based approach together to get more optimized solution to Deblurring textual images.

Acknowledgment

I would like to express my gratitude towards almighty God the mother father of this universe for giving me strength and positive thinking to work on this research topic. I would like extend my gratitude towards my PhD guide Dr. Deepak Kumar Singh for encouraging me to write this paper. He has motivated me whenever I was down and has helped me in writing this paper. With his deep knowledge of the subject he has time and again helped and given his valuable suggestion which have helped to improve my work. I would also like to thank my parents and family for being my strength.