A Survey on Shadow Removal Techniques for Single Image

Published Online December 2016 in MECS

The presence of shadows in an image assists the user in locating the light sources and determining the size and shape of the object casting the shadow. But, they impair the proper execution of computer vision algorithms for segmentation, object recognition, video analysis, etc. In image segmentation and object detection, the shadow region itself may be misclassified as an object or a part of an object. Detection of moving objects in a video recording may also yield improper results due to the presence of moving shadows. Hence, these applications need the shadows to be eliminated in the preprocessing stage.

Human Visual System is capable of locating shadows in an image in most of the situations. However, it is hard to develop algorithms that can automatically detect shadows without any human intervention. Various complications may arise while dealing with shadows in an image. An image may have complex scattered shadows, self-shadows, hard and soft shadows caused by multiple light sources, shadow boundary coinciding with object edge and so on. In addition, shadows of different objects may overlap or the object casting shadows may be absent in the image. The shadows may span multiple surfaces, or have complex underlying texture, or there may be dark regions which appear like shadows. All these issues cause trouble in detecting and removing shadows from an image.

We consider shadow removal as a two-phase process, involving a detection phase and a removal phase. Numerous methods to remove shadows from videos[1], aerial images[2] and outdoor images[3] are available in the literature. A survey on shadow detection and removal methods for real images was done by Xu et al.[4]. Another extensive review with the classification of the shadow removal algorithms was presented by Sasi and Govindan[5]. In this work, we attempt to review the various shadow removal techniques for a single image available in the literature. We have identified five major classes of shadow removal algorithms, namely, reintegration methods, relighting methods, patch-based methods, color transfer methods, and interactive methods. Though some of the works may fit under more than one category, we have classified each of them into the most appropriate category.

The remainder of this paper is structured in various sections as given in the following: Section II introduces different types of shadows. Section III presents the major challenges involved and the expected outcome of shadow removal algorithms. Various classes of techniques employed for shadow removal algorithms are reviewed in Section IV. A comparison of the methods discussed follows in Section V. Finally, the survey is concluded highlighting major observations in SectionVI.

• II.    Types of Shadows

Shadows can be broadly classified as self-shadows and cast shadows. Self-shadow is the shadow cast by an object on itself. This kind of shadow appears on an object when direct light from a source is unable to reach that area of the object.

Cast shadows are the shadows formed by an object on another object or surface. These shadows have two main regions when the scene is illuminated by multiple light sources. The dark inner region in cast shadow is called umbra, and the light outer region is called penumbra. The texture and color information of the underlying surface is mostly lost in the umbra region, while the illumination intensity is non-uniform in the penumbra region. The different shadow regions are shown in Fig.1.

• III.    Challenges in Shadow Removal

Elimination of shadows from an image is particularly challenging since the entire information needed to locate and eliminate shadows should be derived from the input image itself. The automated methods do not possess extra information regarding the number or location of light sources illuminating the scene or depth information. Often, the object casting the shadow may not be present in the image. All these contribute to the difficulty in detecting and removing shadows from an image. An automated algorithm for removing shadows from an image is expected to hold the properties listed below:

• •    Preserve the texture beneath the shadow

• •    Retain the color information of the surface

• •    Make the shadow edges unnoticeable in the shadow-free image

• •    Reduce visual artifacts

• •    Consume less time since shadow removal is dealt with as a preprocessing task

Regaining the texture or color information under hard shadows may be difficult due to the loss of details in these lesser-illuminated areas.

• IV.    Shadow Removal Techniques

Some of the shadow removal techniques include a shadow detection phase prior to the removal stage, while others take a shadow detection result as input and perform removal over it. This survey focuses on the removal phase rather than the detection phase. The major works in shadow removal are classified into the following categories:

• A.    Reintegration methods

• B.    Relighting methods

• C.    Patch-based methods

• D.    Color transfer methods

• E.    Interactive methods

A brief description of the works belonging to each of these categories is given in this section.

• A.    Reintegration Methods

The reintegration methods for shadow removal are built on the concept that nullifying the image gradient along the shadow edges and integrating back the modified gradient will produce a shadow-free image. An extensive study on reintegration based shadow removal methods was performed by Finlayson and team [6,7,9,10,11]. Initially, the shadow edges were detected using an invariant image representation. The image gradient was then computed and nullified along the shadow edges, followed by reintegration. These techniques assume that the variation of image illumination is at a slower rate compared to reflectance.

Finlayson et al.[6] proposed a mechanism to eliminate shadows from images by reintegrating the image gradient in which the x and y derivatives of the pixels in shadow edges were set to zero. The reintegration was done separately for each color band by solving a 2Dimensional Poisson equation. In [7], shadow edges are not set to zero. Instead, iterative diffusion is used to fill the shadow edge derivatives by values from non-shadow neighbors, followed by reintegration as in [6].

The 2-Dimensional reintegration results in artifacts for imperfect shadow edge detection input. Moreover, the technique is computationally expensive due to the reintegration on two derivatives per pixel. Later, the authors proposed a 1-Dimensional path-based reintegration using retinex[8] in [9] to reduce the computation overhead. The average lightness at each pixel was obtained by computing the ratio between the initial pixel and each pixel along multiple paths in each color channel. They used 20 to 40 random paths each of length three-quarters of Image size.

Reintegration along random paths using retinex often included certain pixels multiple times, leading to visible artifacts and higher complexity. A faster reintegration of the image gradient along non-random raster and fractal paths was proposed by Fredembach and Finlayson[10]. Their technique gave good results upon averaging along 16 such Hamiltonian paths. The authors subsequently spotted that shadow edges should be closed and a path should enter and leave the shadow region only once[11]. Hamiltonian paths satisfying these conditions were used for reintegration yielding better shadow removal results in less time.

Discussion:

The reintegration based shadow removal algorithms generally improve the quality of the entire input image. The major drawback of these methods is the need for strong shadow edges in the detection output. In addition, the 2-Dimensional reintegration  is  computationally expensive than the 1-Dimensional path reintegration. Lesser artifacts are produced  by 1-Dimensional reintegration if each pixel is encountered only once, and the reintegration results are averaged along multiple paths.

• B.    Relighting Methods

Shadow regions appear in an image due to a reduction in the amount of light reaching the area compared to the non-shadow regions. The objective of relighting methods for shadow removal is to find a factor that can be used to enhance the lightness of the shadow pixels. In [12], Arbel and Hel-Or mentioned that the factor can either be a multiplicative constant in image domain or an additive constant in log domain as shown in (1) and (2) respectively.

I (x,y) = R(x,y) • Цх.у) • С (x, у)     (1)

Taking logarithm of (1) :

hog ⁽x,y) = R I o_fl ( x,y ) + L , _og(x, y) + C , _og (x , y) (2)

where, I is the image, R and L are the reflectance and luminance components and C is the factor used to correct the shadow pixels. The various methods to relight the shadow regions are discussed in this section.

In [3], the authors demonstrated that the costly reintegration procedure can be replaced by computing a constant in each color channel that will minimize the variation between shadow and non-shadow pixels on either side of shadow edge. They evaluated an array C that minimizes the least square error between the pixel arrays outside and inside the shadow edge, P and S respectively using equation (3).

C = min_a ||P - 5 + a||²            (3)

The values in C corresponding to least error are then averaged to find the constant. This constant was calculated separately for each shadow region and added to the pixels in the region to get the shadow-free image.

Du et al. [13] estimated solar and environmental light, pixel reflectance and light attenuation factor to relight each shadow pixel in an outdoor image. An algorithm to remove shadows on curved surfaces was proposed by

Arbel and Hel-Or[12]. They initially calculated an additive scale factor in log domain for the inner shadow region using cubic smoothing splines. Directional smoothing was then deployed to correct the scale factors in penumbra to eliminate the abrupt variation that may arise at the shadow edge after shadow removal.

Salamati et al.[14] determined the lightness factor for each pixel in umbra and penumbra using a probability shadow map, in the LAB color space. Chromaticity and boundary correction were performed after the lightness correction. Their method preserves both texture and color in the results. Fractional shadow coefficients derived by matting were used by Guo et al. [15] to derive the scale factor for shadow pixels. The pixel relighting was done by using the following equation:

shadowJree = r«_          гдх

• ¹1                 к r +1 ¹             (4)

where r is the ratio of direct light to environment light and I_i is the i -th image pixel. k_i is a value in the range [0,1] that depends on the amount of direct light falling on the region. The value of k_i is 1 for a non-shadow pixel, 0 for an umbra pixel. Other values indicate that the pixel is in the penumbra.

Shadow removal was achieved using region relighting in the work by Vicente et al.[16]. In their work, a trained classifier was employed to determine a shadow region and its corresponding lit region pair. The shadow region was relit by matching the luminance of shadow region and its corresponding lit region. The relighting transformation T is defined based on the shadow region ( R_S ) and the non-shadow region ( R_NS ) of the same material as given in (5). Color and boundary correction were achieved by adding offset in LAB color space.

T (R_S,R_NS) = R_s sudi «гat R_s « R _NS      (5)

Discussion :

The relighting methods for shadow removal aim at finding a scale factor that can be added to the shadow region, thereby reducing the difference in illumination between shadow and non-shadow regions. These methods are usually simple and fast. The major challenge is in finding separate scale factors for umbra and penumbra shadows, and shadow boundary correction. Within the penumbra, different pixels might vary in illumination which ultimately needs multiple scale factors for a single shadow region. Also, a suitable lit region should be found to calculate the relighting factor in most of the cases.

• C.    Patch- based Methods

The patch-based methods for shadow removal attempt to operate on patches rather than on single pixels. These methods work on the assumption that the illumination and reflectance within a shadow patch are almost nonvarying. Some of the works consider overlapping patches while others use non-overlapping patches.

The shadow removal technique proposed by Gryka et al. [17] computed a feature vector for each of the nonoverlapping patches in an image. The feature vector of each shadow patch was mapped to a set of possible shadow mattes using a trained regressor, and the best matching matte was found by Markov Random Field(MRF). The extraction of shadow matte from the red, green and blue color planes resulted in a shadow-free image. An illumination recovering operator computed from a shadow patch and its corresponding lit patch was used to remove shadows in [18]. This method divided the image into overlapped patches. The illumination of overlapped pixels was optimized by a weighted average of the pixels in the patch.

Ma et al. [19] used a patch-based image synthesis approach that reconstructs the shadow region using patches sampled from non-shadow regions. The color and texture of shadow patches were then modified based on correction parameters. This was followed by optimization using a confidence which assured that shadow patches without matching non-shadow patch are also rectified. In [20], Sasi and Govindan extracted a shadow image by finding the difference between an image and its invariant. Geometric and shadow sub-dictionaries were formed from the patches of this shadow image by learning. The geometric component of the shadow image was recovered and finally added to the invariant to get a shadow-free image.

Discussion :

The patch-based methods process each shadow patch instead of single shadow pixels. This requires a considerably less amount of time for shadow removal. Sometimes, it is difficult to find correctly matching nonshadow patch, for a shadow patch. In addition, inappropriate patch matching may lead to unexpected results.

• D.    Color Transfer Methods

The foundation of color transfer methods for shadow removal lies on the work by Reinhard et al. [21].These methods aim at transferring color information from the lit areas to the shadow areas. The colors from the lit regions are transferred to shadow regions using mean and standard deviation of the Gaussian distribution followed by the color intensities in an image [22].

Wu and Tang [22] used a Bayesian formulation to extract the shadow image β from the image I , leaving a shadow-free image Ƒ . The image I is represented as a combination of in β and Ƒ in (6). Within the shadow region, β is estimated using the mean intensity of pixels in shadow and lit regions. For the uncertain regions, β is estimated by considering the affinity of a pixel to the probability distribution of shadow region. The shadow-free image is then computed by solving a Poisson equation followed by optimization using prior of β to get smooth shadow in β, and hence, retaining the texture in Ƒ .

I = β Ƒ (6)

Wu et al. [23] formulated shadow effect as light attenuation problem and generated an initial shadow-free image by color transfer from non-shadow to shadow regions based on the probability of a pixel being a shadow pixel, and color histogram. The effect of color transfer at shadow boundaries was then reduced by affinity map and the shadow matte β was computed by energy minimization. Shor and Lischinski [24] initially computed Laplacian pyramid of the input image and downsampled shadow masks for each level. Four affine parameters modeling the relation between a shadow pixel and its illuminated intensities were estimated based on the mean color and standard deviation of luminance in shadow and corresponding non-shadow regions and applied on shadow pixels at each level. The relation is shown in (7).

I^l_k^lt (P) = a_k(p) + у(р)1—(Р) (7)

Here, a_k is the camera response at k e {R,G,B} ; у (p) =1/a(x) is the inverse of ambient attenuation factor. The modified pyramid was then flattened and edges were inpainted to obtain the shadow-free image. This method used uniform parameters for entire shadow region without considering reflectance variation within shadow. A technique to overcome this limitation was put forward by Xiao et al. [25]. They calculated adaptive parameters for illumination transfer from lit regions to shadow regions on multiple scales and combined the results to get less noisy output. The method used global illumination transfer followed by direct illumination of each shadow pixel in LAB color space, and recovered shadow-free image by (7).

These shadow removal approaches did not consider the varying texture within a shadow region. This often resulted in inaccurate texture recovery. This was addressed in [26] where the shadow and lit regions were segmented into sub-regions based on texture and a matching lit sub-region was found for each shadow subregion based on texture feature and distance, followed by illumination transfer [25]. Khan et al. [27] estimated an initial shadow-free image using multilevel color transfer and improved the estimate by inpainting the boundaries. The mean value of each cluster was used in transferring color at each level and the transfers were integrated at the end. Bayesian formulation was then used to extract shadow model parameters thereby forming a shadow-free image.

Discussion :

The color transfer methods attempt to transmit color details from the lit regions to shadow regions. It is seen that finding a uniform set of parameters to recover the shadow region may produce visual artifacts. Determining adaptive parameters and selecting appropriate nonshadow regions to calculate the parameters were found to improve the results. These techniques may produce unacceptable results for hard shadow edges and highly colored, non-uniform textures.

• E.    Interactive Methods

Shadow removal techniques are either interactive or automatic. Many authors suggest that a simple user input can lead to faster shadow detection. Also, providing the user a platform to interact with the shadow removal system can lead to improved results. This Section deals with various interactive shadow removal methods in the literature.

Liu et al. [28] proposed an interactive shadow removal technique in which the user should mark the shadow boundary in the image. The image gradient, in penumbra region derived from the user input, was modified using illumination change curves and the new gradient was integrated by solving Poisson equation. Miyazaki et al. [29] proposed a way to eliminate shadows by hierarchical graph cut. Their algorithm needed the user to mark shadow, lit and background regions using a stroke. During every iteration of hierarchical graph cut, the user can interact with the system by marking the imperfectly recovered areas, leading to the final shadow-free output.

Arbel and Hel-Or[30] generated shadow, penumbra and lit region masks from the shadow and lit regions marked by the user. Anchor points in the image were then selected based on the monotonicity of pixels to generate a shadow and surround mask. These anchor points were used to find scale factor for umbra and penumbra regions by intensity surface approximation. In [31], the features extracted from shadow and lit regions input by the user was used to train a Granular Reflex Fuzzy Min-Max Neural Network (GrRFMN). The pixels in the Region Of Interest (ROI) marked by the user were examined to find their fuzzy-membership in shadow or lit region, and the shadow pixels were recovered using a correction factor based on mean RGB in the shadow and lit regions. The user can again enter another ROI and repeat the procedure.

A simple user stroke on the umbra region was taken as input by Gong et al. [32] from which the shadow boundary was derived. Shadow was then removed by a scale estimated using illumination variance at intensity samples along different positions in the shadow boundary, followed by color transfer. A fast shadow removal technique that ask the user to scribble samples of lit and shadow pixels was given in [33]. A fusion image that boosts the illumination discontinuity along the shadow edge and conceals the texture was developed from the shadow mask. The illumination change along shadow boundary was used to form a penumbra strip. A sparse shadow scale followed by dense scale was estimated using the penumbra strip and the shadow region was relighted.

Discussion :

The interactive techniques for shadow removal are generally simpler than the automated techniques since the user can provide useful cues for locating the shadows in an image. Also, certain techniques [29][31] let the user to iteratively interact with the removal system thereby deriving a shadow-free image.

• V.    Comparison

This Section gives a listing of the most widely used single image shadow detection and removal datasets. This is followed by a qualitative and quantitative comparison of the shadow removal methods discussed in the previous section.

• A.    Datasets

The four main datasets available online for detection and removal of shadows in single images are as follows:

• •    UIUC dataset by Guo et al.[15]: 108 natural scenes taken under different illumination and their ground truth.

• •    CMU dataset by Lalonde et al.[34]: 135 outdoor consumer photographs with shadow boundary annotations.

• •    Dataset by Gong et al.[33]: 214 images with

ground truth.

• •    UCF dataset by Zhu et al.[35]: 355 images and corresponding manually-labeled ground truths.

• B.    Qualitative Analysis

In this section, we compare the shadow removal algorithms using the visual quality of the output shadow-free images. A comparison of these algorithms is given in Table 1. Shadow removal is usually performed at pixel, region, or patch level. The pixel-based methods usually consume a large amount of time since processing is performed on each pixel at a time. The patch-based and region based removal techniques process a set of pixels together. Some of the algorithms use learning based system to detect or remove shadows. These methods train the system with a set of images and use the learned features to locate and remove shadows from an input image. The learning based methods are found to give good quality results. The shadow removal algorithms also make assumptions on the lighting conditions, camera or surface properties. Many of these methods work for scenes illuminated by point light source and Lambertian reflectance.

Table 2 and Table 3 depicts the shadow removal results of some of the works discussed in this paper. A comparison of the 2-Dimensional reintegration [6] based on Poisson equation and 1-Dimensional path-based reintegration [11] using an average of 4 Hamiltonian paths is given in Table 2(a). The result of 1-Dimensional reintegration looks more pleasing than the 2-Dimensional method. In addition, the results of path based reintegration can be obtained in lesser time. Table 2(b)(iii) illustrates the artifacts present at the shadow boundary due to the poor performance of the boundary processing method in [16]. Table 2(b)(iv) gives the result of Khan et al.[27] in which the transition from shadow to the nonshadow region is almost imperceptible.

A shadow region may not always possess same texture or color reflectance. Hence, applying uniform parameters to eliminate shadows from an image may lead to inappropriate results as shown in Table 2(c)(iii). This problem is addressed by Xiao et al.[25] in which the reflectance and texture variation inside the shadow region is also considered in evaluating the parameters for shadow correction. Table 2(d)(iii) shows that the simple shadow removal by Fredembach and Finlayson[3] were not able to find a constant that could recover umbra and penumbra regions. But, the relighting used by Salamati et al.[14] produced good results without loss of underlying texture.

Table 1. Qualitative comparison of shadow removal methods

Reintegration Methods	level	FI	PT	LB	LD	UP	UI	DO	BC
Finlayson et al.[6]	pixel	✓	✓	X	✓	X	X	edge	X
Finlayson et al.[9]	pixel	✓	✓	X	✓	X	X	edge	X
Finlayson et al.[10]	pixel	✓	✓	X	✓	X	X	edge	X
Fredembach and Finlayson[11]	pixel	✓	✓	X	✓	X	X	edge	inpaint
Finlayson et al.[7]	pixel	✓	✓	X	✓	X	X	edge	inpaint
Relighting Methods	level	FI	PT	LB	LD	UP	UI	DO	BC
Fredembach and Finlayson[3]	pixel	X	✓	X	✓		X	edge	inpaint
Du et al.[13]	pixel	X	✓	X	✓	✓	✓	matte	X
Arbel and Hel-Or[12]	pixel	X	✓	X		✓	X	mask
Salamati et al.[14]	pixel	X	✓	X		✓	X	probability map	gaussian smoothening
Guo et al.[15]	pixel	X	✓	X	✓	✓	X	shadow coefficient
Vicente et al.[16]	region	X	✓	✓			X	mask	gaussian filter
Patch-based Methods	level	FI	PT	LB	LD	UP	UI	DO	BC
Gryka et al.[17]	patch	X	✓	✓	✓	✓	mask	mask	X
Zhang et al.[18]	patch	X	✓	X	✓	✓	trimap	matte	✓
Ma et al.[19]	patch	X	✓	X			X	mask	X
Sasi and Govindan[20]	patch	✓	✓	✓	X		X		X
Color Transfer Methods	level	FI	PT	LB	LD	UP	UI	DO	BC
Wu and Tang[22]	pixel	X	✓	X	✓	✓	✓	extraction	✓
Wu et al.[23]	pixel	X	✓	X	✓		✓	extraction	✓
Shor and Lischinski[24]	pixel	X	✓	X	✓		✓	mask	inpaint
Xiao et al.[25]	region	X	✓	X	✓	X	✓	mask	bayesian
Xiao et al.[26]	region	X	✓	X		X	X	mask	alpha matte interpolation
Khan et al.[27]	pixel	X	✓	✓	✓	✓	X	mask	alpha matte interpolation
Interactive Methods	level	FI	PT	LB	LD	UP	UI	DO	BC
Liu et al.[28]	pixel	X	✓	X	X	✓	edge	edge	X
Miyazaki et al.[29]	region	X	✓	X	X	✓	stroke	extraction	X
Arbel and Hel-Or[30]	pixel	X	✓	X	X	✓	markings	mask	X
Nandekar et al.[31]	pixel	X	X	✓		✓	stroke	fuzzy membership	X
Gong et al. [32]	pixel	X	✓	X	X	✓	rough stroke	mask	X
Gong et al.[33]	pixel	X	✓	X	X	✓	scribbles	mask	X

FI – Modifies entire image, PT- Preserves texture, LB – Learning based, LD – light or camera dependency, UP – both umbra and penumbra,

UI - user input, DO- Detection output, BC - Shadow boundary correction, ✓ - yes, X - no

Gryka et al.[17] designed a method to remove soft shadows. Table 2(e)(iii) illustrates the result of applying this method on hard shadow. The method doesn’t give good results since the training was done for soft shadows only. The region based shadow removal by Guo et al.[15] considers irregular shadow region which might have different textures and colors. This introduces error in the relighting constant for the region. Whereas the patchbased method by Zhang et al.[18] uses adaptive overlapped patches to compute the relighting factor using matching patches. Table 2(g) illustrates the shadow removal from curved surfaces.

Table 3(i) shows the effect of shadow detection output in the final result. The detection result by Guo et al. [15] misclassified the dark pattern on the box as shadows and hence the dark pattern was removed from the image. The Table 3(ii) shows the entire image enhanced by the method proposed by Finlayson et al.[6] on the attempt to remove shadows. An example of the interactive methods is displayed in Table 3(iii). Liu et al. [28] needs the user to input the shadow boundary itself whereas [32] asks the user to give a rough stroke in the shadow region.

• C.    Quantitative Analysis

Most of the authors use Root Mean Square Error(RMSE) to evaluate the shadow removal results. The per-pixel RMSE between shadow removal output and ground truth shadowless images for some of the works are tabulated in Table 4. The comparison figures are obtained from the works discussed in this paper. The table shows RMSE for the shadow regions and nonshadow regions separately. The actual error of an image with and without the shadow is also included. From the table, it can be observed that the RMSE for shadow regions and overall image is least for Vicente et al.[16]. This means their method gives better results in terms of per-pixel accuracy. Table 5 gives average RMSE for some of the methods discussed. Again, it is clear that the one with least RMSE is the method proposed by Sasi and Govindan[20].

Table 2. Qualitative Results(A) : Column (ii) has input images.

Columns(iii) and (iv) has the shadow removal results by the techniques mentioned under each image.

(i)	(ii)	(iii)	(iv)
		^BSv.vy'^i
(a) 2D and 1D	Л^Н^	^pp \
reintegration	input image	Finlayson et al[6]	Fredembach and Finlayson[11]
(b) shadow boundary
correction	input image	Vicente et al[16]	Khan et al[27]
(c) surface reflectance	Oleite
	input image	Shor et al[24]	Xiao et al[25]
(d) umbra and	it	it
penumbra correction	input image	Fredembach and Finlayson[3]	Salamati et al[14]
(e) effect
of training set	input image	Gryka et al[17]	Guo et al[15]
(f) region vs patch	input image	it® Guo et al[15]	Zhang et al[18]
(g) curved surface	input image	Khan et al[27]	Arbel and Hel-Or[12]

The learning based methods for shadow removal consumes much time and memory compared to other methods. In addition, pixel-wise processing brings about immense computation overhead. Among the reintegration based methods, the 1-Dimensional path-based methods are found to be computationally efficient than the 2Dimensional Poisson equation approach. Initially, path lengths were taken as almost three quarters original image size[9]. This needed averaging the results along 20 to 40 paths to get a reasonably good output. Later, the number of paths was reduced to 16 with non-random Hamiltonian paths [10]. By imposing closed shadow edge constraint[11], 4 Hamiltonian paths were able to give good removal results.

Table 3. Qualitative Results (B) : The technique used is mentioned under each image.

(i)	(ii)	(iii)
input image	input image	Liu et al [28]
Guo et al[15]	Finlayson et al[6]	Gong et al [32]

Table 4. Per pixel RMSE for UIUC dataset

Method	Shadow region RMSE	NonShadow RMSE	All region RMSE
Wu et al.[23]	21.3	5.9	9.7
Guo et al.[15]	11.8	4.7	6.4
Khan et al.[27]	10.5	4.7	6.1
Vicente et al.[16]	9.24	4.9	5.9
Actual Error	42	4.6	13.7

Table 5. Average RMSE for UIUC Dataset

Method	Average RMSE
Guo et al.[15]	19.85
Arbel and Hel-Or[12]	18.36
Gryka et al.[17]	13.83
Sasi and Govindan[20]	12.23

• VI.    Conclusion

Shadows are unavoidable entities that appear in an image when direct light is unable to illuminate the entire scene uniformly due to the presence of an obstruction between the light source and a surface. Due to the adverse effect of shadows in the image with various applications, removal of shadows from an image has become an active area of research in Computer Vision. This is considered as a difficult task since the entire input necessary to locate and illuminate the shadow region is to be derived from the single image. In this paper, we have discussed a few of the notable works in the literature to remove the shadows in an image. For the purpose of structuring the review, shadow removal techniques were classified under five categories, namely, reintegration methods, relighting methods, patch-based methods, color transfer methods, and interactive methods.

Numerous works are available for eliminating different kinds of shadows such as cast shadow, self-shadow, soft shadow, hard shadow, shadow in videos, aerial images, and outdoor images. Any shadow removal technique should initially detect the shadows in the input image. This is followed by illuminating the shadow regions to get a shadow-free output.

One of the earliest methods for shadow removal from an image was using reintegration of derivative image in which the effect of shadow edges was reduced to zero. While the 2-Dimensional reintegration methods were observed to be computationally expensive, authors came up with 1-Dimensional path-based methods to remove shadows. All these methods assumed single point light source and needed strong shadow edges.

The relighting methods aim at finding a constant or a set of constants that can relight the shadow regions such that the transition from a shadow region to the adjacent non-shadow regions in the shadow-free image is imperceptible. These methods are usually fast and most of them find separate constants for umbra and penumbra regions. The major focus of patch-based methods is to reduce the computation time needed by pixel based systems. They are based on the assumption that within a patch, the variation in reflectance is very less.

The color transfer methods try to restore the information in the shadow region using the color information from the lit region. The works in this category have evolved from a uniform set of parameters for the shadow region to an adaptive set of parameters considering the varying reflectance within a shadow region. The last category is that of interactive shadow removal techniques. These methods need the user to provide information on the shadow location within the image. From the survey, it can be concluded that the shadow removal algorithms should be able to yield good quality results that preserve the texture and color information underlying the shadow region, without revealing the transition between shadow and non-shadow regions, in real time.