A Novel Boundary Matching Algorithm for Video Temporal Error Concealment

Published Online May 2014 in MECS DOI: 10.5815/ijigsp.2014.06.01

Video data compression is a process to decrease the digital data rate by removing redundant data. Elimination of video data redundancy may cause sensitivity to channel errors. In order to conceal these errors and enhancement of the reconstructed frames visual quality, it is used the error concealment techniques. These techniques are classified according to the used video sequences properties. In general, error concealment techniques are divided into three main categories: spatial domain [1-3], frequency domain [4-6], and temporal domain [7-34].

Spatial error concealment techniques use spatial redundancies among frame pixels to recover the damaged MB. Frequency error concealment techniques use data of adjacent MBs in the frequency domain for error concealment. Temporal error concealment exploits temporal redundancy among consecutive frames for error concealing of damaged MB. If there are no scene changes in consecutive frames, most objects of the current frame can be found in the previous frame. In this paper, we have focused on temporal error concealment.

The simplest temporal error concealment method is the temporal replacement (TR) [7]. In that method, all damaged MVs are replaced by zeros. The method is useful when there is a low motion among consecutive frames. In [8], a whole frame loss error concealment algorithm is proposed to further refine the TR. There are also some other simple methods which have special usages. Some of these methods [9] use the corresponding MB’s MV from the previous frame which has a better performance, assuming that the motion in video sequences is smooth. Another simple and common method is the use of the average or median of adjacent MVs of damaged MB [10]. The simulation results show that the use of median is better than average method. In [11], five simple error concealment methods are analyzed with two similarity metrics.

In the MV interpolation method [12], the MV of each 4×4 block is estimated by interpolation from MVs of adjacent MBs. The distance between adjacent blocks is used as their weights.

The Lagrange interpolation (LI) method [13], is also a simple and useful method for MV recovering of 4×4 blocks. It supposes that the damaged MVs are in the range of MVs from adjacent MBs. Obviously, if this assumption is invalid, the results will not be satisfactory.

The classic boundary matching algorithm (BMA) [14] uses the smoothness assumption from boundary pixels of damaged MB. The algorithm recovers the damaged MB by minimizing the boundary matching distortion among inner and outer boundaries of the reconstructed MB. The BMA method achieves very well results in estimating the MV of the damaged MB. However, slanting edges and rapid gray-level changes may cause extensive variations, which in turn decreases the BMA performance.

Gao and Lie [15] proposed a post-processing method for BMA with the use of Kalman filter. At first, the damaged MV is estimated by BMA. However, because of less information from boundary pixels, most of the estimated MVs are not accurate. Therefore, the obtained MVs are corrected by using the Kalman filter.

The edge adaptive boundary matching algorithm (EA-BMA) [16] uses a mask proportional to the edge strength in damaged blocks’ neighbors. The BMA-based algorithm employs the masks outside a damaged block instead of predicting the boundary pixels of the damaged block. Finally, the best-selected MV from matching process is used for damaged block.

An effective temporal error concealment algorithm [17], constructs a limited candidate MV set among the MVs of neighboring MBs and extrapolates MVs. It selects the best MV from the limited candidate MV set by using the BMA to conceal the corrupted MB.

Choi and Jeon [18] proposed an error concealment technique with block boundary smoothing to improve the video subjective quality. It uses the weighted boundary pixels of reference block to decrease the blocking artifacts compared to conventional temporal error concealment methods.

Huang and Lien [19] proposed a temporal error concealment technique using a self-organizing map. They used a self-organizing map as a predictor to estimate the MVs of damaged MBs. The estimated MVs were utilized to reconstruct the damaged MB by exploiting the spatial information from reference frames via employing a boundary matching criterion. The dynamic temporal error concealment technique using a competitive neural network (CNN) [20] uses a CNN predictor or BMA method for estimating damaged MB’s MV. Different methods are performed based on the video scene motion.

In a fuzzy reasoning-based temporal error concealment method [21], two measuring criterions, namely side match distortion (SMD) and sum of absolute difference (SAD), are considered together for estimating damaged MVs. Thus, the method is adopted to balance the effects of SMD and SAD to accomplish the judgment more accurately for candidate MVs. Also in [22], a fuzzy metric based on Sugeno fuzzy integral is used as the criterion to compare the candidate MVs. Unlike conventional metrics, this metric is more compatible with human visual system (HVS).

Araghi et al. [23] proposed a method for MV optimization of damaged MB with the two best MVs, which have obtained from BMA. Furthermore, a preprocessing step for determination of a proper MVs set is presented. According to BMA criterion with reliability coefficient in double weighted MVs algorithm [24], the two best MVs are selected. Then, the optimal MV is calculated by weighting the MVs in terms of their accuracy.

Thaipanich et al. [25] proposed an outer boundary matching algorithm (OBMA) to estimate damaged MVs. The OBMA uses spatial and temporal smoothness of damaged MB boundaries to conceal the damaged MB. This algorithm with minimizing the boundary matching distortion among outer boundaries of damaged MB and outer boundaries of reconstructed MB, recovers the damaged MB. Also in [26], the TR and the improved outer boundary matching algorithm have used for dynamical error concealment in inter-frames of videos.

In spatio-temporal boundary matching algorithm (STBMA) [27] has been used from BMA and OBMA criterions and side smoothness criterion of damaged MB for estimating the damaged MV.

The MVs interpolation method [28] interpolates the MVs of lost blocks in current frame using extrapolated MVs from the previous frame. It increases the accuracy in recovering the corrupted block MVs.

Wu et al. [29] proposed an enhanced edge-sensitive processing order for temporal error concealment algorithm. It uses an efficient processing order for error concealment by considering the side information of neighboring blocks. In addition, a MV searching algorithm for determining the best MV is presented.

An adaptive error concealment mechanism with the use of decision tree is proposed in [30] for damaged MB error concealment. It uses different spatial and temporal error concealment methods in terms of spatial and temporal features of the video sequences.

Wang et al. [31] proposed an integrated temporal error concealment technique for H.264/AVC. It switches between two modes, adaptively. The first mode is a conventional temporal error concealment step. The second mode is an integrated mode that obtains by integrating two temporal error concealment approaches with an adaptive weight. It can obtain the optimal recovery data for damaged MBs.

Wu et al. [32] proposed a spatial-temporal error concealment algorithm for H.264/AVC. In this algorithm, a frame-level scene-change detection is applied. If the scene change occurs the spatial error concealment is applied, otherwise the temporal error concealment is applied. For temporal error concealment, a predictionbased motion vector estimation scheme is applied to obtain the final MV.

Most of the conventional error concealment methods (such as BMA) apply only one direction to calculate the differences among boundaries in the boundary distortion function. The directional boundary matching (DBM) method [33] determines the direction of comparison for each boundary pixel of the candidate MB. Then, every boundary pixel in determined direction is compared with a pixel of the outer boundary of the damaged MB. It tries to improve the accuracy of damaged MV estimation.

The directional temporal boundary matching algorithm (DTBMA) [34] estimates the real boundary direction by considering pixel differences in three directions for temporal error concealment. If candidate

MBs are from previous stages of error concealment, it will not lead to satisfactory results. Although, in general, it has a better performance compare to BMA and DBM.

To solve the problems of conventional algorithms, an efficient boundary matching algorithm is proposed to estimate the damaged MVs more accurately. In this algorithm, the damaged MBs in each frame are reconstructed according to a priority list of identified error concealment. Then, this list is updated after reconstruction of each damaged MB. Moreover, this algorithm uses the classic boundary matching criterion or the proposed boundary matching criterion to identify boundary matching distortion for each candidate MBs ’ boundary, adaptively. Therefore, the proposed algorithm obtains more accurate estimation of damaged MV. It also prevents from error propagation in next frames.

The rest of this paper is organized as follows. First in Section II, the BMA is described briefly. Afterwards, Section III presents the proposed algorithm in detail. In Section IV, the experimental results are shown and finally Section V, concludes the paper.

• II.    Boundary matching algorithm

The algorithm assumes that there are high spatial correlations in undamaged image pixels. In the classic boundary matching criterion (BMC) the inner boundary pixels of candidate MB are compared with the outer boundary pixels of damaged MB in the current frame. Outer boundaries of damaged MB and inner boundaries of candidate MB are shown in Fig. 1.

The classic boundary matching criterion is defined for each boundary of damaged MB using

BMC top =

S I f _cur ( i + n , j - 1) - f ref ( i + vi + n , j + vj ) | n = 0

BMC bottom =

S 1 f _ ( i + n , j + S ) - f ref ( i + vi + n , j + vj + S - 1) I n = 0

BMC left =

SI f cur ( i - 1, j + n ) - f ref ( i + Vi , j + Vj + n ) I n = 0

BMC right =

S - 1

S ¹ f cur ⁽ i ⁺ S,^{j +} n ) ^- f ref ⁽ i ^{+ Vi +} S ^- 1, ^{j + Vj +} n ) ¹ n = 0

where (i,j) denotes the coordinate of top-left pixel in the damaged MB, fcur(.,.) is the indicator of current frame, fref(.,.) is the indicator of reference frame, MV(vi,vj) is the candidate MV, and S is the number of available pixels in each boundary. Also, top, bottom, left, and right are referred to top, left, bottom, and right boundaries of the damaged MB.