EQ: An Eigen Image Quality Assessment based on the Complement Feature

The demand on visual data access is increasing thanks to the internet and hand held devices. The requirement for a measure to Image Quality Assessment, IQA, is therefore essential. In general, IQA can be categorized into three groups depending on the amount of information needed from the original undistorted image.

The first category is where the original image is completely available, often referred to as Full Reference IQA (FR-IQA). In fact, FR-IQA has been widely used to evaluate and compare algorithms in many topics in image processing [1,2,3]. The comparison between the degraded and the original images can be performed at the pixel level such as the traditional SNR; block-based [4,5]; wavelets [6]; and natural scene statistics NSS [7,8].

The second category is the No Reference/Blind IQA, NR-IQA. Since the original image is unavailable in many situations, a statistical, e.g. generalized Gaussian [9] or its asymmetric version, framework can be learned from some images and hence database dependent. These statistics can be in any domain using intensity, gradient, or Laplacian at many scales with raw or normalized features. NR-IQA is then found as the amount of deviation the degraded image has from this statistics [10,11,12,13,14]. Some algorithms rely on deep neural networks, e.g. CNN, to avoid the dependency on the subjective scores needed in the previous schemes [15].

The third category is the Reduced Reference IQA, RR-IQA, that is the midpoint between the previous two categories. In general, few values describing certain features from the original image are needed to evaluate the degraded image [16,17].

In this paper, IQA is based on the complement feature recently proposed [18]. A correlation/covariance matrix is composed using the data in the block, can also be used for the whole image or any regularly shaped portion. The data is typically multi-dimensional with a plenty of possibilities. For example, the intensity of all the pixels in the block is composing just one feature vector. After constructing the features, a complement, e.g., Euclidean norm, is appended to create a feature array. The auto-correlation matrix is then decomposed through Eigen formulation in a similar fashion to the principal component analysis PCA framework [19]. The proposed IQA, named Eigen Quality EQ, is then found as the deviation of the Eigen values of the degraded image from that of the original image.

A singular value decomposition SVD [20] is proposed where each block in a gray-scale image is considered as a matrix and the singular values are then found. FR-IQA is then computed as the deviation of the degraded image singular values from that of the original image. On the contrary, the proposed scheme regard the whole block as a single feature vector. Hence, it can be easily generalized to use more than one feature, e.g. intensity, and can even go beyond the scope of FR-IQA.

In a different context, [21] proposed a complex structure in the wavelet domain where a covariance matrix is constructed for each coefficient using a set of neighboring coefficients. The algorithm depends on finding the Eigen values of the covariance matrix obtained for each coefficient. However, the resultant FR-IQA scheme depends on comparisons that are not explicitly these Eigen values.

The main goal of this work is to explore the auto correlation matrix [18] and the wealth of information embedded in it, EQ (as an FR-IQA scheme) is one of these products.

Comparison with complex scheme such as CNN, despite their potential, will not be considered. As suggested in the experimental section, comparisons will be made with two highly efficient schemes namely SSIM [4] and GMSD [8]. Despite the simplicity of [4] and [8], as they use first and second order statistics, there performance is remarkable. However, due to their dependence on MOS, some distortion types may experience biased results.

2.    Method

Without loss of generality, following the derivation in [18], the image is normalized to have a maximum of one, however, better performance was noticed when normalizing to [–1/√2, 1/√2]. Intensities of all the pixels in any nxn block are concatenated to produce a column vector of size Nx1, N=n². This representation is then extended to be of a “ unit” vector type given by

9 i          9 t

1- ^g 1 ^1- ^gt

9 n   1

7¹ - Sn-I

G is now a 2xN array, where N is the size of the block selected. An auto correlation matrix, AG, of size 2x2 is then constructed from G as

Solving the Eigen formula,

A_g = GG^T

A_gV = AV

The Eigen vectors of AG represent the axes of inertia for the data set, while the Eigen values represent the strength in that axis [18]. Hence, the largest vector Vmax, corresponding to the largest Eigen value λmax, points toward the direction of maximum inertia.

Unfortunately, distorted images show high similarity with undistorted ones in terms of low frequency power. This motivates the author to use the second largest Eigen value λM-1. Although other Eigen values can contribute to less extent, a more elaborate study is needed to channel these contributions.

The proposed IQA scheme, named Eigen Quality EQ , is simply the deviation of λM-1, defined in the previous paragraph, of the distorted image blocks from that of the corresponding blocks in the original image. Defining the distortion per block as

D t

_ min(O_i,T_i) max(O ^ ,T ( )

where, O is the Eigen value from the original block i, and T is the Eigen value from the corresponding test/distorted block. The distortion vector can then be formed by attaching the distortions from all the blocks in the image.

As a demonstration, image #5 is chosen from TID2013 [22] to illustrate some statistical observations supporting (4) and the effect of block size.

Fig.1. shows sorted distortion values calculated using (4) on 32x32 blocks. For comparison, Fig.2. shows the sorted absolute differences for the same images used in Fig.1. Obviously, the use of (4) is justified as Fig.1. is more diverse than Fig.2. In addition, Fig.3. shows the results of (4) on different types of distortion. Fig.4. and Fig.5. illustrate the results of (4) for each block, higher size is used to enable visual distinction. As can be seen from these figures, the proposed method can distinguish between different types of distortions and different levels of distortion.

Fig.1. Sorted distortions, block size 32x32, for the 5 levels of distortion #7 from TID2013-i05.

Fig.2. Sorted absolute differences, block size 32x32, for the 5 levels of distortion #7 from TID2013-i05.

Fig.3. Sorted distortions, block size 32x32, for level 4 of distortions 5 – 9 from TID2013-i05.

Fig.4. Distortion of each block, size 64x64, for the 5 levels of distortion #7 from TID2013-i05.

Fig.5. Distortion of each block, size 64x64, for level 4 of distortions 5 – 9 from TID2013-i05.

Two interesting polling schemes on the distortion vector will be demonstrated in the experimental results. The first is simply a weighted sum of the mean and maximum value. The second is the 99% rank.

The block size used is 21x21, however, it was noticed that the effect is marginal around this size. As with any other IQA scheme, decreasing/increasing the block size can lead to over/under-estimate the distortion. In addition, it was noticed that choosing sizes that are multiple of 8 can be biased in the IQA obtained for jpeg images.

The scheme in (1) can be extended in many ways to include more features. A higher dimensional paradigm is also possible by using powers, not necessarily integers, of the pixel intensity. Care should be taken to take the power of the magnitude part and leave the sign part unchanged, this can not be followed in the calculation of the complement feature. Some suggestions are as follows

G =

£ 2 +£ 4 +£ i

J*

д+й+щ.

9n

On

/9 n

0 n +0 n +0 N

¹--3       .

(5a)

G =

G =

/1- 91

J ¹-"⁹ ?  V¹ - ⁹ n

(5b)

■    91

/9 1

Z ^1- 9 1

/1- 91/1- 91

9 i

9 i

// J ¹ " ⁹ j ^1-9 V ^1- 9 i

9n   "

9 N

^{/1- 9}n

/1-9n

(5c)

Each resultant column of G should be normalized to have a unit magnitude to avoid biasing. However, this normalization can be relaxed for (5b) and (5c) as all the vectors have the same magnitude. However, minor differences were noticed in the performance of the variants of (5).

In fact, (5) can be easily extended to incorporate other features such as color, particularly (5a). However, the best aggregation method is still under investigation.

The proposed EQ scheme is tested on all the images in TID2013 using all degradation types and there associated levels. In effect, the resultant score is assessed against the MOS, mean opinion score, reported in the database. The performance is assessed through three indices. The first index is the traditional mean absolute error, MAE, given by

MAE =

E m,n |^ mn У тА

S

where, x and y stand for original and segmented images, and S is the number of samples, here images in the database. Adjustment should be placed when the range of images are different.

The second index is the Linear Correlation coefficient, Pearson LC, given by

LC =

Z m,n СГ тп -М хХ У тП -М у )

SO ^ O y

where, µ is the mean and σ is the standard deviation.

The third index is the Rank order Correlation coefficient, Spearman RC, given by

RC = 1

61 m,n {Rank⁽x_mn⁾-Rank⁽y_mn)}²

S(S2-1)

where, the difference is between the ranks rather than the actual values.

3.    Experimental Results

The three indices given by (6)–(8) are evaluated for the proposed EQ, using (1), against the traditional RMSE, SSIM [4], and GMSD [8] quality measures. Two versions of the proposed scheme will be tested, see (4), namely EQ MeanMax , 0.3 mean{D} + 0.7 max{D}, and EQ 0.99Rank . However, slight deviations can improve performance in an image dependent fashion.

Table 1. shows the evaluations for the intensity component only. It is clear that the two versions of the proposed scheme are superior to SSIM and RMSE in all aspects. However, it is superior to GMSD in terms of absolute difference but inferior in terms of correlation.

In a similar fashion, Table 2. shows the rsults using the RGB components. GMSD only uses gradient and hence the values are the same. A similar conclusion can be drawn regarding the comparison with other schemes.

Unfortunately, comparisons are dependent on MOS. Hence, some distortion types may experience biased results. In fact, for some distortions, the relative distance between the levels of degradation is better acquired by EQ rather than SSIM or GMSD. However, more elaboration is required in future work.

Interestingly, MAE and LC are better for EQ using RGB component than using the intensity component alone. This note is worth further exploration as what are the best features to be incorporated.

Unfortunately, the variants of (5) were not superior to that of (1).

4.    Conclusion and Future Work

An Eigen structure is proposed in this paper to perform image quality assessment. The proposed schemes are very effective as demonstrated by the values of MAE, Pearson, and Spearman correlation coefficients. Comparison with SSIM and GMSD indicate the remarkable performance of the proposed scheme EQ.

Work is currently in progress to explore other color spaces, the information from the Eigen vectors, and other Eigen values. In addition, the effect of scaling, normalizing, and aggregation.

The component added to obtain a unit vector in (1) can be generalized to any fuzzy complement. However, more work is needed to find the best formula and whether improvements can be attained, see [18] for some suggestions.

The proposed scheme can be generalized to RR-IQA by comparing some statistics between the degraded and original images. A no reference NR-IQA was tested, however, it is limited to certain degradation types.

Table 1. MAE, LC, and RC for the schemes RMSE, SSIM, GMSD, EQ MeanMax , and EQ 0.99Rank using the intensity component only.

	RMSE              SSIM              GMSD              EQ MeanMax            EQ 0.99Rank
MAE LC RC	0.461                  0.372                 0.434                 0.144                 0.176 0.567                  0.695                 0.855                 0.713                 0.736 0.640                  0.679                 0.804                 0.751                 0.759

Table 2. MAE, LC, and RC for the schemes RMSE, SSIM, GMSD, and EQ using RGB components.

	RMSE	SSIM	GMSD	EQ MeanMax	EQ 0.99Rank
MAE	0.450	0.329	0.434	0.128	0.162
LC	0.597	0.633	0.855	0.746	0.753
RC	0.686	0.629	0.804	0.748	0.749