CN104243977B

CN104243977B - Based on the theoretical stereo image quality evaluation methodology with parallax compensation of ocular dominance

Info

Publication number: CN104243977B
Application number: CN201410491118.9A
Authority: CN
Inventors: 杨小康; 刘敏; 翟广涛; 顾锞
Original assignee: Shanghai Jiao Tong University
Current assignee: Shanghai Jiao Tong University
Priority date: 2014-09-23
Filing date: 2014-09-23
Publication date: 2016-07-06
Anticipated expiration: 2034-09-23
Also published as: CN104243977A

Abstract

The present invention provides a stereoscopic image quality evaluation method based on eye dominance theory and parallax compensation, the method: the first step is to use the 2D no-reference quality evaluation method to calculate the quality scores of the left-eye and right-eye images, and obtain Q(L) , Q(R); the second step is to calculate the eye dominance difference between the two 2D image quality, and obtain H(L, R); the third step is to calculate the compensation amount of different parallax degrees, and obtain d ^θ ; the fourth step , add the quality scores Q(L) and Q(R) of the left-eye and right-eye images to the compensation amount d ^θ of the parallax degree, and subtract the eye dominance difference H(L, R), the result is based on the eye dominance theory and parallax-compensated no-reference 3D image quality assessment scores. The present invention establishes the connection between 2D image quality evaluation and 3D image quality evaluation methods, obtains the degree of parallax shift of left and right eye images, and compensates the influence of image content on 3D image quality, thereby improving the accuracy of 3D image quality evaluation.

Description

Stereoscopic Image Quality Evaluation Method Based on Eye Dominance Theory and Parallax Compensation

技术领域technical field

本发明涉及立体图像质量评价领域的方法，具体地，涉及一种基于眼优势理论和视差补偿的无参考立体图像质量评价方法。The invention relates to a method in the field of stereoscopic image quality evaluation, in particular to a no-reference stereoscopic image quality evaluation method based on eye dominance theory and parallax compensation.

背景技术Background technique

3D成像是一个广泛的研究领域，从娱乐(包括视频和游戏)到专业应用(教育，医疗等)。与此同时，立体图像领域也涉及到越来越多的图像处理技术，对于高效地立体图像质量评估的需求也越来越明显。尽管近年来在2D图像，视频的质量评价方面已经取得了相当大的成果，在3D图像质量评价上的进展仍是屈指可数，这主要是因为3D图像与2D图像的视觉感知有着很大不同。现有的3D质量评价方法可分为全参考，半参考和无参考三种。3D imaging is a broad field of research ranging from entertainment (including video and gaming) to professional applications (education, medical, etc.). At the same time, more and more image processing technologies are involved in the field of stereoscopic images, and the demand for efficient stereoscopic image quality assessment is becoming more and more obvious. Although considerable achievements have been made in the quality evaluation of 2D images and videos in recent years, the progress in 3D image quality evaluation is still very small, mainly because the visual perception of 3D images is very different from that of 2D images. . The existing 3D quality evaluation methods can be divided into three types: full reference, semi-reference and no reference.

对于全参考3D质量评价方法，Benoit等人2008年在《EURASIPJournalonImageandVideoProcessing》上发表了“Qualityassessmentofstereoscopicimages”，它不仅利用SSIM计算出左右眼2D图像的质量分数，而且考虑了视差信息(Disparityinformation)。Ming-JunChen等人2013年在《SignalProcessing:Imagecommunication》上发表了“Full-referencequalityassessmentofstereopairsaccountingforrivalry”,它试图建立起人脑中的单眼图像(cyclopeanimage)。然而这些全参考评价方法需要全部的原始立体图像信息，这在实际应用中很难满足。因此，无参考的3D质量评价有着更大的应用前景与实际意义，但与此同时也面临着更多的困难。For the full reference 3D quality evaluation method, Benoit et al. published "Quality assessment of stereoscopic images" in "EURASIP Journalon Image and Video Processing" in 2008. It not only uses SSIM to calculate the quality scores of left and right eye 2D images, but also considers disparity information (Disparity information). Ming-Jun Chen et al. published "Full-reference quality assessment of stereo pairs accounting for rivalry" in "Signal Processing: Image communication" in 2013, which tried to establish a monocular image (cyclopean image) in the human brain. However, these full-reference evaluation methods require all the original stereo image information, which is difficult to meet in practical applications. Therefore, 3D quality evaluation without reference has greater application prospects and practical significance, but at the same time it faces more difficulties.

为此，Akhter等人2010年在《ProceedingsofProc.SPIE》上发表了“No-referencestereoscopicimagequalityassessment”，提出了一种无参考的3D质量评价方法，它不仅结合了传统的失真类型(块效应)，而且将视差图(Disparitymap)考虑在内。此方法符合人眼对3D图像的视觉感知。但是它十分依赖于对视差图的估计，而目前对于视差图还缺乏深入的理解，准确地计算出视差图也是非常困难的，因此这种方法效果一般。For this reason, Akhter et al. published "No-reference stereoscopic image quality assessment" on "Proceedings of Proc. SPIE" in 2010, and proposed a no-reference 3D quality assessment method, which not only combines traditional distortion types (blocking effect), but also Disparitymap is taken into account. This method conforms to the visual perception of 3D images by human eyes. However, it is very dependent on the estimation of the disparity map, and there is still a lack of in-depth understanding of the disparity map, and it is very difficult to accurately calculate the disparity map, so this method is not effective.

发明内容Contents of the invention

针对现有技术中的缺陷，本发明的目的是提供一种基于眼优势理论(ODI)和视差补偿(Parallaxcompensation)的立体图像质量评价方法(ODDM4)，该方法有效地提升了3D图像质量评价的准确性。For the defects in the prior art, the object of the present invention is to provide a stereoscopic image quality evaluation method (ODDM4) based on eye dominance theory (ODI) and parallax compensation (Parallaxcompensation), which effectively improves the quality evaluation of 3D images. accuracy.

为实现以上目的，本发明所述的基于眼优势理论和视差补偿的立体图像质量评价方法，包括以下步骤：In order to achieve the above object, the stereoscopic image quality evaluation method based on eye dominance theory and parallax compensation according to the present invention comprises the following steps:

第一步，利用2D无参考质量评价方法计算出左眼右眼图像的质量分数，得到Q(L)，Q(R)；The first step is to use the 2D no-reference quality evaluation method to calculate the quality scores of the left-eye and right-eye images, and obtain Q(L), Q(R);

第二步，计算两幅2D图像质量之间的眼优势差异，得到：H(L，R)；The second step is to calculate the eye dominance difference between the two 2D image qualities, and get: H(L, R);

第三步，计算不同视差程度的补偿量，得到d^θ；The third step is to calculate the compensation amount of different parallax degrees to obtain d ^θ ;

第四步，将左眼右眼图像的质量分数Q(L)、Q(R)与视差程度的补偿量d^θ相加，并减去眼优势差异H(L，R)，结果即为基于眼优势理论和视差补偿的无参考3D图像质量评价分数。The fourth step is to add the quality scores Q(L) and Q(R) of the left-eye and right-eye images to the compensation amount d ^θ of the parallax degree, and subtract the eye dominance difference H(L, R), the result is based on No-reference 3D image quality assessment scores for eye dominance theory and parallax compensation.

本发明的原理是：尽管人眼对2D、3D图像的视觉感知有所不同，但2D、3D图像的质量仍有着紧密的联系。本发明利用眼优势理论，与此同时，由于了图像内容与3D质量也有关系，因此本发明通过左右眼图像的视差偏移程度，以补偿图像内容对3D图像质量的影响。最后，本发明将左右眼2D图像质量，眼优势差异，视差偏移补偿结合起来，通过线性组合得到了新的3D无参考的质量评价分数。The principle of the present invention is: although human eyes have different visual perceptions of 2D and 3D images, the quality of 2D and 3D images is still closely related. The present invention utilizes the theory of eye dominance. At the same time, since the image content is also related to the 3D quality, the present invention compensates the impact of the image content on the 3D image quality through the degree of parallax shift of the left and right eye images. Finally, the present invention combines the left and right eye 2D image quality, eye dominance difference, and parallax offset compensation, and obtains a new 3D quality evaluation score without reference through linear combination.

与现有技术相比，本发明具有如下的有益效果：Compared with the prior art, the present invention has the following beneficial effects:

本发明利用眼优势理论，建立了2D图像质量评价和3D图像质量评价方法的联系。除此之外，本发明计算出了左右眼图像的视差偏移程度，以补偿图像内容对3D图像质量的影响。根据在现有的Toyama数据库上的测试结果来看，ODDM4方法取得了优越的3D图像质量评价效果，并且视差偏移补偿有效地提升了3D图像质量评价的准确性。The invention utilizes the eye dominance theory to establish the connection between the 2D image quality evaluation method and the 3D image quality evaluation method. In addition, the present invention calculates the degree of parallax offset of the left and right eye images, so as to compensate the influence of image content on the quality of 3D images. According to the test results on the existing Toyama database, the ODDM4 method has achieved superior 3D image quality evaluation results, and the parallax offset compensation has effectively improved the accuracy of 3D image quality evaluation.

附图说明Description of drawings

通过阅读参照以下附图对非限制性实施例所作的详细描述，本发明的其它特征、目的和优点将会变得更明显：Other characteristics, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

图1是本发明实施例总体流程图；Fig. 1 is the overall flowchart of the embodiment of the present invention;

图2是仅利用2D质量评价分数和眼优势理论得到的3D质量分数(ODDM3)，在Toyama图像质量数据库上与主观评价值MOS对比图；Figure 2 is a comparison chart of the 3D quality score (ODDM3) obtained by using only the 2D quality evaluation score and eye dominance theory, and the subjective evaluation value MOS on the Toyama image quality database;

图3是ODDM3经过视差补偿d^θ后，得到最终的3D质量分数(ODDM4)，在Toyama图像质量数据库上与主观评价值MOS对比图。Figure 3 is a comparison chart of the final 3D quality score (ODDM4) obtained by ODDM3 after parallax compensation d ^θ , compared with the subjective evaluation value MOS on the Toyama image quality database.

具体实施方式detailed description

下面结合具体实施例对本发明进行详细说明。以下实施例将有助于本领域的技术人员进一步理解本发明，但不以任何形式限制本发明。应当指出的是，对本领域的普通技术人员来说，在不脱离本发明构思的前提下，还可以做出若干变形和改进。这些都属于本发明的保护范围。The present invention will be described in detail below in conjunction with specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

本实施例提供一种基于眼优势理论和视差补偿的立体图像质量评价方法，整个过程的流程如图1所示，包括：This embodiment provides a stereoscopic image quality evaluation method based on eye dominance theory and parallax compensation. The flow of the whole process is shown in FIG. 1 , including:

第一步、分别计算左眼右眼的2D质量评价分数，得到Q(L)，Q(R)；具体方法如下：The first step is to calculate the 2D quality evaluation scores of the left and right eyes respectively to obtain Q(L) and Q(R); the specific method is as follows:

i)分别利用下述公式计算左右眼图像L，R水平方向上的块效应D_h，水平方向上的模糊效应A_h，Z_h：i) Use the following formulas to calculate the block effect D _h in the horizontal direction of the left and right eye images L and R, and the blurring effects A _h and Z _h in the horizontal direction:

${A A}_{b b} = = \frac{11}{B B - - 11} [[((\frac{B B}{M m ((N N - - 11))} {Σ Σ}_{i i = = 11}^{M m} {Σ Σ}_{j j = = 11}^{N N - - 11} | | {d d}_{b b} ((i i,, j j)) | |)) - - {D D.}_{b b}]] \cdot \cdot$

${Z Z}_{b b} = = \frac{11}{M m ((N N - - 22))} {Σ Σ}_{i i = = 11}^{M m} {Σ Σ}_{j j = = 11}^{N N - - 22} {z z}_{b b} ((i i,, j j)),,$

其中：M，N分别指图像的高度和宽度，B指图像块的大小；d_h、z_h分别指图像特征的计算函数；i,j分别指图像的第i行和第j列；B_j指的是第j个图像块；Among them: M, N refer to the height and width of the image, B refers to the size of the image block; d _h , z _h refer to the calculation function of the image features; i, j refer to the i-th row and j-th column of the image respectively; B _j Refers to the jth image block;

ii)根据同样的方法，计算出左右眼图像L，R垂直方向上的块效应D_v，图像垂直方向上的模糊效应A_v，Z_v；全部特征即可通过计算水平和垂直特征的平均值得到：ii) According to the same method, calculate the block effect D _v in the vertical direction of the left and right eye images L and R, and the blurring effect A _v and Z _v in the vertical direction of the image; all features can be obtained by calculating the average value of the horizontal and vertical features arrive:

$D D. = = \frac{{D D.}_{h h} + + {D D.}_{v v}}{22},, A A = = \frac{{A A}_{h h} + + {A A}_{v v}}{22},, Z Z = = \frac{{Z Z}_{h h} + + {Z Z}_{v v}}{22}$

iii)最后，所有特征通过一个全局非线性方程进行拟合，得到左右眼2D图像的质量分数：iii) Finally, all features are fitted by a global non-linear equation to obtain the quality scores of the left and right eye 2D images:

Q＝α+βD^γ1A^γ2Z^γ3，Q=α+βD ^γ1 A ^γ2 Z ^γ3 ,

其中：α、β、γ1、γ2和γ3是模型参数，在图像数据库上训练可得，在一实施例中，可Wherein: α, β, γ1, γ2 and γ3 are model parameters, which can be trained on the image database, and in one embodiment, can

以取值：α＝-245.8909；β＝261.9373；γ1＝-239.8886；γ2＝160.1664；γ3＝64.2859。Take values: α=-245.8909; β=261.9373; γ1=-239.8886; γ2=160.1664; γ3=64.2859.

第二步、计算左右眼之间的眼优势差异H(L，R)，具体方法如下：The second step is to calculate the eye dominance difference H(L, R) between the left and right eyes. The specific method is as follows:

所述的眼优势差异的计算方法：The calculation method of the eye dominance difference is as follows:

H(L，R)＝ODI·||Q(L)-Q(R)||H(L, R)=ODI·||Q(L)-Q(R)||

其中：Q(L)，Q(R)分别为第一步中计算出的左眼，右眼图像质量分数。in: Q(L), Q(R) are the left-eye and right-eye image quality scores calculated in the first step, respectively.

第三步、计算视差的补偿量d^θ，具体方法如下：The third step is to calculate the compensation amount d ^θ of parallax, the specific method is as follows:

所述的视差程度的补偿量d^θ：The compensation amount d ^θ of the parallax degree:

${d d}^{θ θ} = = {cos cos}^{- - 11} ((\frac{L L ((M m)) \cdot \cdot R R ((M m))}{{| | | | L L ((M m)) | | | |}_{22} \cdot \cdot {| | | | R R ((M m)) | | | |}_{22}}))$

其中：L，R分别为左眼、右眼图像；M代表相应图像的中央区域。Among them: L, R are the left-eye and right-eye images respectively; M represents the central area of the corresponding image.

第四步、将左右眼图像的2D质量分数，眼优势差异与视差补偿进行线性结合，得到最终的3D无参考图像质量分数ODDM4；具体方法如下：The fourth step is to linearly combine the 2D quality scores of the left and right eye images, the eye dominance difference and the parallax compensation to obtain the final 3D no-reference image quality score ODDM4; the specific method is as follows:

所述的无参考3D图像质量评价分数：The no-reference 3D image quality rating score:

ODDM4＝Q(L)+Q(R)-H(L，R)+d^θ ODDM4=Q(L)+Q(R)-H(L, R)+d ^θ

其中：Q(L)，Q(R)由第一步获得，H(L，R)由第二步获得，d^θ由第三步获得。Among them: Q(L), Q(R) is obtained by the first step, H(L, R) is obtained by the second step, d ^θ is obtained by the third step.

实施效果Implementation Effect

依据上述步骤，本发明对Toyama3D数据库的所有3D图像进行了无参考质量评价，并利用视频质量专家组(VQEG)推荐的指标与[1]方法进行了比较，指标包括：PearsonLinearCorrelationCoefficient(PLCC)、SpearmanRank-orderCorrelationCoefficient(SRCC)、AverageAbsoluteError(AAE)以及RootMean-SquaredError(RMSE)。比较结果见表1，可以发现本发明的ODDM4准确性优于[1]方法。ODDM3在Toyama3D图像质量数据库上与主观评价值MOS比较图如图2所示，ODDM4与主观评价值MOS的比较图如图3所示。According to the above steps, the present invention has carried out no reference quality evaluation to all 3D images of Toyama3D database, and utilizes the index recommended by Video Quality Experts Group (VQEG) to compare with [1] method, and index comprises: PearsonLinearCorrelationCoefficient (PLCC), SpearmanRank -orderCorrelationCoefficient(SRCC), AverageAbsoluteError(AAE), and RootMean-SquaredError(RMSE). The comparison results are shown in Table 1, and it can be found that the accuracy of ODDM4 of the present invention is better than that of [1]. The comparison chart of ODDM3 and subjective evaluation value MOS on the Toyama3D image quality database is shown in Figure 2, and the comparison chart of ODDM4 and subjective evaluation value MOS is shown in Figure 3.

表1Table 1

方法method PLCCPLCC SRCCSRCC AAEAAAE RMSERMSE ODDM1ODDM1 0.89340.8934 0.88970.8897 0.41360.4136 0.52340.5234 ODDM2ODDM2 0.92070.9207 0.90760.9076 0.35940.3594 0.45480.4548 ODDM3ODDM3 0.93180.9318 0.92660.9266 0.33580.3358 0.42280.4228 ODDM4ODDM4 0.95780.9578 0.95000.9500 0.25940.2594 0.33540.3354 [1][1] 0.93500.9350 —— 0.35000.3500 0.42100.4210

表1中所述的图像质量评价方法具体是指：The image quality evaluation methods described in Table 1 specifically refer to:

ODDM1：ODDM1＝Q(L)+Q(R)；ODDM1: ODDM1=Q(L)+Q(R);

ODDM2：ODDM2＝Q(L)+Q(R)-||Q(L)-Q(R)||_∞；ODDM2: ODDM2=Q(L)+Q(R)-||Q(L)-Q(R)|| _∞ ;

ODDM3：ODDM3＝Q(L)+Q(R)-ODI·||Q(L)-Q(R)||_∞， ODDM3: ODDM3=Q(L)+Q(R)-ODI·||Q(L)-Q(R)|| _∞ ,

ODDM4：ODDM4＝Q(L)+Q(R)-ODI·||Q(L)-Q(R)||_∞+d^θ，其中：ODDM4: ODDM4=Q(L)+Q(R)-ODI||Q(L)-Q(R)|| _∞+ d ^θ , where:

[1]：Akhter等人在2010年《ProceedingsofProc.SPIE》上发表的“Noreferencestereoscopicimagequalityassessment”。[1]: "Noreference stereoscopic image quality assessment" published by Akhter et al. in "Proceedings of Proc. SPIE" in 2010.

根据在现有的Toyama数据库上的测试结果来看，本发明ODDM4方法取得了优越的3D图像质量评价效果，并且视差偏移补偿有效地提升了3D图像质量评价的准确性。值得注意的是，本发明建立了2D与3D图像质量评价的联系。任何2D质量评价方法的提高，都可能通过本发明改善相应的3D质量评价效果。本发明中的视差补偿方法也为将来3D质量评价方法的提升指明了方向。According to the test results on the existing Toyama database, the ODDM4 method of the present invention achieves a superior 3D image quality evaluation effect, and the parallax offset compensation effectively improves the accuracy of the 3D image quality evaluation. It is worth noting that the present invention establishes a connection between 2D and 3D image quality evaluation. The improvement of any 2D quality evaluation method may improve the corresponding 3D quality evaluation effect through the present invention. The parallax compensation method in the present invention also points out the direction for the improvement of the 3D quality evaluation method in the future.

以上对本发明的具体实施例进行了描述。需要理解的是，本发明并不局限于上述特定实施方式，本领域技术人员可以在权利要求的范围内做出各种变形或修改，这并不影响本发明的实质内容。Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for evaluating the quality of stereoscopic images based on eye dominance theory and parallax compensation, characterized in that said method may further comprise the steps:

The first step is to use the 2D no-reference quality evaluation method to calculate the quality scores of the left-eye and right-eye images, and obtain Q(L), Q(R);

The second step is to calculate the eye dominance difference between the two 2D image qualities, and get: H(L, R);

The third step is to calculate the compensation amount of different parallax degrees, and obtain d ^θ , where θ represents the angle;

The fourth step is to add the quality scores Q(L) and Q(R) of the left-eye and right-eye images to the compensation amount d ^θ of the parallax degree, and subtract the eye dominance difference H(L, R), the result is based on No-reference 3D image quality assessment scores for eye dominance theory and parallax compensation.

2. a kind of stereoscopic image quality evaluation method based on eye dominance theory and parallax compensation according to claim 1, is characterized in that, in the first step, the computing method of described left eye right eye image quality fraction:

i) Use the following formulas to calculate the block effect D _h in the horizontal direction of the left and right eye images L and R, and the blurring effects A _h and Z _h in the horizontal direction:

{A A}_{h h} = = \frac{11}{B B - - 11} [[((\frac{B B}{M m ((N N - - 11))} {Σ Σ}_{i i = = 11}^{M m} {Σ Σ}_{j j = = 11}^{N N - - 11} | | {d d}_{h h} ((i i,, j j)) | |)) - - {D D.}_{h h}]] . .

Among them: M, N refer to the height and width of the image, B refers to the size of the image block; d _h , z _h refer to the calculation function of the image features; i, j refer to the i-th row and j-th column of the image respectively; B _j Refers to the jth image block;

ii) According to the same method, calculate the blocking effect D _v in the vertical direction of the left and right eye images L and R, the blurring effect A _v and Z _v in the vertical direction of the image, and calculate the average value of the horizontal and vertical features to get all the features:

\begin{matrix} D D. = = \frac{{D D.}_{h h} + + {D D.}_{v v}}{22} & A A = = \frac{{A A}_{h h} + + {A A}_{v v}}{22} & Z Z = = \frac{{Z Z}_{h h} + + {Z Z}_{v v}}{22} \end{matrix}

Dh, Ah, Zh are features in the vertical direction, Zv, Dv, Av are features in the horizontal direction;

iii) Finally, all features are fitted by a global non-linear equation to obtain the quality scores of the left and right eye 2D images:

Q＝α+βD ^γ1 A ^γ2 Z ^γ3

Among them: α, β, γ ₁ , γ ₂ and γ ₃ are model parameters, which are obtained by training on the image database.

3. a kind of stereoscopic image quality evaluation method based on eye dominance theory and parallax compensation according to claim 1, is characterized in that, in the second step, the computing method of described eye dominance difference:

H(L, R)=ODI·||Q(L)-Q(R)||

in: Q(L), Q(R) are the left-eye and right-eye image quality scores calculated in the first step, respectively.

4. A kind of stereoscopic image quality evaluation method based on eye dominance theory and parallax compensation according to any one of claims 1-3, characterized in that, in the third step, the compensation amount d ^θ of the parallax degree:

{d d}^{θ θ} = = {cos cos}^{- - 11} ((\frac{L L ((M m)) \cdot &Center Dot; R R ((M m))}{| | | | L L ((M m)) | | {| |}_{22} \cdot &Center Dot; | | | | R R ((M m)) | | {| |}_{22}}))

Among them: L, R are left-eye and right-eye images respectively; M represents the central area of the image.