CN102800092B

CN102800092B - Point-to-surface image significance detection

Info

Publication number: CN102800092B
Application number: CN201210239939.4A
Authority: CN
Inventors: 张萌萌
Original assignee: North China University of Technology
Current assignee: North China University of Technology
Priority date: 2012-07-12
Filing date: 2012-07-12
Publication date: 2015-01-07
Anticipated expiration: 2032-07-12
Also published as: CN102800092A

Abstract

The invention relates to a point-to-plane image saliency detection method, device and computer program product.

Description

Image saliency detection from point to area

联合研究 joint research

本申请由北方工业大学与北京交通大学信息所联合研究，并得到以下基金资助：国家自然科学基金(No.61103113)，北京市属高等学校人才强教深化计划项目(PHR201008187)。 This application is jointly researched by North China University of Technology and the Information Institute of Beijing Jiaotong University, and supported by the following funds: National Natural Science Foundation of China (No.61103113), Beijing Municipal Higher Education Intensification Program for Talents (PHR201008187). the

技术领域 technical field

本发明涉及基于一种从点至面的图像显著性检测方法、装置和计算机程序产品。 The invention relates to a point-to-plane image saliency detection method, device and computer program product. the

背景技术 Background technique

图像的显著性检测对于图像信息提取有着重要作用。在绝大多数基于内容的图像处理中都或多或少需要提取图像中的显著部分，因为图像的显著区域相对提供了整幅图像的所要表达的大部分信息。因此准确的提取图像中的显著区域对诸如基于内容的图像检索、图像自适应压缩、物体识别、图像适配等图像处理工作有极大的帮助。 Image saliency detection plays an important role in image information extraction. In most content-based image processing, it is more or less necessary to extract the salient parts of the image, because the salient regions of the image relatively provide most of the information to be expressed in the entire image. Therefore, accurate extraction of salient regions in images is of great help to image processing tasks such as content-based image retrieval, image adaptive compression, object recognition, and image adaptation. the

人类总是能够根据自己的经验及判断很容易的找到图像中显著区域，从而准确获取图像所传达的信息。然而对于机器来说，智能识别出图像中的显著区域就没有那么容易了。多年来许多学者进行了基于生物、生理、神经系统的研究，获取了图像的显著区域应该具有的一些特征，这些特征包括：独特性、随机性、奇异性等。由此产生了自顶向下和自底向上的视觉显著性机制。 Humans can always easily find the salient areas in the image according to their own experience and judgment, so as to accurately obtain the information conveyed by the image. However, it is not so easy for a machine to intelligently identify salient areas in an image. Over the years, many scholars have conducted research based on biology, physiology, and nervous system, and obtained some characteristics that the salient regions of the image should have, including: uniqueness, randomness, singularity, etc. From this arises top-down and bottom-up visual saliency mechanisms. the

图像视觉显著性检测在最近的二十年经历了快速的发展。诞生了各种各样优秀的方法，不同的方法有不同的侧重点，也取得了在某些方面出色的效果。 Image visual saliency detection has experienced rapid development in the last two decades. Various excellent methods have been born, different methods have different emphases, and have achieved excellent results in some aspects. the

其中，Koch和Ullman等人(文献[7])提出的生物视觉启发模型影响了一大批基于图像基本特征的显著性检测算法。Itti等人(文献[8])定义了图像的显著性。通过分析图像的Intensity、Color和Orientation等视觉特征计算图像的显著性特征图。但由于该方法只对图像进行了整体的分析，所以在显著定位上能取得较好的效果，但显著区域的细节定位略显不足。S.Goferman等人(文献[5])的方法则开始对图像的局部、全局综合考虑，相比Itti等人在局部的细节定位有很大进步，但对于均匀的突出整个显著区域做的还不够优秀。 Among them, the biological visual inspiration model proposed by Koch and Ullman et al. (document [7]) has influenced a large number of saliency detection algorithms based on the basic features of images. Itti et al. ([8]) defined image saliency. The saliency feature map of the image is calculated by analyzing the visual features such as Intensity, Color and Orientation of the image. However, since this method only analyzes the image as a whole, it can achieve better results in saliency positioning, but the detail locating of salient areas is slightly insufficient. The method of S.Goferman et al. (document [5]) began to consider the local and global comprehensive consideration of the image. Compared with Itti et al., it has made great progress in local detail positioning, but it still does not do enough to uniformly highlight the entire salient area. Not good enough. the

在生物视觉的基础上，一些算法开始考虑加入一些数学上模型进行建模。Harel等人(文献[9])和Gopalakrishnan等人(文献[15])利用马尔科夫随机场模型对图像的基本特征进行了处理。而Duan等人(文献[16])的方法则利用PCA对图像的颜色空间进行转换和降维处理。Li等人(文献[17])使用了基于稀疏编码长度的求解方式计算图像的显著区域。该方法将显著性视为编码长度的直接体现，给出了影响视觉显著性的成因的一种可能解释。这些方法均能取得一定程度的优秀效果，且扩大适用的图像类型的范围，但数学模型的引入给算法带来了运算复杂度的增加。 On the basis of biological vision, some algorithms began to consider adding some mathematical models for modeling. Harel et al. (document [9]) and Gopalakrishnan et al. (document [15]) used the Markov random field model to process the basic features of the image. The method of Duan et al. (literature [16]) uses PCA to convert and reduce the dimensionality of the color space of the image. Li et al. (document [17]) used a solution based on the length of the sparse code to calculate the salient region of the image. This method regards saliency as a direct reflection of code length, and gives a possible explanation for the cause of visual saliency. These methods can achieve a certain degree of excellent results, and expand the range of applicable image types, but the introduction of mathematical models brings an increase in computational complexity to the algorithm. the

对于大量图像的显著性检测，Liu等人(文献[6])和Judd等人(文献[1])给出了系统性的检测方法。两种方法均加入的机器学习的过程。Liu等人的方法使用了CRF(Conditional Random Filed)建立了机器学习的模型。两种方法同时都对图像进行了人工测试显著区域进行显著特征的采集。毫无疑问，这类机器学习的方法无论对于处理图像的类型还是显著性检测的准确性都有较强的鲁棒性。对于场景及其复杂的的图像，两者也能做出准确的判断。但是系统级的检测所需要的复杂的设备及计算代价却不适合很多的实时显著性检测场合。而且对于日常的自然图像，这种复杂度的增加所带来的检测性能的提升并不是线性关系。 For the saliency detection of a large number of images, Liu et al. (document [6]) and Judd et al. (document [1]) gave a systematic detection method. Both methods are added to the process of machine learning. The method of Liu et al. uses CRF (Conditional Random Filed) to establish a machine learning model. At the same time, the two methods have carried out manual testing on the salient areas of the image to collect salient features. Undoubtedly, this type of machine learning method is robust to both the type of image processing and the accuracy of saliency detection. For scenes and their complex images, both can also make accurate judgments. However, the complex equipment and computational cost required by system-level detection are not suitable for many real-time saliency detection occasions. Moreover, for everyday natural images, the improvement in detection performance brought about by the increase in complexity is not linear. the

相比于通用的图像特征分析，Hou等人(文献[11])在图像的频域进行显著性分析，作者发现大量图像的log频谱的平均值和频率呈现正比关系。通过一幅图像的log振幅谱减去平均log振幅谱得到图像的显著部分。而Guo等人(文献[18])的方法认为通过图像傅里叶变换的相位谱能得到更好的显著图检测结果。这方面的分析为图像的显著性检测提供了另一个分析途径。 Compared with general image feature analysis, Hou et al. (Reference [11]) performed saliency analysis in the frequency domain of the image. The authors found that the average value and frequency of the log spectrum of a large number of images showed a proportional relationship. The salient part of an image is obtained by subtracting the mean log amplitude spectrum from the log amplitude spectrum of an image. The method of Guo et al. (document [18]) believes that better saliency map detection results can be obtained through the phase spectrum of the Fourier transform of the image. This aspect of analysis provides another analytical approach for image saliency detection. the

我们的方法分析图像的基本特征，采用定位-定量的分析过程，首先得到可能的显著性点，然后对得到的基准点进行定量分析，完成显著区域的检测。我们的算法综合考虑了图像的局部和全局特征，能充分体现出整个显著区域，同时由于我们并不是逐像素比较计算，因此需要的计算复杂度较少。 Our method analyzes the basic features of the image, adopts the localization-quantitative analysis process, first obtains the possible salient points, and then conducts quantitative analysis on the obtained benchmark points to complete the detection of salient regions. Our algorithm comprehensively considers the local and global features of the image, and can fully reflect the entire salient area. At the same time, since we do not compare and calculate pixel by pixel, it requires less computational complexity. the

具体而言，本文将图像的显著区域检测视为图像像素二值化的随机分布问题。具体来说，对于一幅待分析的图像，我们可以认为图像的所有像素集合是一个由取值只有{1，0}两个取值的序列，也就是说对于每一个像素针对我们的目标只有两种选择，要么属于显著区域，要么不属于显著区域。所以，像素的取值分布具有两个重要的性质：随机性和相关性。随机性是指尽管我们知道图像中可能存在属于显著区域的一组像素集合，但并不知道这些像素的数量、位置及组合方式，也就是显著区域的尺寸、位置和形状。相关性是指尽管像素的分布是随机的，但并不是无规律的完全“自由随机分布”。这些像素总是通过一些互相关联的特征相互影响。比如图像的对比度、多尺度特征、颜色分布特征。 Specifically, this paper treats salient region detection in images as a random distribution problem of binarization of image pixels. Specifically, for an image to be analyzed, we can consider that all pixel sets of the image are a sequence with only two values {1, 0}, that is to say, for each pixel, there is only Two choices, either belong to the salient area, or do not belong to the salient area. Therefore, the value distribution of pixels has two important properties: randomness and correlation. Randomness means that although we know that there may be a set of pixels belonging to the salient area in the image, we do not know the number, position and combination of these pixels, that is, the size, position and shape of the salient area. Correlation means that although the distribution of pixels is random, it is not a completely "free random distribution" with no regularity. These pixels always influence each other through some interrelated features. For example, image contrast, multi-scale features, and color distribution features. the

本文的算法建立了由点到面的计算机制。算法首先从信息含量丰富的角点开始，我们通过改进的SUSAN算子获取图像中角点。然后从角点区域结合图像的区域对比度、全局颜色分布等图像像素相关信息得到图像的显著性图。对于显著性检测结果的体现，我们认为应该有两个方面的技术指标：对单幅图像显著区域检测的精确度及对大量图像检测的成功率。我们在国际上通用的数据集上进行结果测试，并与现有的一些代表性方法进行比较，结果表明我们的方法在两个方面的技术指标上都能取得较好的结果。 The algorithm in this paper establishes a computer mechanism from point to surface. The algorithm starts from the corner points with rich information, and we obtain the corner points in the image through the improved SUSAN operator. Then the saliency map of the image is obtained from the corner area combined with the image pixel-related information such as the regional contrast of the image and the global color distribution. For the embodiment of saliency detection results, we believe that there should be two technical indicators: the accuracy of detection of salient regions in a single image and the success rate of detection of a large number of images. We test the results on an international common dataset and compare with some existing representative methods. The results show that our method can achieve better results in both technical indicators. the

本申请是基于以下文献提出的改进算法。 This application is based on the improved algorithm proposed by the following documents. the

[1]T.Judd，K.Ehinger，F.Durand and A.Torralba，“2010Learning to Predict Where Humans Look，”IEEE ICCV.Proc.，pp.2106-2113，Sep.2009. [1] T.Judd, K.Ehinger, F.Durand and A.Torralba, "2010 Learning to Predict Where Humans Look," IEEE ICCV.Proc., pp.2106-2113, Sep.2009.

[2]J.S.Kim，S.G.Jeong，Y.H.Joo and C.S.Kim，“Content-aware image and video resizing based on frequency domain analysis，”IEEE Trans.Consumer Electronics.，Vol.57，no.1，pp.615-622，July.2011. [2] J.S.Kim, S.G.Jeong, Y.H.Joo and C.S.Kim, "Content-aware image and video resizing based on frequency domain analysis," IEEE Trans.Consumer Electronics., Vol.57, no.1, pp.615-622 , July.2011.

[3]Mengmeng Cheng，GX Zhang，N J.Mitra Xl Huang and SM Hu，“Global contrast based saliency region detection，”，IEEE CVPR，p.409-416，Colorado Springs，Colorado，USA，June 21-23，2011. [3] Mengmeng Cheng, GX Zhang, N J. Mitra Xl Huang and SM Hu, "Global contrast based saliency region detection," IEEE CVPR, p.409-416, Colorado Springs, Colorado, USA, June 21-23, 2011.

[4]Huihui Bai，Ce Zhu and Yao Zhao，“Optimized Multiple Description Lattice Vector Quantization for Wavelet Image Coding，”IEEE Trans.Circuits and Systems for Video Technology.，Vol.17，no.7，pp.912-917，2007. [4] Huihui Bai, Ce Zhu and Yao Zhao, "Optimized Multiple Description Lattice Vector Quantization for Wavelet Image Coding," IEEE Trans. Circuits and Systems for Video Technology., Vol.17, no.7, pp.912-917, 2007.

[5]Goferman S，Zelnik-Manor L and Tal A.“Context-aware saliency detection.”IEEE CVPR.Proc.Pp.2376-2383.San Francisco，2010. [5]Goferman S, Zelnik-Manor L and Tal A. "Context-aware saliency detection." IEEE CVPR.Proc.Pp.2376-2383.San Francisco, 2010.

[6]T.Liu，Z.Yuan，J.Sun，J.Wang，N.Zheng，T.X.，and S.H.Y.“Learning to detect a salient object.”IEEE TPAMI，pp.33(2)：353-367，2011. [6] T.Liu, Z.Yuan, J.Sun, J.Wang, N.Zheng, T.X., and S.H.Y. "Learning to detect a salient object." IEEE TPAMI, pp.33(2): 353-367, 2011.

[7]C.Koch，and S.Ullman，“Shifts in selective visual attention：Towards the underlying neuronal circuitry，”Human Neurobiology，pp.219-227，Apr.1985. [7] C.Koch, and S.Ullman, "Shifts in selective visual attention: Towards the underlying neuronal circuitry," Human Neurobiology, pp.219-227, Apr.1985.

[8]Itti，Koch and Niebur，“A model of saliency-based visual attention for rapid scene analysis，”IEEE Trans.Pattern Analysis and Machine Intelligence.，Vol.20，pp.1254-1259，Nov.1998. [8] Itti, Koch and Niebur, "A model of saliency-based visual attention for rapid scene analysis," IEEE Trans.Pattern Analysis and Machine Intelligence., Vol.20, pp.1254-1259, Nov.1998.

[9]J.Harel，C.Koch and P.Perona，“Graph-Based Visual Saliency，”Advances in Neural Information Processing Systems，2006，pp.545-552. [9] J.Harel, C.Koch and P.Perona, "Graph-Based Visual Saliency," Advances in Neural Information Processing Systems, 2006, pp.545-552.

[10]R.Achanta，S.Hemami，F.Estrada and S.Susstrunk，“Frequency-tuned salient region detection，”Proc.，pp.1597-1604，June.2009. [10] R.Achanta, S.Hemami, F.Estrada and S.Susstrunk, "Frequency-tuned salient region detection," Proc., pp.1597-1604, June.2009.

[11]Xiaodi Hou，Liqing Zhang，“Saliency detection：A spectral residual approach，”IEEE CVPR.Proc.，pp.1-8，June.2007. [11] Xiaodi Hou, Liqing Zhang, "Saliency detection: A spectral residual approach," IEEE CVPR.Proc., pp.1-8, June.2007.

[12]Smith，M.Stephen，Brady and J.Michael，“SUSAN-a new approach to low level image processing，”International Journal of Computer Vision，Vol.23，no.1，pp.45-78，May.1997. [12]Smith, M.Stephen, Brady and J.Michael, "SUSAN-a new approach to low level image processing," International Journal of Computer Vision, Vol.23, no.1, pp.45-78, May. 1997.

[13]C.Harris，and M.Stephens，“A Combined Corner and Edge Detector，”Alvey Vision Conf，Univ.Manchester，1988. [13]C.Harris, and M.Stephens, "A Combined Corner and Edge Detector," Alvey Vision Conf, Univ.Manchester, 1988.

[14]U.Rutishauser，D.Walther，C.Koch，and P.Perona.“Is bottom-up attention useful for object recognition？”IEEE CVPR，pp.37-44，2004.409 [14] U.Rutishauser, D.Walther, C.Koch, and P.Perona. "Is bottom-up attention useful for object recognition?" IEEE CVPR, pp.37-44, 2004.409

[15]V.Gopalakrishnan，Y.Hu and D.Rajan.“Random walks on graphs to model saliency in images.IEEE CVPR.Proc.2009. [15]V.Gopalakrishnan, Y.Hu and D.Rajan. "Random walks on graphs to model saliency in images.IEEE CVPR.Proc.2009.

[16]Lijuan Duan，Chunpeng Wu，Jun Miao，Laiyun Qing and Yu Fu.“Visual Saliency Detection by Spatially Weighted Dissimilarity.”IEEE CVPR，Colorado Springs，USA，pp.473-480，Jun.2011. [16] Lijuan Duan, Chunpeng Wu, Jun Miao, Laiyun Qing and Yu Fu. "Visual Saliency Detection by Spatially Weighted Dissimilarity." IEEE CVPR, Colorado Springs, USA, pp.473-480, Jun.2011.

[17]Yin Li，Yue Zhou，Lei Xu，Jie Yang and Xiaochao Yang.“Incremental Sparse Saliency Detection，”IEEE International Conference on Image Processing(ICIP)，2009. [17] Yin Li, Yue Zhou, Lei Xu, Jie Yang and Xiaochao Yang. "Incremental Sparse Saliency Detection," IEEE International Conference on Image Processing (ICIP), 2009.

[18]C.Guo，Q.Ma and L Zhang.“Spatio-temporal saliency detection using phase spectrum of quaternion fourier transform.”IEEE CVPR.2008. [18]C.Guo, Q.Ma and L Zhang. "Spatio-temporal saliency detection using phase spectrum of quaternion fourier transform." IEEE CVPR.2008.

[19]Ozyer GT，and Vural FY，“A Content-Based Image Retrieval system using Visual Attention，”IEEE SIU.Proc.，pp.399-402，April.2010. [19] Ozyer GT, and Vural FY, "A Content-Based Image Retrieval system using Visual Attention," IEEE SIU.Proc., pp.399-402, April.2010.

[20]Dewen Zhuang，and Shoujue Wang，“Content-Based Image Retrieval Based on Integrating Region Segmentation and Relevance Feedback，”IEEE ICMT.Proc.，pp.1-3，Oct.2010. [20] Dewen Zhuang, and Shoujue Wang, "Content-Based Image Retrieval Based on Integrating Region Segmentation and Relevance Feedback," IEEE ICMT.Proc., pp.1-3, Oct.2010.

[21]Zhihong Chen，Zhiyong Feng and Yongxin Yu，“Image retrieve using visual attention weight model，”IEEE ICSESS.Proc.，pp.718-721，July.2011. [21] Zhihong Chen, Zhiyong Feng and Yongxin Yu, "Image retrieval using visual attention weight model," IEEE ICSESS.Proc., pp.718-721, July.2011.

附图说明 Description of drawings

图1示出了根据本发明的一个实施例的图像处理系统； Fig. 1 shows an image processing system according to an embodiment of the present invention;

图2示出了根据本发明的一些实施例的从点至面的图像显著性检测的流程图； Figure 2 shows a flow chart of image saliency detection from point to surface according to some embodiments of the present invention;

图3示出了根据本发明的一些实施例的从点至面的图像显著性检测装置；以及 Figure 3 shows a point-to-plane image saliency detection device according to some embodiments of the present invention; and

图4示出了根据一些实施例的示例性处理结果图。 Figure 4 illustrates an exemplary processing results graph, according to some embodiments. the

具体实施方式 Detailed ways

现在参考附图来描述各种方案。在以下描述中，为了进行解释，阐述了多个具体细节以便提供对一个或多个方案的透彻理解。然而，显然，在没有这些具体细节的情况下也能够实现这些方案。 Various aspects are now described with reference to the figures. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. It may be evident, however, that such aspects can be practiced without these specific details. the

如在本申请中所使用的，术语“组件”、“模块”、“系统”等等旨在指代与计算机相关的实体，例如但不限于，硬件、固件、硬件和软件的组合、软件，或者是执行中的软件。例如，组件可以是但不限于：在处理器上运行的进程、处理器、对象、可执行体(executable)、执行线程、程序、和/ 或计算机。举例而言，运行在计算设备上的应用程序和该计算设备都可以是组件。一个或多个组件可以位于执行进程和/或者执行线程内，并且组件可以位于一台计算机上和/或者分布在两台或更多台计算机上。另外，这些组件可以从具有存储在其上的各种数据结构的各种计算机可读介质执行。组件可以借助于本地和/或远程进程进行通信，例如根据具有一个或多个数据分组的信号，例如，来自于借助于信号与本地系统、分布式系统中的另一组件交互和/或者与在诸如因特网之类的网络上借助于信号与其他系统交互的一个组件的数据。 As used in this application, the terms "component", "module", "system" and the like are intended to refer to a computer-related entity such as, but not limited to, hardware, firmware, a combination of hardware and software, software, Or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. For example, both an application running on a computing device and the computing device can be components. One or more components can reside within a process and/or thread of execution and a component can be localized on one computer and/or distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures stored thereon. Components can communicate by means of local and/or remote processes, such as from signals having one or more data packets, for example, from interacting with another component in a local system, a distributed system, and/or with another component in a distributed system by means of a signal. Data of a component on a network such as the Internet that interacts with other systems by means of signals. the

图1示出了根据本发明的一个实施例的图像处理系统100。装置101为图像采集设备，用于依据现有技术中已知的任何图像采集技术来获取待处理的图像，所采集的图像可以经由通信装置直接传送给图像处理装置103，或者可以存储在存储装置105中以待后续处理。在本发明的一个实施例中，图像采集装置101直接在用户所访问的网页上获取与网页相关联的图像。 FIG. 1 shows an image processing system 100 according to an embodiment of the present invention. The device 101 is an image acquisition device, which is used to obtain images to be processed according to any image acquisition technology known in the prior art, and the collected images can be directly transmitted to the image processing device 103 via a communication device, or can be stored in a storage device 105 for subsequent processing. In one embodiment of the present invention, the image acquisition device 101 directly acquires the image associated with the webpage on the webpage visited by the user. the

由图像采集设备101所采集到的图像通过通信装置102以有线和/或无线的方式传送至图像处理装置103，该图像处理装置103对接收到的图像进行基于边缘的图像显著性检测，以检测图像中的显著物体或其他显著区域。但是应该理解，图像处理装置103还可以对输入图像进行其它各种处理，例如图像去噪、图像配准、模式识别等等。 The image collected by the image acquisition device 101 is transmitted to the image processing device 103 in a wired and/or wireless manner through the communication device 102, and the image processing device 103 performs edge-based image saliency detection on the received image to detect Salient objects or other salient regions in an image. However, it should be understood that the image processing device 103 may also perform various other processing on the input image, such as image denoising, image registration, pattern recognition and so on. the

图像处理装置103可以用通用处理器、数字信号处理器(DSP)、专用集成电路(ASIC)、现场可编程门阵列(FPGA)或其它可编程逻辑器件、分立门或晶体管逻辑器件、分立硬件组件或者设计为执行本文所述功能的其任意组合，来实现或执行。通用处理器可以是微处理器，但是可替换地，该处理器也可以是任何常规的处理器、控制器、微控制器或者状态机。处理器也可以实现为计算设备的组合，例如，DSP和微处理器的组合、多个微处理器的组合、一个或多个微处理器与DSP内核的组合或者任何其它此种结构。另外，至少一个处理器可以包括可操作以执行上述的一个或多个步骤和/或操作的一个或多个模块。 The image processing device 103 may use a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components Or any combination thereof designed to perform the functions described herein is implemented or performed. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a DSP and a microprocessor, multiple microprocessors, one or more microprocessors with a DSP core, or any other such architecture. Additionally, at least one processor may comprise one or more modules operable to perform one or more of the steps and/or operations described above. the

当用ASIC、FPGA等硬件电路来实现图像处理装置103时，其可以包括被配置为执行各种功能的各种电路块。本领域技术人员可以根据施加在整个系统上的各种约束条件来以各种方式设计和实现这些电路，来实现本发明所公开的各种功能。例如，用ASIC、FPGA等硬件电路实现的图像处理装置103可以包括图像显著性检测电路及/或其它电路模块，其用来依据本文公开的各种图像显著性检测方案来对输入图像执行图像显著性检测。本领域技术人员应该可以理解和认识到，本文所述的图像处理装置103可选地可以包括除图像显著性检测电路之外的其它任何可用电路模块，例如被配置为进行边缘检测、图像配准、模式识别的任何电路模块。以下结合图3的流程图详细描述了滤波器电路所实现的功能。 When the image processing device 103 is implemented with a hardware circuit such as ASIC, FPGA, it may include various circuit blocks configured to perform various functions. Those skilled in the art can design and implement these circuits in various ways according to various constraints imposed on the entire system, so as to realize various functions disclosed in the present invention. For example, the image processing device 103 implemented by hardware circuits such as ASIC and FPGA may include an image saliency detection circuit and/or other circuit modules, which are used to perform image saliency on the input image according to various image saliency detection schemes disclosed herein. Sex detection. Those skilled in the art should be able to understand and realize that the image processing device 103 described herein may optionally include any other available circuit modules other than the image saliency detection circuit, for example, configured to perform edge detection, image registration, etc. , any circuit module for pattern recognition. The functions realized by the filter circuit are described in detail below in conjunction with the flowchart of FIG. 3 . the

图像存储装置105可以耦合至图像采集设备101及/或图像处理装置103，以存储图像采集设备101所采集的原始数据及/或经过图像处理装置103处理后的输出图像。 The image storage device 105 can be coupled to the image acquisition device 101 and/or the image processing device 103 to store the raw data collected by the image acquisition device 101 and/or the output image processed by the image processing device 103 . the

对于图像的显著性区域检测，本质上的问题即为决定在一幅图像的所有像素中哪些属于显著区域，哪些不属于显著区域。为此对一幅给定的图像I我们定义A＝{a_pix(w，h)}，对任意像素pix(w，h)，a_pix(w，h)∈{1，0}表示该像素是否为显著像素点。对于一幅尺寸为W×H的图像，我们用P(w，h)表示坐标为(w，h)的像素是否属于显著像素的概率，那么P(w，h)∈(0，1)。 For image salient region detection, the essential problem is to decide which of all the pixels in an image belong to the salient region and which do not belong to the salient region. For this reason, we define A={a _{pix(w, h} )} for a given image I, and for any pixel pix(w, h), a _{pix(w, h)} ∈ {1, 0} represents the pixel Whether it is a salient pixel. For an image of size W×H, we use P(w, h) to represent the probability of whether the pixel with coordinates (w, h) belongs to the salient pixel, then P(w, h) ∈ (0, 1).

我们针对的目标为通常的自然图像，对于一幅含有显著性区域的自然图像，其中的显著区域包含了该图像想要传达的大部分信息。当然图像背景对于整幅图像内容的表达也有很大贡献，这就像红花和绿叶的关系。作为图像中的单个像素点，不同的像素点含有的能量也是不同的，即P(w，h)不同，因此我们从一些P(w，h)取值较大的像素点开始算法的进行。 Our target is the usual natural image. For a natural image with salient regions, the salient regions contain most of the information that the image wants to convey. Of course, the image background also contributes a lot to the expression of the whole image content, just like the relationship between red flowers and green leaves. As a single pixel in the image, different pixels contain different energies, that is, P(w, h) is different, so we start the algorithm from some pixels with larger values of P(w, h). the

以下描述根据本发明的采用改进的SUSAN算法的角点检测。 Corner detection using the improved SUSAN algorithm according to the present invention is described below. the

对于图像中像素点的信息含量，一般认为在一幅图像中出现频率越小的像素取值，其能量越高。而对于图像中一些能量较高，信息含量丰富点的检测已有多种多样的方法，比如SUSAN角点(文献[12])，Harris角点(汶文献[13])，尺度旋转不变特征点等。在本文的方法中为了减少前期的计算量，我们使用改进的SUSAN角点检测的方法获取图像中P(w，h)可能会较大的点。 Regarding the information content of pixels in an image, it is generally believed that the smaller the value of a pixel in an image, the higher its energy. For the detection of some points with high energy and rich information content in the image, there are various methods, such as SUSAN corner point (literature [12]), Harris corner point (Wen literature [13]), scale rotation invariant feature wait. In the method of this paper, in order to reduce the amount of calculation in the early stage, we use the improved SUSAN corner detection method to obtain points in the image where P(w, h) may be larger. the

常规的SUSAN角点检测采用一个圆形模板在图像中移动，并在每个位置上比较模板内各像素的亮度与模板核的亮度，比较式如下： Conventional SUSAN corner detection uses a circular template to move in the image, and compares the brightness of each pixel in the template with the brightness of the template core at each position. The comparison formula is as follows:

$D D. ((r r,, {r r}_{00})) = = \{\begin{matrix} 11 & if if | | p p ((r r)) - - p p (({r r}_{00})) | | \leq \leq t t \\ 00 & if if | | p p ((r r)) - - p p (({r r}_{00})) | | > > t t | | \end{matrix} - - - - - - ((11))$

其中，r₀是圆形模板的核心像素点，r是圆形模板的其它像素点。p(r₀)指的是核心像素点的像素值，而p(r)指的就是其它对应像素点的像素值，t是灰度差阈值，D可以被认为是判决结果，其也被称为SUSAN距离。 Among them, r ₀ is the core pixel point of the circular template, and r is other pixel points of the circular template. p(r ₀ ) refers to the pixel value of the core pixel point, and p(r) refers to the pixel value of other corresponding pixel points, t is the gray level difference threshold, and D can be considered as the judgment result, which is also called is the SUSAN distance.

每个像素点的USAN区域大小用式(2)给出： The USAN area size of each pixel is given by formula (2):

$n no (({r r}_{00})) = = \underset{r r}{Σ Σ} D D. ((r r,, {r r}_{00})) - - - - - - ((22))$

然后根据式(3)可得该像素点的初始角响应： Then according to formula (3), the initial angular response of the pixel can be obtained:

$R R (({r r}_{00})) = = \{\begin{matrix} g g - - n no (({r r}_{00})) & if n if n (({r r}_{00})) < < g g \\ 00 & others others \end{matrix} - - - - - - ((33))$

式中，g表示几何阈值，一般取值为 N，表示模板内像素数量的一半。 In the formula, g represents the geometric threshold, and the general value is N, representing half the number of pixels within the template.

在SUSAN算法中，圆形模板是在图像上进行移动，因此该式(3)能够计算出每一个像素的初始角响应，如果某个像素的初始角响应不为0，那么该像素点被视为角点。所以，由这个公式(3)可以得到图像中的角点。 In the SUSAN algorithm, the circular template moves on the image, so the formula (3) can calculate the initial angular response of each pixel, if the initial angular response of a pixel is not 0, then the pixel is regarded as is the corner point. Therefore, the corner points in the image can be obtained from this formula (3). the

我们对适用于灰度图像的SUSAN算法作出如下改进，使其适合于本文的彩色图像角点检测。首先，式(1)中的像素比较仅仅取像素的亮度值作为衡量标准，这对于更加复杂的彩色图像来说显然是不够的，我们将式中的p(r)-p(r₀)改为： We make the following improvements to the SUSAN algorithm suitable for grayscale images, making it suitable for the color image corner detection in this paper. First of all, the pixel comparison in formula (1) only takes the brightness value of the pixel as a measure, which is obviously not enough for more complex color images. We change p(r)-p(r ₀ ) in the formula to for:

${D D.}^{' '} ((r r,, {r r}_{00})) = = | | | | p p ((r r)) - - p p (({r r}_{00})) | | | |,, &ForAll; &ForAll; P P ((r r)) &Element; &Element; I I - - - - - - ((44))$

上式是CIELab颜色空间中，相应像素的颜色矢量(即将每个像素的三个颜色分量视为一个矢量)间的L2范数。 The above formula is the L2 norm between the color vectors of the corresponding pixels (that is, the three color components of each pixel are regarded as a vector) in the CIELab color space. the

其次，式(1)中的t的取值，根据实验结果一般取25，但固定不变的取值显然缺乏灵活性，在我们的计算中，定义自适应的t值为： Secondly, the value of t in formula (1) is generally 25 according to the experimental results, but the fixed value obviously lacks flexibility. In our calculation, the adaptive t value is defined as:

$t t = = | | | | \frac{\underset{N N}{Σ Σ} (({p p}_{x x} - - \overset{&OverBar; &OverBar;}{p p}))}{N N} | | | | - - - - - - ((55))$

其中，p_x表示模板内部任一像素的像素矢量，N表示模板内像素数量，表示模板内部所有像素矢量的平均值。 Among them, p _x represents the pixel vector of any pixel inside the template, N represents the number of pixels in the template, Represents the average of all pixel vectors inside the template.

所以，本文的算法应用下式来进行在CIELab空间中模板核像素与模板内其他像素的比较： Therefore, the algorithm in this paper uses the following formula to compare the template kernel pixel with other pixels in the template in the CIELab space:

$D D. ((r r,, {r r}_{00})) = = \{\begin{matrix} 11 & if if | | | | p p ((r r)) - - p p (({r r}_{00})) | | | | \leq \leq | | | | \frac{\underset{N N}{Σ Σ} (({p p}_{x x} - - \overset{&OverBar; &OverBar;}{p p}))}{N N} | | | | \\ 00 & if if | | | | p p ((r r)) - - p p (({r r}_{00})) | | | | > > | | | | \frac{\underset{N N}{Σ Σ} (({p p}_{x x} - - \overset{&OverBar; &OverBar;}{p p}))}{N N} | | | | \end{matrix} - - - - - - ((66))$

这样就得到了图像中的角点集合。 In this way, the set of corner points in the image is obtained. the

在常规的显著区域检测中，仅考虑了每个角点像素与其邻域内的数十个像素之间的相关性，但事实上，对于一幅图像每个像素之间都具有一定的相关性，我们称之为全局相关。下一节，我们将针对这种全局相关性，对得到的角点进行处理，从而得到显著区域。以下将集合中的角点也称为基准点。 In conventional salient region detection, only the correlation between each corner pixel and dozens of pixels in its neighborhood is considered, but in fact, there is a certain correlation between each pixel in an image, We call this global correlation. In the next section, we will process the obtained corner points for this global correlation to obtain salient regions. Hereinafter, the corner points in the set are also referred to as reference points. the

首先，对集合中每一个像素(即角点像素)依据下式计算其全局对比度： First, calculate the global contrast of each pixel (corner pixel) in the set according to the following formula:

$S_{A} = \underset{w \times h}{Σ} D (P_{x, y}, P_{i})$ 公式(7) $S_{A} = \underset{w \times h}{Σ} D. (P_{x, the y}, P_{i})$ Formula (7)

其中，P_x，y表示坐标为(x，y)的基准点像素的矢量值(亦即该位置处的像素的像素值，其是一个3×1的矢量)，P_i为图像中任意一点的像素矢量值，D(P_x，y，P_i)为图像I的CIELab空间中的像素距离，表达式如公式(4)所示，w和h表示图像的宽度和高度，因此表示进行针对整个图像的全局处理。 Among them, P _{x, y} represents the vector value of the reference point pixel whose coordinates are (x, y) (that is, the pixel value of the pixel at this position, which is a 3×1 vector), and P _i is any point in the image The pixel vector value of D(P _{x, y} , P _i ) is the pixel distance in the CIELab space of image I, the expression is shown in formula (4), w and h represent the width and height of the image, so Indicates global processing for the entire image.

对所有像素的S_A，我们定义一个阈值T对基准点进行筛选： For S _A of all pixels, we define a threshold T to filter the reference points:

$T = \overset{&OverBar;}{S_{A}} = \frac{\underset{N}{Σ} S_{A}}{N}$ 公式(8) $T = \overset{&OverBar;}{S_{A}} = \frac{\underset{N}{Σ} S_{A}}{N}$ Formula (8)

T称为基准点平均显著值，N是集合中所有基准点的数量。 T is called the benchmark mean saliency value, and N is the number of all benchmarks in the set. the

我们将所有基准点中S_A＜T的点去除，然后对剩下基准点进行去孤立点处理。例如，可以采用N*N邻域进行去孤立点处理。去孤立点处理可以删除基准点集合中的孤立点，从而仅留下聚集在显著性区域内部的点。本领域技术人员可以采用任意方式进行这种去孤立点处理。 We remove the points where S _A < T among all the reference points, and then de-isolate the remaining reference points. For example, an N*N neighborhood may be used to remove isolated points. De-outlier processing can remove the isolated points in the reference point set, so as to leave only the points clustered inside the salient region. Those skilled in the art can use any method to perform such de-isolation processing.

通过实验发现，在利用本发明的对图像进行处理后的点大都位于人工确定的显著性区域内部。 It is found through experiments that most of the points after image processing by the present invention are located inside the artificially determined salient regions. the

在经过处理后的图像中剩余的基准点全部位于显著区域内部或者边界。现在的任务是根据这些基准点的位置得到整个显著性区域。对于自然图像，我们注意到显著区域内部具有相似的颜色及纹理结构，但这些特征区别于背景部分，我们采用以基准点为中心的随机像素搜索算法完成整个显著性区域检测。 All remaining fiducial points in the processed image are located inside or on the boundary of the salient area. Now the task is to get the whole saliency region according to the positions of these fiducials. For natural images, we notice that salient regions have similar colors and texture structures, but these features are different from background parts. We use a random pixel search algorithm centered on the fiducial point to complete the entire salient region detection. the

分析图像中像素及像素与基准像素点之间的关系，各个像素点的显著性值计算应该遵循以下基本准则： To analyze the relationship between pixels in the image and the relationship between pixels and reference pixels, the calculation of the significance value of each pixel should follow the following basic principles:

(1)经过全局处理后剩余的基准像素点兼具较高的局部显著概率和全局显著概率，那么与之具有相近特征的像素也应该有较高的显著概率。 (1) After the global processing, the remaining reference pixels have both high local saliency probability and global saliency probability, so the pixels with similar features should also have high saliency probability. the

(2)像素的显著概率具有距离相关性。即尽管某些像素与基准点有相近的特征，但在图像上分布的距离较远而使得其显著概率下降。 (2) The saliency probability of a pixel has distance correlation. That is, although some pixels have similar features to the reference point, their distribution distance on the image is far away, which makes their significant probability decrease. the

(3)对于整幅图像的所有显著像素具有向心性。即对于一幅自然图像，其显著性位置应该偏向图像中心位置。 (3) All salient pixels of the entire image have centripetal properties. That is, for a natural image, its salient position should be biased towards the center of the image. the

(4)尽管像素的信息量与其在图像中出现的概率成反比，但事实上无论其出现的概率极小或者极大其显著性概率都应该较小。 (4) Although the amount of information of a pixel is inversely proportional to the probability of its appearance in the image, in fact, no matter whether the probability of its appearance is extremely small or extremely large, its significance probability should be small. the

我们依据以上四条准则，从所有M个基准点依次取每一个点p(c)作为中心点，考虑以该点为中心的圆周邻域，依次搜索其圆周的上每一个基准点像素，依据下式计算其显著性取值： Based on the above four criteria, we take each point p(c) as the center point in sequence from all M reference points, consider the circular neighborhood centered on this point, and search for each reference point pixel on its circumference in turn, according to the following The formula calculates its significance value:

$Saliency Saliency ((c c,, r r)) = = {ω ω}_{33} ((c c,, r r)) \cdot &Center Dot; {ω ω}_{44} ((c c,, r r)) \cdot &Center Dot; \underset{M m,, &ForAll; &ForAll; {p p}_{c c} &Element; &Element; I I,, &ForAll; &ForAll; {p p}_{c c + + r r} &Element; &Element; I I}{Σ Σ} {ω ω}_{22} ((c c,, r r)) \cdot &Center Dot; exp exp ((- - | | | | {p p}_{c c} - - {p p}_{c c + + r r} | | | |)) - - - - - - ((99))$

其中，c表示基准点坐标，r表示圆周的半径；依据规则(2)定义的权重ω₂(c，r)表示如下： Among them, c represents the coordinates of the reference point, and r represents the radius of the circle; the weight ω ₂ (c, r) defined according to the rule (2) is expressed as follows:

${ω ω}_{22} ((c c,, r r)) = = \frac{11}{11 + + λ λ \cdot &Center Dot; r r} - - - - - - ((1010))$

其中，λ＝3。当然，该值仅是发明人依据经验得到一个示例性值，本领域技术人员可以依据实际显著性检测的深度而采用各种不同的值。 Among them, λ=3. Certainly, this value is only an exemplary value obtained by the inventor based on experience, and those skilled in the art may adopt various values according to the actual depth of saliency detection. the

||p_c-p_c+r||仍表示像素c与(c+r)的像素矢量值的L2范数，其中像素(c+r)表示在所述圆中的任一基准点像素。 ||p _c -p _c+r || still represents the L2 norm of the pixel vector values of pixels c and (c+r), where pixel (c+r) represents any reference point pixel in the circle.

依据规则(3)定义的权重ω₃(c，r)表示计算的像素点距离图像中心的偏移量，定义如下： The weight ω ₃ (c, r) defined according to the rule (3) represents the offset between the calculated pixel point and the image center, which is defined as follows:

${ω ω}_{33} ((c c,, r r)) = = 11 - - \frac{DistToCenter DistToCenter (({p p}_{c c + + r r}))}{{max max}_{c c} {{DistTocenter DistTocenter (({p p}_{c c}))}}} - - - - - - ((1111))$

其中，DistToCenter(p_c)表示像素p_c距离图像中心的空间距离。max_c{DistTocenter(p_c)}为归一化因子，表示全部基准点中DistTocenter(p_c)的最大值。 Among them, DistToCenter(p _c ) represents the spatial distance of pixel p _c from the center of the image. max _c {DistTocenter(p _c )} is a normalization factor, indicating the maximum value of DistTocenter(p _c ) among all reference points.

依据规则(4)，我们用高斯函数来表示该像素在图像中出现的概率对其显著性概率的影响，定义ω₄(c，r)如下式所示： According to rule (4), we use a Gaussian function to represent the influence of the probability of the pixel appearing in the image on its significance probability, and define ω ₄ (c, r) as shown in the following formula:

${ω ω}_{44} ((c c,, r r)) = = Aexp Aexp [[- - \frac{{(({P P}_{c c} - - \overset{&OverBar; &OverBar;}{P P}))}^{22}}{{σ σ}_{s the s}^{22}}]] - - - - - - ((1212))$

其中，A表示高斯函数的常系数，P_c表示像素p_c在图像中出现的概率，表示所述集合中全部基准点像素的平均出现概率，控制像素概率权值强度。越大，偏向两个极端的像素出现概率对其显著概率影响越大。在我们的实验中，当然，该值仅是发明人依据经验得到一个示例性值，本领域技术人员可以依据实际显著性检测的深度而采用各种不同的值。 Among them, A represents the constant coefficient of the Gaussian function, P _c represents the probability that the pixel p _c appears in the image, Indicates the average occurrence probability of all reference point pixels in the set, Controls the pixel probability weight strength. The larger is, the greater the impact of the occurrence probability of pixels biased towards the two extremes on its significance probability. In our experiments, Certainly, this value is only an exemplary value obtained by the inventor based on experience, and those skilled in the art may adopt various values according to the actual depth of saliency detection.

从而，可以确定每一个基准点的显著性值Saliency(c，r)，并获得整个图像的显著性特征图，如图4右侧所示。 Thus, the saliency value Saliency(c, r) of each reference point can be determined, and the saliency feature map of the entire image can be obtained, as shown on the right side of Fig. 4 . the

在进一步的实施例中，可以利用高斯多尺度方式进行多尺度计算。我们分别在尺度上应用式(9)进行计算。不同的图像尺度具有不同数量的基准点，如果某个尺度的图像基准点数量小于上一个尺度图像数量一半，则将尺度缩放时的图像抽样行列偏移1个单位重新缩放。 In a further embodiment, a Gaussian multi-scale approach can be used for multi-scale calculations. We are in the scale Apply formula (9) to calculate. Different image scales have different numbers of reference points. If the number of image reference points of a certain scale is less than half of the number of images of the previous scale, the image sampling row and column during scaling will be shifted by 1 unit for re-scaling.

典型的多尺度是高斯金字塔模型L(x，y，σ)＝G(x，y，σ)*I(x，y)，其中所述金字塔模型采用上述尺度集其中为高斯核，σ为所述尺度因子，x和y为图像像素坐标，I(x，y)为坐标(x，y)处的像素值。 A typical multi-scale is the Gaussian pyramid model L(x, y, σ)=G(x, y, σ)*I(x, y), wherein the pyramid model adopts the above scale set in is the Gaussian kernel, σ is the scaling factor, x and y are image pixel coordinates, and I(x, y) is the pixel value at coordinates (x, y).

最后，我们将三个不同尺度的显著性图像进行合并得到最终的显著性图像。在合并时，由于尺寸不同，需要将其变换为相同的尺度。我们采用加权的的显著图合并方法进行合并。我们以三个尺度的基准点数目的比例作为权重标准，合并的结果表示为： Finally, we merge three saliency images of different scales to get the final saliency image. When merging, due to the different sizes, they need to be transformed to the same scale. We employ a weighted saliency map merging method for merging. We use the ratio of the number of reference points of the three scales as the weight standard, and the combined result is expressed as:

$SaliencyMap SaliencyMap {| |}_{final final} = = \frac{{N N}_{s the s}}{N N} \cdot &Center Dot; SaliencyMap SaliencyMap {| |}_{s the s} + + \frac{{N N}_{s the s / / 22}}{N N} \cdot &Center Dot; SaliencyMap SaliencyMap {| |}_{\frac{11}{22} s the s} + + \frac{{N N}_{s the s / / 44}}{N N} \cdot \cdot SaliencyMap SaliencyMap {| |}_{\frac{11}{44} s the s}$

其中，N_s、N_s/2、N_s/4分别表示不同尺度的基准点数量，N表示三个尺度的基准点数量之和，SaliencyMap|_s，分别表示不同尺度上的显著性取值。 Among them, N _s , N _s/2 , and N _s/4 respectively represent the number of reference points of different scales, N represents the sum of the number of reference points of the three scales, SaliencyMap| _s , Respectively represent the significance values on different scales.

图2示出了根据本发明的一些实施例的从点至面的图像显著性检测的流程图。 Fig. 2 shows a flowchart of image saliency detection from point to surface according to some embodiments of the present invention. the

在步骤201，输入彩色图像，所述图像为Lab(L为亮度，a和b分别表示色度)颜色空间格式的图像，其一个像素的像素值为由L、a、b三个分量构成的矢量； In step 201, a color image is input, and the image is an image in a Lab (L is brightness, a and b represent chromaticity respectively) color space format, and the pixel value of one pixel is formed by three components of L, a, and b vector;

在步骤202，利用SUSAN角点检测方法获取所述图像的基准点集合，其中，采用一个圆形模板在所述图像中移动，并在每个位置上利用以下公式(6)比较所述模板内各像素的亮度与模板核的亮度： In step 202, the SUSAN corner point detection method is used to obtain the reference point set of the image, wherein a circular template is used to move in the image, and the following formula (6) is used to compare each position in the template at each position The brightness of the pixel and the brightness of the template kernel:

其中，r₀是圆形模板的核心像素点，r是圆形模板的其它像素点，p(r₀)指的是核心像素点的像素值，而p(r)指的就是其它对应像素点的像素值，||.||表示在Lab空间中的L2范数，p_x表示模板内部任一像素的像素矢量，N表示模板内像素数量，表示模板内部所有像素矢量的平均值，D是判决结果， Among them, r ₀ is the core pixel of the circular template, r is other pixels of the circular template, p(r ₀ ) refers to the pixel value of the core pixel, and p(r) refers to other corresponding pixel points The pixel value of , ||.|| represents the L2 norm in the Lab space, p _x represents the pixel vector of any pixel inside the template, N represents the number of pixels in the template, Indicates the average value of all pixel vectors inside the template, D is the judgment result,

然后，利用以下公式(2)计算每一个像素点的USAN区域大小： Then, use the following formula (2) to calculate the USAN area size of each pixel:

然后，利用以下公式(3)计算每一个像素点的初始角响应： Then, use the following formula (3) to calculate the initial angular response of each pixel:

其中，g表示几何阈值， where g represents the geometric threshold,

其中，如果某个像素的初始角响应不为0，那么该像素点被视为角点，并加入所述基准点集合中， Among them, if the initial angular response of a certain pixel is not 0, then the pixel is regarded as a corner point and added to the set of reference points,

在步骤203，依据所述图像的全局相关性所述基准点集合中的一些基准点，其中，对所述集合中每一个像素依据下式(7)计算其全局对比度： In step 203, some reference points in the reference point set according to the global correlation of the image, wherein, calculate its global contrast for each pixel in the set according to the following formula (7):

${S S}_{A A} = = \underset{w w \times \times h h}{Σ Σ} D D. (({P P}_{x x,, y the y},, {P P}_{i i})) - - - - - - ((77))$

其中，P_x，y表示坐标为(x，y)的基准点像素的矢量值(亦即该位置处的像素的像素值，其是一个3×1的矢量)，P_i为所述图像中任意一点的像素矢量值，D(P_x，y，P_i)为所述图像I的Lab空间中的像素距离，表达式如公式(4)所示，w和h表示所述图像的宽度和高度，因此表示进行针对整个图像的全局处理， Wherein, P _{x, y} represents the vector value of the reference point pixel whose coordinates are (x, y) (that is, the pixel value of the pixel at this position, which is a 3×1 vector), and P _i is The pixel vector value of any point, D (P _{x, y} , _Pi ) is the pixel distance in the Lab space of described image I, and expression is as shown in formula (4), and w and h represent the width and the width of described image height, therefore Indicates global processing for the entire image,

然后，对所有基准点像素的S_A，使用阈值T进行筛选，将所有基准点中S_A＜T的点从所述集合中去除： Then, for S _A of all reference point pixels, a threshold T is used to filter, and all reference points with S _A < T are removed from the set:

$T T = = \overset{&OverBar; &OverBar;}{{S S}_{A A}} = = \frac{\underset{N N}{Σ Σ} {S S}_{A A}}{N N} - - - - - - ((88))$

其中，T是基准点平均显著值，N是所述集合中所有基准点的数量， Wherein, T is the average significant value of the reference points, N is the number of all reference points in the set,

在步骤204，对所述集合中剩下的基准点进行去孤立点处理，以去除所述集合中的孤立点； In step 204, the remaining reference points in the set are processed to remove isolated points, so as to remove the isolated points in the set;

在步骤205，从所述集合中的M个基准点依次取每一个点p(c)作为中心点，考虑以该点为中心的圆周邻域，依次搜索其圆周内每一个基准点像素，依据下式计算其显著性取值： In step 205, each point p(c) is sequentially taken as the center point from the M reference points in the set, and the circular neighborhood centered on this point is considered, and each reference point pixel in the circle is searched in turn, according to The following formula calculates its significance value:

其中，c表示当前基准点，r表示圆周的半径，||p_c-p_c+r||仍表示当前基准点像素c与所述圆周内任一基准点(c+r)的像素矢量值在Lab空间中的L2范数，并且 Among them, c represents the current reference point, r represents the radius of the circle, ||p _c -p _c+r || still represents the pixel vector value of the current reference point pixel c and any reference point (c+r) in the circle L2 norm in Lab space, and

其中，λ是经验常数， where λ is an empirical constant,

其中，DistToCenter(p_c)表示像素p_c距离图像中心的空间距离，max_c{DistTocenter(p_c)}为归一化因子，表示所述集合的全部基准点中DistTocenter(p_c)的最大值， Among them, DistToCenter(p _c ) represents the spatial distance between pixel p _c and the center of the image, and max _c {DistTocenter(p _c )} is a normalization factor, representing the maximum value of DistTocenter(p _c ) among all reference points of the set ,

其中，A表示高斯函数的常系数，P_c表示像素p_c在图像中出现的概率，表示所述集合中全部基准点像素的平均出现概率，控制像素概率权值强度； Among them, A represents the constant coefficient of the Gaussian function, P _c represents the probability that the pixel p _c appears in the image, Indicates the average occurrence probability of all reference point pixels in the set, Control the pixel probability weight strength;

最后，输出显著特征图，以便进行进一步分析。 Finally, a salient feature map is output for further analysis. the

作为一个优选实施例，使用高斯金字塔模型L(x，y，σ)＝G(x，y，σ)*I(x，y)，采用尺度集对以上步骤502-505进行多尺度计算，其中为高斯核，σ为所述尺度因子，x和y为图像像素坐标，I(x，y)为坐标(x，y)处的像素值，从而获得在三个尺寸上的显著性值SaliencyMap|_s，并且在步骤(f)之后使用以下公式来获得最终的显著性值： As a preferred embodiment, use the Gaussian pyramid model L(x, y, σ)=G(x, y, σ)*I(x, y), using the scale set Multi-scale calculations are performed on the above steps 502-505, wherein is the Gaussian kernel, σ is the scale factor, x and y are the image pixel coordinates, I(x, y) is the pixel value at the coordinates (x, y), so as to obtain the significance value SaliencyMap| _s , And use the following formula after step (f) to get the final significance value:

$SaliencyMap SaliencyMap {| |}_{final final} = = \frac{{N N}_{s the s}}{N N} \cdot &Center Dot; SaliencyMap SaliencyMap {| |}_{s the s} + + \frac{{N N}_{s the s / / 22}}{N N} \cdot &Center Dot; SaliencyMap SaliencyMap {| |}_{\frac{11}{22} s the s} + + \frac{{N N}_{s the s / / 44}}{N N} \cdot &Center Dot; SaliencyMap SaliencyMap {| |}_{\frac{11}{44} s the s}$

其中，N_s、N_s/2、N_s/4分别表示不同尺度的基准点数量，N表示三个尺度的基准点数量之和。 Among them, N _s , N _s/2 , and N _s/4 respectively represent the number of reference points of different scales, and N represents the sum of the number of reference points of the three scales.

在步骤205a示出了上述的对多个尺度的显著性值进行合并的过程。 Step 205a shows the above-mentioned process of combining the significance values of multiple scales. the

图3示出了根据本发明的一些实施例的基于边缘的图像显著性检测装置。该装置的各个组件301-305的功能与以上方法中的步骤201-205类似，因此在此不再赘述。 Fig. 3 shows an apparatus for edge-based image saliency detection according to some embodiments of the present invention. The functions of the components 301-305 of the device are similar to the steps 201-205 in the above method, so details are not repeated here. the

本发明还涵盖实现图2所述方法的计算机程序产品和处理器。 The invention also covers computer program products and processors implementing the method described in FIG. 2 . the

尽管前述公开文件论述了示例性方案和/或实施例，但应注意，在不背离由权利要求书定义的描述的方案和/或实施例的范围的情况下，可以在此做出许多变化和修改。而且，尽管以单数形式描述或要求的所述方案和/或实施例的要素，但也可以设想复数的情况，除非明确表示了限于单数。另外，任意方案和/或实施例的全部或部分都可以与任意其它方案和/或实施例的全部或部分结合使用，除非表明了有所不同。 Although the foregoing disclosures discuss exemplary aspects and/or embodiments, it should be noted that many changes and/or changes may be made therein without departing from the scope of the described aspects and/or embodiments as defined by the claims. Revise. Furthermore, although elements of the described aspects and/or embodiments are described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated. In addition, all or part of any aspect and/or embodiment can be used in combination with all or part of any other aspect and/or embodiment, unless a difference is indicated. the

Claims

1. A method for image saliency detection comprising

(a) Input a color image I, the image is an image in the Lab color space format, and the pixel value of one pixel is a vector composed of three components L, a, and b, wherein L is brightness, and a and b respectively represent Chroma;

(b) Utilize the SUSAN corner point detection method to obtain the reference point set of the image, wherein a circular template is used to move in the image, and the following formula (6) is used to compare the pixels in the template at each position The brightness of and the brightness of the template kernel:

D D. ((r r,, {r r}_{00})) = = \{\begin{matrix} 11 if if | | | | p p ((r r)) - - p p (({r r}_{00})) | | | | \leq \leq | | | | \frac{\underset{N N}{Σ Σ} (({p p}_{x x} - - \overset{&OverBar; &OverBar;}{p p}))}{N N} | | | | \\ 00 if if | | | | p p ((r r)) - - p p (({r r}_{00})) | | | | > > | | | | \frac{\underset{N N}{Σ Σ} (({p p}_{x x} - - \overset{&OverBar; &OverBar;}{p p}))}{N N} | | | | \end{matrix} - - - - - - ((66))

Among them, r ₀ is the core pixel of the circular template, r is other pixels of the circular template, p(r ₀ ) refers to the pixel value of the core pixel, and p(r) refers to other corresponding pixel points The pixel value of , ||.|| represents the L2 norm in the Lab space, p _x represents the pixel vector of any pixel inside the template, N represents the number of pixels in the template, Indicates the average value of all pixel vectors inside the template, D is the judgment result,

Then, use the following formula (2) to calculate the USAN area size of each pixel:

n no (({r r}_{00})) = = \underset{r r}{Σ Σ} D D. ((r r,, {r r}_{00})) - - - - - - ((22))

Then, use the following formula (3) to calculate the initial angular response of each pixel:

R R (({r r}_{00})) = = \{\begin{matrix} g g - - n no (({r r}_{00})) & ifn ifn (({r r}_{00})) < < g g \\ 00 & others others \end{matrix} - - - - - - ((33))

where g represents the geometric threshold,

Among them, if the initial corner response of a certain pixel is not 0, then the pixel is regarded as a corner point and added to the set of reference points,

(c) remove some reference points in the reference point set according to the global correlation of the image, wherein, calculate its global contrast for each pixel in the set according to the following formula (7):

{S S}_{A A} = = \underset{w w \times \times h h}{Σ Σ} D D. (({P P}_{x x,, y the y},, {P P}_{i i})) - - - - - - ((77))

Among them, P _{x, y} represents the vector value of the reference point pixel whose coordinates are (x, y), that is, the pixel value of the pixel at this position, and is a 3×1 vector, and P _i is any The pixel vector value of a point, D (P _{x, y} , _Pi ) is the pixel distance in the Lab space of described image I, and expression is as shown in formula (4), and w and h represent the width and the height of described image ,therefore Indicates global processing for the entire image,

{D D.}^{' '} ((r r,, {r r}_{00})) = = | | | | P P ((r r)) - - p p (({r r}_{00})) | | | |,, &ForAll; &ForAll; P P ((r r)) &Element; &Element; I I - - - - - - ((44))

Then, for S _A of all reference point pixels, the threshold T is used to filter, and the points with S _A < T in all reference points are removed from the set:

T T = = \overset{&OverBar; &OverBar;}{{S S}_{A A}} = = \frac{\underset{N N}{Σ Σ} {S S}_{A A}}{N N} - - - - - - ((88))

Wherein, T is the average saliency value of benchmarks, N is the number of all benchmarks in the set,

(d) performing de-isolation processing on the remaining reference points in the set, so as to remove the isolated points in the set;

(e) From the M reference points in the set, take each point p(c) as the center point in turn, consider the circular neighborhood centered on this point, and search for each reference point pixel in its circumference in turn, according to the following The formula calculates its significance value:

Saliency Saliency ((c c,, r r)) = = {ω ω}_{33} ((c c,, r r)) \cdot &Center Dot; {ω ω}_{44} ((c c,, r r)) \cdot &Center Dot; \underset{M m,, &ForAll; &ForAll; {p p}_{c c} &Element; &Element; I I,, &ForAll; &ForAll; {p p}_{c c + + r r} &Element; &Element; I I}{Σ Σ} {ω ω}_{22} ((c c,, r r)) \cdot &Center Dot; exp exp ((- - | | | | {p p}_{c c} - - {p p}_{c c + + r r} | | | |)) - - - - - - ((99))

Among them, c represents the current reference point, r represents the radius of the circle, ||p _c -p _c+r || still represents the pixel vector value of the current reference point pixel c and any reference point (c+r) in the circle L2 norm in Lab space, and

{ω ω}_{22} ((c c,, r r)) = = \frac{11}{11 + + λ λ \cdot \cdot r r} - - - - - - ((1010))

where λ is an empirical constant,

{ω ω}_{33} ((c c,, r r)) = = 11 - - \frac{DistToCenter DistToCenter (({p p}_{c c + + r r}))}{{max max}_{c c} {{DistTocenter DistTocenter (({p p}_{c c}))}}} - - - - - - ((1111))

Among them, DistToCenter(p _c ) represents the spatial distance between pixel p _c and the center of the image, and max _c {DistTocenter(p _c )} is a normalization factor, representing the maximum value of DistTocenter(p _c ) among all reference points of the set ,

{ω ω}_{44} ((c c,, r r)) = = Aexp Aexp [[- - \frac{{(({P P}_{c c} - - \overset{&OverBar; &OverBar;}{P P}))}^{22}}{{σ σ}_{s the s}^{22}}]] - - - - - - ((1212))

Among them, A represents the constant coefficient of the Gaussian function, P _c represents the probability that the pixel p _c appears in the image, Indicates the average occurrence probability of all reference point pixels in the set, Control the pixel probability weight strength;

(f) Output salient feature maps.

2. The method of claim 1, further comprising:

Use the Gaussian pyramid model L(x, y, σ)=G(x, y, σ)*I(x, y), using the scale set Perform multiscale calculations for steps (b)-(e), where is the Gaussian kernel, σ is the scale factor, x and y are the image pixel coordinates, I(x, y) is the pixel value at the coordinates (x, y), so as to obtain the significance value SaliencyMap| _s on three dimensions, And use the following formula after step (e) to get the final significance value:

{SaliencyMap SaliencyMap | |}_{fimal fimal} = = \frac{{N N}_{s the s}}{N N} \cdot \cdot {SaliencyMap SaliencyMap | |}_{s the s} + + \frac{{N N}_{s the s / / 22}}{N N} \cdot &Center Dot; {SaliencyMap SaliencyMap | |}_{\frac{11}{22} s the s} + + \frac{{N N}_{s the s / / 44}}{N N} \cdot &Center Dot; {SaliencyMap SaliencyMap | |}_{\frac{11}{44} s the s}

Among them, N _s , N _s/2 , and N _s/4 respectively represent the number of reference points of different scales, and N represents the sum of the number of reference points of the three scales.

3. The method of claim 1, wherein, in step (b), g is selected from Represents half of the number of pixels within the template.

4. The method of claim 1, wherein, in step (e), λ=3.

5. A device for image saliency detection, comprising

(a) module for input color image I, described image is the image of Lab color space format, and the pixel value of its one pixel is the vector that is made of three components of L, a, b, wherein, L is brightness, a and b represent chromaticity respectively;

(b) A set of reference points for obtaining the image using the SUSAN corner detection method, wherein a circular template is used to move in the image, and the following formula (6) is used to compare the The module of the brightness of each pixel and the brightness of the template core:

D D. ((r r,, {r r}_{00})) = = \{\begin{matrix} 11 if if | | | | p p ((r r)) - - p p (({r r}_{00})) | | | | \leq \leq | | | | \frac{\underset{N N}{Σ Σ} (({p p}_{x x} - - \overset{&OverBar; &OverBar;}{p p}))}{N N} | | | | \\ 00 if if | | | | p p ((r r)) - - p p (({r r}_{00})) | | | | > > | | | | \frac{\underset{N N}{Σ Σ} (({p p}_{x x} - - \overset{&OverBar; &OverBar;}{p p}))}{N N} | | | | \end{matrix} - - - - - - ((66))

n no (({r r}_{00})) = = \underset{r r}{Σ Σ} D D. ((r r,, {r r}_{00})) - - - - - - ((22))

R R (({r r}_{00})) = = \{\begin{matrix} g g - - n no (({r r}_{00})) & ifn ifn (({r r}_{00})) < < g g \\ 00 & others others \end{matrix} - - - - - - ((33))

where g represents the geometric threshold,

(c) a module for removing some reference points in the set of reference points according to the global correlation of the image, wherein the global contrast is calculated for each pixel in the set according to the following formula (7):

{S S}_{A A} = = \underset{w w \times \times h h}{Σ Σ} D D. (({P P}_{x x,, y the y},, {P P}_{i i})) - - - - - - ((77))

{D D.}^{' '} ((r r,, {r r}_{00})) = = | | | | P P ((r r)) - - p p (({r r}_{00})) | | | |,, &ForAll; &ForAll; P P ((r r)) &Element; &Element; I I - - - - - - ((44))

Then, for S _A of all reference point pixels, a threshold T is used to filter, and all reference points with S _A < T are removed from the set:

T T = = \overset{&OverBar; &OverBar;}{{S S}_{A A}} = = \frac{\underset{N N}{Σ Σ} {S S}_{A A}}{N N} - - - - - - ((88))

(d) a module for performing de-isolation processing on the remaining reference points in the set, so as to remove the isolated points in the set;

(e) for taking each point p(c) as a center point successively from the M reference points in the set, considering the circular neighborhood centered on this point, and searching for each reference point pixel in its circumference in turn, The module whose significance value is calculated according to the following formula:

Saliency Saliency ((c c,, r r)) = = {ω ω}_{33} ((c c,, r r)) \cdot &Center Dot; {ω ω}_{44} ((c c,, r r)) \cdot &Center Dot; \underset{M m,, &ForAll; &ForAll; {p p}_{c c} &Element; &Element; I I,, &ForAll; &ForAll; {p p}_{c c + + r r} &Element; &Element; I I}{Σ Σ} {ω ω}_{22} ((c c,, r r)) \cdot \cdot exp exp ((- - | | | | {p p}_{c c} - - {p p}_{c c + + r r} | | | |)) - - - - - - ((99))

{ω ω}_{22} ((c c,, r r)) = = \frac{11}{11 + + λ λ \cdot &Center Dot; r r} - - - - - - ((1010))

where λ is an empirical constant,

{ω ω}_{33} ((c c,, r r)) = = 11 - - \frac{DistToCenter DistToCenter (({p p}_{c c + + r r}))}{{max max}_{c c} {{DistTocenter DistTocenter (({p p}_{c c}))}}} - - - - - - ((1111))

{ω ω}_{44} ((c c,, r r)) = = Aexp Aexp [[- - \frac{{(({P P}_{c c} - - \overset{&OverBar; &OverBar;}{P P}))}^{22}}{{σ σ}_{s the s}^{22}}]] - - - - - - ((1212))

(f) Module for outputting salient feature maps.