CN101651772A - Method for extracting video interested region based on visual attention - Google Patents

Abstract

The invention discloses a method for extracting a region of interest in a video based on visual attention. The region of interest extracted by the method combines visual attention in static image domain, visual attention in movement and depth visual attention, effectively suppressing the inherent singleness of each visual attention extraction. It solves the noise problem caused by the complex background in the visual attention of the static image domain, and solves the problem that the motion visual attention cannot extract the local motion and the region of interest with small motion range, thereby improving the calculation accuracy and enhancing the stability of the algorithm , can extract the region of interest from the background with complex texture and moving environment; in addition, the region of interest obtained by this method is not only in line with the human eye's visual interest characteristics of static texture video frames and the human eye's interest in moving objects In addition to visual characteristics, it also conforms to the depth perception characteristics of being interested in objects with a strong sense of depth or close distance in stereo vision, and conforms to the semantic characteristics of human stereo vision.

Description

A Visual Attention-Based Method for Extracting Regions of Interest in Video

technical field

The invention relates to a method for processing video signals, in particular to a method for extracting video interest regions based on visual attention.

Background technique

Stereoscopic TV, also known as 3DTV (Three Dimensional Television, three-dimensional TV), because stereoscopic TV can provide a leap from flat to three-dimensional, and give viewers a unique sense of three-dimensionality and reality, it has been highly recognized by research institutions and industries at home and abroad. Pay attention to. In 2002, an ATTEST (Advanced 3D Television System Technology) project was launched in the IST program supported by the European Commission. The project aims to establish a complete backward compatible 3D digital TV broadcasting chain system. The goal of the ATTEST project is to propose a new concept of 3DTV broadcasting chain, realize backward compatibility with existing 2D broadcasting, and widely support various forms of 2D and 3D display. The main design concept of the ATTEST project is to propose adding a depth map (Depth Map) as the enhancement layer information on the basis of traditional two-dimensional video image transmission, that is, the data representation of "two-dimensional color video plus depth". The way of adding depth is to decode and reconstruct 3D video on the display terminal, and some advanced naked-eye autostereoscopic display terminals in the industry have also supported the display mode of adding depth to 2D color video.

In the human visual receiving and processing system, due to limited brain resources and differences in the importance of external environmental information, the human brain does not treat external environmental information equally in the processing process, but shows selective characteristics, that is, different degrees of interest. For a long time, the extraction of video regions of interest has been one of the core and difficult technologies of content-based video processing methods in the fields of video compression and communication, video retrieval, and pattern recognition. Visual psychology studies have shown that the human eye's selectivity or interest in external visual input is inseparable from the human visual attention characteristics. At present, the research on visual attention cues is mainly divided into two aspects: top-down (Top-down) (also called Concept-driven) attention cues and bottom-up (Bottom-up) (also called Concept-driven) attention cues. Stimulus-driven attentional cues. Top-down attention cues mainly come from complex psychological processes, and directly pay attention to certain objects in the scene, including object shapes, actions, patterns and other related identification features, which are influenced by personal knowledge, hobbies, subconscious The influence of other factors varies from person to person. Another kind of clue is the bottom-up attention clue, which mainly comes from the direct stimulation of the visual cortex caused by the visual feature factors of the video scene, mainly including color, brightness, direction and other stimuli. The bottom-up attention clue is instinctive, Automatic, has good universal applicability, and is relatively stable, and is basically not affected by conscious factors such as personal knowledge and hobbies. Therefore, bottom-up attention clues are one of the hot topics in the research of automatic region-of-interest extraction methods. .

However, the current automatic region of interest extraction is mainly divided into three categories, 1) using the internal information of a single viewpoint image, including stimulus information such as brightness, color, texture or direction, to extract the region of interest of the human eye on the current video frame, This method mainly extracts regions with large differences in brightness, color and texture contrast as regions of interest, which makes this method difficult to apply to the region of interest extraction in complex background environments; The principle is to use the motion information between video frames as the main clue to extract the region of interest. However, this method is difficult to accurately extract slow or local moving objects, and it is also difficult to apply to the region of interest extraction in the case of global motion; 3 ), using the extraction method combining static texture and motion information. Due to the weak redundancy and correlation between static texture and motion information, this method cannot effectively suppress the existing extraction errors and noises, so that the extraction accuracy is not good. high. Due to the limitation of the amount of information available, these three types of traditional methods cause the extracted regions of interest to be inaccurate and poor in stability; on the other hand, the traditional methods do not consider being interested in objects with a strong sense of depth or closer to the viewer. The stereoscopic characteristics of stereoscopic vision cannot well represent the true degree of interest of the human eye with stereoscopic vision, so it is difficult to apply to the region of interest extraction that conforms to the semantic features of stereoscopic vision in the new generation of stereoscopic (3D)/multi-viewpoint videos.

Contents of the invention

The technical problem to be solved by the present invention is to provide a visual attention-based system that can make the extracted video region of interest have higher precision and better stability, and the extracted video region of interest conforms to the semantic characteristics of human stereo vision. Method for extracting regions of interest from videos.

The technical solution adopted by the present invention to solve the above-mentioned technical problems is: a method for extracting a video region of interest based on visual attention, comprising the following steps:

①Define the two-dimensional color video as texture video, and define the size of the texture video frame at each moment in the texture video as W×H, where W is the width of the texture video frame at each moment in the texture video, and H is the The height of the texture video frame at time, record the texture video frame at time t in the texture video as F _t , define the texture video frame F _t at time t in the texture video as the current texture video frame, and use the known static image visual attention detection method to detect The static image domain visual attention of the current texture video frame obtains the distribution map of the static image domain visual attention of the current texture video frame, which is denoted as S _I , and the size of the distribution map S _I of the static image domain visual attention of the current texture video frame is: W×H and it is a grayscale image represented by Z _S bit depth;

② Use the motion visual attention detection method to detect the motion visual attention of the current texture video frame, and obtain the distribution map of the motion visual attention of the current texture video frame, denoted as S _M , the size of the motion visual attention distribution map S _M of the current texture video frame A grayscale image of size W×H and Z _bit depth representation;

③ Define the depth video frame at each moment in the depth video corresponding to the texture video as a grayscale image represented by Z _D bit depth, set the size of the depth video frame at each moment in the depth video to W×H, W is the depth video The width of the depth video frame at each moment in the depth video, H is the height of the depth video frame at each moment in the depth video, record the depth video frame at time t in the depth video as D _t , define the depth video frame D _t at time t in the depth video is the current depth video frame, using the depth visual attention detection method to detect the depth visual attention of the 3D video image jointly displayed by the current depth video frame and the current texture video frame, and obtain the distribution map of the depth visual attention of the 3D video image, denoted as _SD , The distribution map _SD of the depth visual attention of three-dimensional video image has a size of W×H and it is a grayscale image represented by Z _S bit depth;

④Adopt the visual attention fusion method based on depth perception to combine the static image domain visual attention distribution map S _I of the current texture video frame, the motion visual attention distribution map S _M of the current texture video frame, the current depth video frame and the current depth video frame The depth visual attention distribution map S _D of the 3D video image jointly displayed with the current texture video frame is fused to extract the 3D visual attention distribution map that conforms to the stereoscopic perception of the human eye, denoted as S, the size of the 3D visual attention distribution map S A grayscale image of size W×H and Z _bit depth representation;

⑤ Thresholding and macro-blocking post-processing are performed on the distribution map S of 3D visual attention, and the final region of interest conforming to the stereoscopic perception of the human eye is obtained for the current texture video frame;

⑥Repeat steps ①～⑤ until all the texture video frames in the texture video are processed, and the video ROI of the texture video is obtained.

The specific process of the motion visual attention detection method in the described step 2. is:

②-1. Record the texture video frame at time t+j continuous with the current texture video frame in the texture video as F _t+j , record the texture video frame at the time tj time continuous with the current texture video frame in the texture video is F _tj , where, j∈(0, N _F /2], N _F is a positive integer less than 10;

②-2, using the known optical flow method to calculate the motion vector image of the current texture video frame and the texture video frame F t+j at the moment t _+j in the horizontal direction and the motion vector image of the vertical direction, and the current texture video frame and The motion vector image of the texture video frame F _tj in the horizontal direction and the motion vector image in the vertical direction of the texture video frame F tj at the moment tj, record the motion vector image of the current texture video frame and the texture video frame F t+j of the moment t _+j in the horizontal direction as The motion vector image of V _t+j ^H and the vertical direction is V _t+j ^V , and the motion vector image of the current texture video frame and the texture video frame F tj at the moment _tj in the horizontal direction is V _tj ^H and the motion vector image of the vertical direction The motion vector image is V _tj ^V , the width of V _t+j ^H , V _t+j ^V , V _tj ^H and V _tj ^V is W and the height is H;

②-3. Superimpose the absolute value of V _t+j ^H and the absolute value of V _t+j ^V to obtain the motion amplitude image of the current texture video frame and the texture video frame F _t+ j at time t+j, denoted as M _{t +j} , $m_{t+j}=\left|V_{t+j}^h\right|+\left|V_{t+j}^V\right|,$ Record the motion amplitude value of the pixel whose coordinates are (x, y) in M _t+j is m _t+j (x, y); superimpose the absolute value of V _tj ^H and the absolute value of V _tj ^V to obtain the current texture video frame and the motion amplitude image of texture video frame F _tj at time tj, denoted as M _tj , $m_{t-j}=\left|V_{t-j}^h\right|+\left|V_{t-j}^V\right|,$ Note that the motion amplitude value of the pixel whose coordinates are (x, y) in M _tj is m _tj (x, y);

否则，确定联合运动图M_j ^Δ中坐标为(x，y)的像素的像素值为0，其中，min()为取最小值函数；②-4. Use the current texture video frame and the texture video frame F t+j at time _t+j and the texture video frame F tj at time _tj to extract the joint motion map, denoted as M _j ^Δ , and extract the joint motion map M _j ^Δ The specific process is: judging the current texture video frame and the texture video frame F t+j at the time _t+j of each pixel in the motion range image M _t+j and the current texture video frame and the texture video frame F tj at the time _tj Whether the minimum value of the motion amplitude value of the pixel corresponding to the coordinate in the motion amplitude image M _tj is greater than the set first threshold T ₁ , if yes, then determine the pixel value M of the pixel corresponding to the coordinate in the joint motion map M _j ^Δ The average of the sum of the motion amplitude values of the pixels corresponding to the coordinates in _t+j and M _tj , otherwise, the pixel value of the pixel corresponding to the coordinates in the joint motion map M _j ^Δ is determined to be 0; for M _t+j , the coordinates are (x , y) and the pixel whose coordinates are (x, y) in M _tj , determine whether min(m _t+j (x, y), m _tj (x, y)) is greater than the set first threshold T ₁ , if yes, determine the pixel value of the pixel with coordinates (x, y) in the joint motion map M _j ^Δ

Otherwise, determine that the pixel value of the pixel whose coordinates are (x, y) in the joint motion map M _j ^Δ is 0, where min() is a minimum value function;

2.-5. The weighted joint motion map obtained from the weighted joint motion map of the current texture video frame is obtained by the weighted joint motion map of each moment from 1 moment to _NF /2 moment with the t moment in time, denoted as M, and the number of the current texture video frame The pixel value of the pixel whose coordinates are (x, y) in the weighted joint motion map M is m(x, y), $m(x,\mathrm{the\; y})=\underset{j=1}{\overset{N_f/2}{\mathrm{\Σ}}}{\mathrm{\ζ}}_j{m}_j^{\mathrm{\Δ}}(x,\mathrm{the\; y}),$ Among them, m _j ^Δ (x, y) represents the pixel value of the pixel whose coordinates are (x, y) in the joint motion map M _j ^Δ at time j at a distance from time t in time, and ζ _i is the weighting coefficient, and the weighting coefficient ζ _i satisfies $\underset{j=1}{\overset{N_f/2}{\mathrm{\Σ}}}{\mathrm{\ζ}}_j=1;$

②-6. Decompose the weighted joint motion map M of the current texture video frame into a Gaussian pyramid, decompose it into n _L layers of weighted joint motion maps, and record the i-th layer weighted joint motion map obtained after the Gaussian pyramid decomposition of the weighted joint motion map M is M(i), the width and height of the i-th layer weighted joint motion map M(i) are W/2 ⁱ and H/2 ⁱ respectively, where n _L is a positive integer less than 20, i∈[0,n _L -1], W is the width of the current texture video frame, and H is the height of the current texture video frame;

s，c∈[0，n_L-1]，s＝c+δ，δ＝{-3，-2，-1，1，2，3}，

为归一化至-1区间的归一化函数，符号“||”为绝对值运算符号，M(c)为第c层加权联合运动图，M(s)为第s层加权联合运动图，符号为M(c)与M(s)进行跨层级作差运算符，如果c＜s，则将M(s)上采样至与M(c)具有相同分辨率的图像上，然后将M(c)的各个像素与上采样后的M(s)相对应像素分别进行作差，如果c＞s，则将M(c)上采样至与M(s)具有相同分辨率的图像上，然后将M(s)的各个像素与上采样后的M(c)相对应像素分别进行作差，符号

为M(c)与M(s)进行跨层级相加运算符，如果c＜s，则将M(s)上采样至与M(c)具有相同分辨率的图像上，然后将M(c)的各个像素与上采样后的M(s)相对应像素分别进行求和，如果c＞s，则将M(c)上采样至与M(s)具有相同分辨率的图像上，然后将M(s)的各个像素与上采样后的M(c)相对应像素分别进行求和。2.-7. Utilize the weighted joint motion map of the n _L layer weighted joint motion map M of the weighted joint motion map M of the current texture video frame to extract the distribution map S _M of the motion visual attention of the current texture video frame, and record the coordinates in S _M as (x, y) The pixel value of the pixel is s _m (x, y), S _M =F _M , where,

s, c ∈ [0, n _L -1], s = c + δ, δ = {-3, -2, -1, 1, 2, 3},

to normalize to The normalization function of the -1 interval, the symbol "||" is the absolute value operation symbol, M(c) is the weighted joint motion map of the c-th layer, M(s) is the weighted joint motion map of the s-th layer, the symbol Perform a cross-level difference operator for M(c) and M(s), if c<s, then upsample M(s) to an image with the same resolution as M(c), and then M(c ) and the corresponding pixels of the upsampled M(s) are respectively differentiated. If c>s, then M(c) is upsampled to an image with the same resolution as M(s), and then The difference between each pixel of M(s) and the corresponding pixel of M(c) after upsampling is respectively performed, and the sign

Perform a cross-level addition operator for M(c) and M(s), if c<s, then upsample M(s) to an image with the same resolution as M(c), and then M(c ) and the corresponding pixels of the upsampled M(s) are summed separately, if c>s, then M(c) is upsampled to an image with the same resolution as M(s), and then Each pixel of M(s) is summed with the corresponding pixel of M(c) after upsampling.

The first threshold T ₁ set in the step ②-4=1.

The concrete process of the deep visual attention detection method in described step 3. is:

③-1. Decompose the current depth video frame into a Gaussian pyramid, and decompose it into n _L -layer depth video frames. Record the i-th layer depth video frame obtained after the Gaussian pyramid decomposition of the current depth video frame as D(i), and the i-th layer depth The width and height of the video frame D(i) are W/2 ⁱ and H/2 ⁱ respectively, where n _L is a positive integer less than 20, i∈[0,n _L -1], and W is the current depth video frame width, H is the height of the current depth video frame;

s，c ∈[0，n_L-1]，s＝c+δ，δ＝{-3，-2，-1，1，2，3}，

为归一化至区间的归一化函数，符号“||”为绝对值运算符号，D(c)为第c层深度视频帧，D(s)为第s层深度视频帧，符号

为D(c)与D(s)进行跨层级作差运算符，如果c＜s，则将D(s)上采样至与D(c)具有相同分辨率的图像上，然后将D(c)的各个像素与上采样后的D(s)相对应像素分别进行作差，如果c＞s，则将D(c)上采样至与D(s)具有相同分辨率的图像上，然后将D(s)的各个像素与上采样后的D(c)相对应像素分别进行作差，符号

为D(c)与D(s)进行跨层级相加运算符，如果c＜s，则将D(s)上采样至与D(c)具有相同分辨率的图像上，然后将D(c)的各个像素与上采样后的D(s)相对应像素分别进行求和，如果c＞s，则将D(c)上采样至与D(s)具有相同分辨率的图像上，然后将D(s)的各个像素与上采样后的D(c)相对应像素分别进行求和；③-2. Utilize the n _L depth video frames of the current depth video frame to extract the depth feature map of the current depth video frame, denoted as F _D , in,

s, c ∈ [0, n _L -1], s = c + δ, δ = {-3, -2, -1, 1, 2, 3},

to normalize to The normalization function of the interval, the symbol "||" is the absolute value operation symbol, D(c) is the depth video frame of the c-th layer, D(s) is the depth video frame of the s-th layer, the symbol

Perform a cross-level difference operator for D(c) and D(s), if c<s, then upsample D(s) to an image with the same resolution as D(c), and then use D(c Each pixel of ) corresponds to the upsampled D(s) The pixels are respectively differentiated. If c>s, D(c) is up-sampled to an image with the same resolution as D(s), and then each pixel of D(s) is combined with the up-sampled D(c ) for the corresponding pixels to make a difference respectively, and the symbol

Perform a cross-level addition operator for D(c) and D(s), if c<s, then upsample D(s) to an image with the same resolution as D(c), and then D(c ) and the corresponding pixels of the upsampled D(s) are summed separately, if c>s, then D(c) is upsampled to an image with the same resolution as D(s), and then Each pixel of D(s) is summed with the corresponding pixel of D(c) after upsampling;

③-3. Use the well-known 0 degree, π/4 degree, π/2 degree and 3π/4 degree direction Gabor filter to perform convolution operation on the current depth video frame to extract 0 degree, π/4 degree, π/ The four direction components of the 2 degree and 3π/4 degree directions are obtained to obtain the four direction component maps of the current depth video frame, and the four direction component maps are represented as O ₀ ^D , O _π/4 ^D , O _π/2 ^D and O _3π/4 ^D ; Gaussian pyramid decomposition is performed on the O ₀ ^D direction component map, O _π/4 ^D direction component map, O _π/2 ^D direction component map and O _3π/4 ^D direction component map of the current depth video frame , each is decomposed into n _L layer direction component graphs, and the i-th layer direction component graph obtained after Gaussian pyramid decomposition of the direction component graph in the θ degree direction is O _θ ^D (i), the width and height of O _θ ^D (i) are W/2 ⁱ and H/2 ⁱ respectively, where, θ∈{0, π/4, π/2, 3π/4}i∈[0, n _L -1], W is the width of the current depth video frame , H is the height of the current depth video frame;

s，c ∈[0，n_L-1]，s＝c+δ，δ＝{-3，-2，-1，1，2，3}，

为归一化至

区间的归一化函数，符号“||”为绝对值运算符号，O_θ ^D(c)为θ度方向的方向分量图的第c层方向分量图，O_θ ^D(s)为θ度方向的方向分量图的第s层方向分量图，符号

为O_θ ^D(c)与O_θ ^D(s)进行跨层级作差运算符，如果c＜s，则将O_θ ^D(s)上采样至与O_θ ^D(c)具有相同分辨率的图像上，然后将O_θ ^D(c)的各个像素与上采样后的O_θ ^D(s)相对应像素分别进行作差，如果c＞s，则将O_θ ^D(c)上采样至与O_θ ^D(s)具有相同分辨率的图像上，然后将O_θ ^D(s)的各个像素与上采样后的O_θ ^D(c)相对应像素分别进行作差，符号

为O_θ ^D(c)与O_θ ^D(s)进行跨层级相加运算符，如果c＜s，则将O_θ ^D(s)上采样至与O_θ ^D(c)具有相同分辨率的图像上，然后将O_θ ^D(c)的各个像素与上采样后的O_θ ^D(s)相对应像素分别进行求和，如果c＞s，则将O_θ ^D(c)上采样至与O_θ ^D(s)具有相同分辨率的图像上，然后将O_θ ^D(s)的各个像素与上采样后的O_θ ^D(c)相对应像素分别进行求和；3.-4, utilize the _nL layer direction component map of the direction component map of each degree direction of the current depth video frame to extract the preliminary depth direction feature map of the current depth video frame, denoted as F ' _DO , ${\stackrel{\&OverBar;}{f}}_{\mathrm{do}}^{\&prime;}=\frac{1}{4}\underset{\mathrm{\&theta;}\&Element;\{0,\frac{\mathrm{\&pi;}}{4},\frac{\mathrm{\&pi;}}{2},\frac{3\mathrm{\&pi;}}{4}\}}{\mathrm{\&Sigma;}}{\stackrel{\&OverBar;}{f}}_{o_{\mathrm{\&theta;}}},$ in,

s, c ∈ [0, n _L -1], s = c + δ, δ = {-3, -2, -1, 1, 2, 3},

to normalize to

The normalization function of the interval, the symbol "||" is the absolute value operation symbol, O _θ ^D (c) is the c-th layer direction component map of the direction component map in the θ degree direction, O _θ ^D (s) is the θ degree direction The s-th layer direction component map of the direction component map, symbol

Perform a cross-level difference operator for O _θ ^D (c) and O _θ ^D (s), if c<s, then upsample O _θ ^D (s) to the same resolution as O _θ ^D (c) On the image, then make a difference between each pixel of O _θ ^D (c) and the corresponding pixel of the up-sampled O _θ ^D (s), if c>s, then up-sample O _θ ^D (c) to the same O _θ ^D (s) has the same resolution on the image, and then make a difference between each pixel of O _θ ^D (s) and the corresponding pixel of O _θ ^D (c) after upsampling, the symbol

Perform a cross-level addition operator for O _θ ^D (c) and O _θ ^D (s), if c < s, then upsample O _θ ^D (s) to the same resolution as O _θ ^D (c) On the image, each pixel of O _θ ^D (c) is summed with the corresponding pixel of the upsampled O _θ ^D (s), if c>s, then O _θ ^D (c) is upsampled to the same value as O _θ ^D (s) on an image with the same resolution, and then sum each pixel of O _θ ^D (s) with the corresponding pixel of the upsampled O _θ ^D (c);

③-5. Using the known morphological dilation algorithm, take the block of size w ₁ ×h ₁ as the basic dilation unit to perform n ₁ dilation operations on the preliminary depth direction feature map F′ _DO of the current depth video frame to obtain the current depth video The depth direction feature map of the frame, denoted as F _DO ;

记S′_D中坐标为(x，y)的像素的像素值为s′_d(x，y)，其中，

为归一化至

区间的归一化函数；③-6. Utilize the depth feature map F _D and the depth direction feature map F _DO of the current depth video frame to obtain the distribution map of the preliminary depth visual attention of the current depth video frame, denoted as S′ _D ,

Note that the pixel value of the pixel whose coordinates are (x, y) in S′ _D is s′ _d (x, y), where,

to normalize to

The normalization function of the interval;

③-7. Using the distribution map S′ _D of the preliminary depth visual attention of the current depth video frame, obtain the distribution map S _D of the depth visual attention of the 3D video image jointly presented by the current depth video frame and the current texture video frame, denote S _D The pixel value of the pixel whose coordinates are (x, y) is s _d (x, y), s _d (x, y) = s′ _d (x, y) g(x, y), where, $g(x,\mathrm{the\; y})=\left\{\begin{array}{ccc}0.2& \mathrm{if}& x<b\left|\right|\mathrm{the\; y}<b\left|\right|x>W-b\left|\right|\mathrm{the\; y}>h-b\\ 0.4& \mathrm{else\; if}& x<2b\left|\right|\mathrm{the\; y}<2b\left|\right|x>W-2b\left|\right|\mathrm{the\; y}>h-2b\\ 0.6& \mathrm{else\; if}& x<3b\left|\right|\mathrm{the\; y}<3b\left|\right|x>W-3b\left|\right|\mathrm{the\; y}>h-3\mathrm{<figure-callout\; id="1"\; label="b"\; filenames="A2009101525200039H1.png"\; state="\{\{state\}\}">b</figure-callout>}\\ 1& & \mathrm{else}\end{array}\right.,$ W is the width of the current depth video frame, H is the height of the current depth video frame, b is the set second threshold, and the symbol "|| is an "or" operator.

In the step ③-5, w ₁ =8, h ₁ =8, n ₁ =2, and the second threshold b set in the step ③-7 is 16.

The specific process of the visual attention fusion method based on depth perception in described step 4. is:

范围内的系数，d(x，y)表示当前深度视频帧中坐标为(x，y)的像素的像素值，Q(d(x，y))表示尺度变换后的当前深度视频帧中坐标为(x，y)的像素的像素值；④-1. Carry out scale transformation to the current depth video frame by Q(d(x, y))=d(x, y)+γ, wherein, γ is the value in

Coefficients within the range, d(x, y) represents the pixel value of the pixel whose coordinates are (x, y) in the current depth video frame, Q(d(x, y)) represents the coordinates in the current depth video frame after scale transformation is the pixel value of the pixel at (x, y);

其中，K_D、K_M和K_I分别S_D、S_M以及S_I的加权系数，加权系数满足条件： $\underset{a\&Element;\{D,M,I\}}{\mathrm{\&Sigma;}}{K}_a=1,$ 0≤K_a≤1，

为归一化至

区间的归一化函数，s_D(x，y)、s_M(x，y)和s_I(x，y)分别表示S_D、S_M以及S_I中坐标为(x，y)的像素的像素值，Θ_ab(x，y)为视觉注意的相关值，Θ_ab(x，y)＝min(s_a(x，y)，s_b(x，y))，min()为取最小值函数，C_ab为相关系数，相关系数满足条件： $\underset{a,b\&Element;\{D,M,I\},a\&NotEqual;b}{\mathrm{\&Sigma;}}{C}_{\mathrm{ab}}=1,$ 0≤C_ab＜1，相关系数C_DM表示S_D与S_M的相关度，相关系数C_DI表示S_D与S_I的相关度，相关系数C_IM表示S_I与S_M的相关度，a，b ∈{D，M，I}且a≠b。④-2. Using the scale-transformed current depth video frame, the current depth video frame, and the depth visual attention distribution map _SD of the 3D video image jointly presented by the current depth video frame and the current texture video frame, and the motion of the current texture video frame The distribution map S _M of visual attention and the distribution map S _I of visual attention in the static image domain of the current texture video frame, the distribution map S of three-dimensional visual attention is obtained, and the coordinates in the distribution map S of three-dimensional visual attention are (x, y) The pixel value of the pixel is s(x, y),

Among them, K _D , K _M and K _I are the weighting coefficients of S _D , S _M and S _I respectively, and the weighting coefficients meet the conditions: $\underset{a\&Element;\{\mathrm{D.},m,I\}}{\mathrm{\&Sigma;}}{K}_a=1,$ 0≤K _a ≤1,

to normalize to

The normalization function of the interval, s _D (x, y), s _M (x, y) and s _I (x, y) represent the pixel with coordinates (x, y) in S _D , S _M and S _I respectively , Θ _ab (x, y) is the relevant value of visual attention, Θ _ab (x, y)=min(s _a (x, y), s _b (x, y)), min() is the The minimum value function, C _ab is the correlation coefficient, and the correlation coefficient satisfies the conditions: $\underset{a,b\&Element;\{\mathrm{D.},m,I\},a\&NotEqual;b}{\mathrm{\&Sigma;}}{C}_{\mathrm{ab}}=1,$ 0≤C _ab <1, the correlation coefficient C _DM represents the correlation between _SD and S _M , the correlation coefficient C _DI represents the correlation between SD and S _I , and the correlation coefficient C _IM represents the correlation between S _{I and} S _M , a , b ∈ {D, M, I} and a≠b.

In the described step 5., the distribution map S of three-dimensional visual attention is carried out the specific process of thresholding and macroblocking post-processing as:

⑤-1. The pixel value of the pixel whose coordinates are (x, y) is s(x, y) in the distribution map S that remembers the three-dimensional visual attention, defines the third threshold T _S , $T_S=k_T\&Center\; Dot;\underset{\mathrm{the\; y}=0}{\overset{h-1}{\mathrm{\&Sigma;}}}\underset{x=0}{\overset{W-1}{\mathrm{\&Sigma;}}}\mathrm{the\; s}(x,\mathrm{the\; y})/(W\&times;h),$ Among them, W is the width of the distribution map S of 3D visual attention, H is the height of the distribution map S of 3D visual attention, k _T ∈ (0, 3); create a new preliminary binary mask image, judge s(x, y ) ≥ T _S is true, if it is true, mark the pixel with coordinates (x, y) in the preliminary binary mask image as the pixel of interest, otherwise, set the coordinates (x, y) in the preliminary binary mask image The pixels of are marked as non-interest pixels;

⑤-2. Divide the preliminary binary mask image into (W/w ₂ )×(H/h ₂ ) blocks with a size of w ₂ ×h ₂ , and the blocks do not overlap each other, mark horizontally The block with coordinate u and ordinate v is B _{u, v} , where u ∈ [0, W/w ₂ -1], v ∈ [0, H/h ₂ -1], according to the preliminary binary mask Each block in the image determines whether the pixel in the corresponding block in the current texture video frame is a pixel of interest or a non-interest pixel. For block B _{u, v} , determine the pixel marked as a pixel of interest in block B _{u, v} Whether the number of is greater than the set fourth threshold T _b , wherein, 0≤T _b ≤w ₂ ×h ₂ , if yes, all pixels in the block corresponding to the block B _{u, v} in the current texture video frame Mark as a pixel of interest, and use the block corresponding to the block B _{u, v} as the region of interest block, otherwise, mark all the pixels in the block corresponding to the block B _{u, v} in the current texture video frame as non-interest pixels, And the block corresponding to block B _{u, v} is regarded as the non-interest area block, obtains the initial interest area mask image of the current texture video frame, and the preliminary interest area mask image is composed of the interest area block and the non-interest area block composition;

⑤-3. Mark all the pixels in the non-region of interest block closest to the region of interest block in the preliminary region of interest mask image as the N _R transition region of interest, and update the preliminary region of interest mask image ; Then, mark all the pixels in the non-ROI block closest to the _NRth- level transitional ROI in the updated preliminary ROI mask image as the _NR -1th transitional ROI, recursively Update the initial ROI mask image; repeat the recursive above process until the first-level transition ROI is marked; finally get the final ROI mask image of the current texture video frame, the final ROI mask The image is composed of ROI blocks, _NR- level transition ROIs and non-ROI blocks;

⑤-4. Record the pixel value of the pixel whose coordinates are (x, y) in the final region of interest mask image r(x, y), and place the non-interest region block in the final region of interest mask image Set the pixel values of all pixels in r(x, y)=255, and set the pixel values of all pixels in the N _R- level transition ROI in the final ROI mask image as $r(x,\mathrm{the\; y})=\frac{e}{N_R+1}\&times;f(x,\mathrm{the\; y}),$ Set the pixel values of all pixels in the ROI block in the final ROI mask image to r(x, y)=f(x, y), to obtain the ROI of the current texture video frame, where e Indicates the series of the transition region of interest, e∈[1, N _R ], f(x, y) indicates the pixel value of the pixel whose coordinates are (x, y) in the current texture video frame.

In the step ⑤-2, w ₂ =16, h ₂ =16, and the set fourth threshold T _h =50.

Compared with the prior art, the present invention has the advantage of jointly utilizing the temporally synchronized texture video frame and the depth video frame corresponding to the texture video frame, firstly by extracting the visual attention of the static image domain of the texture video frame, and obtaining the The distribution map of visual attention in the static image domain, extracting motion visual attention through temporally continuous texture video frames, obtaining the distribution map of motion visual attention of texture video frames, and obtaining depth video frames and texture by extracting depth visual attention of depth video frames The depth visual attention distribution map of the 3D video image jointly presented by the video frames, and then using the obtained static image domain visual attention distribution map, motion visual attention distribution map and depth visual attention distribution map and depth information, through depth-based The perceptual fusion method obtains the distribution map of three-dimensional (stereo) visual attention that conforms to the characteristics of human stereo vision, and then after thresholding and macroblocking post-processing operations, the final video region of interest and its corresponding sense of interest conforming to human stereo perception are obtained. Mask images of regions of interest and non-interest regions. The region of interest extracted by this method combines static image domain visual attention, motion visual attention and depth visual attention, which effectively suppresses the inherent singleness and inaccuracy of each visual attention extraction, and solves the problem of complex background in static image domain visual attention. The noise problem solves the problem that motion visual attention cannot extract local motion and the region of interest with small motion range, thereby improving the calculation accuracy, enhancing the stability of the algorithm, and being able to extract the region of interest from the background with complex texture and the moving environment. In addition, the region of interest obtained by this method not only conforms to the visual characteristics of the human eye for static texture video frames and the visual characteristics of the human eye for moving objects, but also conforms to the strong sense of depth or short distance in stereo vision. The depth perception characteristics of the object of interest conform to the semantic characteristics of human stereo vision.

Description of drawings

Figure 1a is the color video frame at time t in the test sequence "Ballet" two-dimensional color video;

Figure 1b is the color video frame at time t in the test sequence "Door Flower" two-dimensional color video;

Figure 2a is the depth video frame at time t in the depth video corresponding to the test sequence "Ballet" two-dimensional color video;

Figure 2b is the depth video frame at time t in the depth video corresponding to the test sequence "Door Flower" two-dimensional color video;

Fig. 3 is the overall flow chart of the inventive method;

Fig. 4 is the flow chart diagram that adopts known static image visual attention detection method to detect the static image domain visual attention of current texture video frame;

Fig. 5 is the flowchart of motion visual attention detection method;

Fig. 6 is the flowchart of deep visual attention detection method;

Figure 7a is a luminance feature map of the color video frame at time t in the test sequence "Ballet" two-dimensional color video;

Figure 7b is a chromaticity feature map of the color video frame at time t in the test sequence "Ballet" two-dimensional color video;

Figure 7c is the direction feature map of the color video frame at time t in the test sequence "Ballet" two-dimensional color video;

Figure 8a is a distribution diagram of visual attention in the static image domain of the color video frame at moment t in the test sequence "Ballet" two-dimensional color video;

Fig. 8 b is the distribution diagram of the motion visual attention of the color video frame at moment t in the test sequence "Ballet" two-dimensional color video;

Fig. 8c is a distribution diagram of the depth visual attention of the three-dimensional video image jointly presented by the color video frame and the corresponding depth video frame at time t in the test sequence "Ballet" two-dimensional color video;

Fig. 9 is the distribution diagram of the three-dimensional visual attention obtained after the color video frame and the corresponding depth video frame are processed by the present invention at time t in the test sequence "Ballet" two-dimensional color video;

Fig. 10a is the final ROI mask image extracted by the present invention of the texture video frame at time t of the test sequence "Ballet";

Fig. 10b is the region of interest extracted by the present invention of the texture video frame at time t of the test sequence "Ballet";

Fig. 11a is the region of interest extracted by the traditional region of interest extraction method based only on visual attention clues in the static image domain of the texture video frame at time t of the test sequence "Ballet";

Figure 11b is the region of interest extracted by the traditional region of interest extraction method based only on motion visual attention clues of the texture video frame at time t of the test sequence "Ballet";

Figure 11c is the region of interest extracted by the traditional static image domain visual attention and motion visual attention joint region of interest extraction method of the texture video frame at time t of the test sequence "Ballet";

Figure 12a is a luminance feature map of the color video frame at time t in the test sequence "Door Flower" two-dimensional color video;

Figure 12b is a chromaticity feature map of the color video frame at time t in the test sequence "Door Flower" two-dimensional color video;

Figure 12c is the direction feature map of the color video frame at time t in the test sequence "Door Flower" two-dimensional color video;

Figure 13a is a distribution diagram of visual attention in the static image domain of the color video frame at time t in the test sequence "Door Flower" two-dimensional color video;

Figure 13b is a distribution diagram of the motion visual attention of the color video frame at moment t in the test sequence "Door Flower" two-dimensional color video;

Figure 13c is a distribution diagram of the depth visual attention of the three-dimensional video image jointly presented by the color video frame and the corresponding depth video frame at time t in the test sequence "Door Flower" two-dimensional color video;

Fig. 14 is the distribution diagram of the three-dimensional visual attention obtained after the color video frame and the corresponding depth video frame are processed by the present invention at time t in the two-dimensional color video of the test sequence "Door Flower";

Fig. 15a is the final ROI mask image extracted by the present invention for the texture video frame at time t of the test sequence "Door Flower";

Fig. 15b is the region of interest extracted by the present invention of the texture video frame at time t of the test sequence "Door Flower";

Figure 16a is the region of interest extracted by the traditional method of extracting the region of interest based only on the visual attention clues of the static image domain for the texture video frame at time t of the test sequence "Door Flower";

Figure 16b is the region of interest extracted by the traditional method of extracting the region of interest only based on the motion visual attention clues of the texture video frame at time t of the test sequence "Door Flower";

Fig. 16c is the region of interest extracted by the joint region of interest extraction method of static image domain visual attention and motion visual attention of the texture video frame at time t of the test sequence "Door Flower".

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments.

A method for extracting a region of interest in a video based on visual attention in the present invention mainly utilizes the information of the texture video and the information of the depth video synchronized in time to extract the region of interest in the video. In this embodiment, the texture video mainly uses Two-dimensional color video, texture video Taking the test sequence "Ballet" two-dimensional color video and "Door Flower" two-dimensional color video as examples, Figure 1a shows the color video frame at time t in the test sequence "Ballet" two-dimensional color video , Figure 1b shows the color video frame at time t in the test sequence "Door Flower" two-dimensional color video, Figure 2a shows the depth video frame at time t in the depth video corresponding to the test sequence "Ballet" two-dimensional color video, Figure 2b is the depth video frame at time t in the depth video corresponding to the two-dimensional color video of the test sequence "Door _Flower ". The grayscale value of the degree map represents the relative distance of the object represented by each pixel in the depth video frame to the camera. The size of the texture video frame at each moment in the texture video is defined as W×H, and for the depth video frame at each moment in the depth video corresponding to the texture video, if the size of the depth video frame is different from the size of the texture video frame , then the existing scale transformation and interpolation methods are generally used to set the size of the depth video frame to the same size as the texture video frame, which is also W×H, and W is the texture video frame at each moment in the texture video. The width of the depth video frame at each moment in the wide or depth video, H is the height of the texture video frame at each moment in the texture video or the height of the depth video frame at each moment in the depth video, and the size of the depth video frame is set to be the same as The size of the texture video frames is the same, the purpose is to extract the video region of interest more conveniently.

The overall block diagram of the inventive method as shown in Figure 3, specifically comprises the following steps:

①Define the two-dimensional color video as texture video, and define the size of the texture video frame at each moment in the texture video as W×H, where W is the width of the texture video frame at each moment in the texture video, and H is the The height of the texture video frame at time, record the texture video frame at time t in the texture video as F _t , define the texture video frame F _t at time t in the texture video as the current texture video frame, and use the known static image visual attention detection method to detect The static image domain visual attention of the current texture video frame obtains the distribution map of the static image domain visual attention of the current texture video frame, which is denoted as S _I , and the size of the distribution map S _I of the static image domain visual attention of the current texture video frame is: W×H and it is a grayscale image represented by _ZS bit depth. The larger the pixel value of a certain pixel in the grayscale image, the higher the relative attention of the human eye to the corresponding pixel of the current texture video frame, and the higher the pixel value is. A small value means that the human eye pays less attention to the current texture video frame.

In this specific embodiment, adopt known static image visual attention detection method to detect the flow chart of the still image domain visual attention of current texture video frame as shown in Figure 4, in Figure 4 each rectangle represents a kind of data processing process, Each rhombus represents an image, and diamonds of different sizes represent images of different resolutions, which are the input and output data of the corresponding operation; the current texture video frame is an image in RGB format, and each pixel in the image is composed of R, G and B three color channel representations, first linearly transform each color channel component of each pixel of the current texture video frame, and decompose it into a luminance component map and two chrominance component maps, namely the red-green component map and the blue-yellow component map, the brightness Component diagram, red-green component diagram and blue-yellow component diagram are denoted as I, RG and BY respectively, and the pixel value of luminance component diagram I in (x, y) coordinate is represented as I _{x, y} =(r _{x, y} +g _{x , y} +b _{x, y} )/3, wherein, I _{x, y} represent the pixel value of the brightness component at (x, y) coordinates, r _{x, y} , g _{x, y} , b _{x, y} are respectively the current texture video The pixel values of the pixels of the RGB three color channels of the frame at the (x, y) coordinates, and the pixel values of the two chromaticity component maps of the red-green component map RG and the blue-yellow component map BY respectively at the (x, y) coordinates represent for:

$\&ForAll;l\&Element;\left\{I\right\}\&cup;\{\mathrm{RG},\mathrm{BY}\}\&cup;\{O_0^T,O_{\mathrm{\&pi;}/4}^T,O_{\mathrm{\&pi;}/2}^T,O_{3\mathrm{\&pi;}/4}^T\},$ 其中，s，c∈[0，n_L-1]，s＝c+δ，δ＝{-3，-2，-1，1，2，3}，为归一化至-1区间的归一化函数，

表示最容易引起注意，0表示最不容易引起注意，l(c)表示分量图l的第c层层分量图，l(s)表示分量图l的第s层层分量图，符号

为l(c)与l(s)进行跨层级作差运算符，如果c＜s则将l(s)上采样至与l(c)具有相同分辨率的图像上，然后将l(c)与上采样后的l(s)对应像素分别进行作差，如果c＞s，则将l(c)上采样至与l(s)具有相同分辨率的图像上，然后将l(s)与上采样后的l(c)对应像素分别进行作差，符号

表示l(c)与l(s)跨层级相加运算符，如果c＜s则将l(s)上采样至与l(c)具有相同分辨率的图像上，然后将l(c)与上采样后的l(s)对应像素分别进行求和，如果c＞s，则将l(c)上采样至与l(s)具有相同分辨率的图像上，然后将l(s)与上采样后的l(c)对应像素分别进行求和，将各分量的特征图线性融合并归一化，得到当前纹理视频帧的静态图像域视觉注意的分布图S_I，

$\left\{\begin{array}{c}{R}_{x,\mathrm{the\; y}}=r_{x,\mathrm{the\; y}}-(g_{x,\mathrm{the\; y}}+b_{x,\mathrm{the\; y}})/2\\ G_{x,\mathrm{the\; y}}=g_{x,\mathrm{the\; y}}-(r_{x,\mathrm{the\; y}}+b_{x,\mathrm{the\; y}})/2\\ B_{x,\mathrm{the\; y}}=b_{x,\mathrm{the\; y}}-(r_{x,\mathrm{the\; y}}+g_{x,\mathrm{the\; y}})/2\\ Y_{x,\mathrm{the\; y}}=r_{x,\mathrm{the\; y}}+g_{x,\mathrm{the\; y}}-2\left(\right|r_{x,\mathrm{the\; y}}-g_{x,\mathrm{the\; y}}|+b_{x,\mathrm{the\; y}})\\ {\mathrm{RG}}_{x,\mathrm{the\; y}}=R_{x,\mathrm{the\; y}}-G_{x,\mathrm{the\; y}}\\ {\mathrm{BY}}_{x,\mathrm{the\; y}}=B_{x,\mathrm{the\; y}}-Y_{x,\mathrm{the\; y}}\end{array}\right.,$ Among them, RG _{x, y} represents the pixel value of the red-green component map RG at the (x, y) coordinates, BY _{x, y} represents the pixel value of the blue-yellow component map BY at the (x, y) coordinates; the known Gabor filter is used Extract the 0 degree, 45 degree, 90 degree and 135 degree four direction component diagrams of the brightness component diagram, and the extracted four direction component diagrams are respectively recorded as O _θ ^T , $\mathrm{\&theta;}\&Element;\{0,\frac{\mathrm{\&pi;}}{4},\frac{\mathrm{\&pi;}}{2},\frac{3\mathrm{\&pi;}}{4}\};$ Decompose a luminance component, two chrominance components and four direction components into a Gaussian pyramid, each decomposed into n _L layers, where n _L is a positive integer less than 20, and each component map is uniformly represented by l, and each component is recorded The layer component graph of the i-th layer obtained after graph l is decomposed by Gaussian pyramid is l(i), where, i∈[0,n _L -1], $l\&Element;\left\{I\right\}\&cup;\{\mathrm{RG},\mathrm{BY}\}\&cup;\{o_0^T,o_{\mathrm{\&pi;}/4}^T,o_{\mathrm{\&pi;}/2}^T,o_{3\mathrm{\&pi;}/4}^T\},$ Therefore, the 7 component graphs produced in total, each component graph is decomposed into n _L layer component graphs, a total of 7×n _L layer layer component graphs, and the value of n _L in this embodiment is 9; using the extracted layer component graphs Calculate the feature map of each component (chroma, brightness, direction component) as:

$\&ForAll;l\&Element;\left\{I\right\}\&cup;\{\mathrm{RG},\mathrm{BY}\}\&cup;\{o_0^T,o_{\mathrm{\&pi;}/4}^T,o_{\mathrm{\&pi;}/2}^T,o_{3\mathrm{\&pi;}/4}^T\},$ Among them, s, c ∈ [0, n _L -1], s = c + δ, δ = {-3, -2, -1, 1, 2, 3}, to normalize to The normalization function of the -1 interval,

Indicates that it is the easiest to attract attention, 0 indicates that it is the least likely to attract attention, l(c) indicates the c-th layer component graph of component graph l, l(s) represents the s-th layer component graph of component graph l, and the symbol

Perform a cross-level difference operator for l(c) and l(s), if c<s, upsample l(s) to an image with the same resolution as l(c), and then use l(c) Make a difference with the pixel corresponding to the upsampled l(s), if c>s, then upsample l(c) to an image with the same resolution as l(s), and then compare l(s) with The corresponding pixels of l(c) after upsampling are respectively subtracted, and the symbol

Indicates the cross-level addition operator of l(c) and l(s), if c<s, upsample l(s) to an image with the same resolution as l(c), and then combine l(c) with The corresponding pixels of the upsampled l(s) are summed separately. If c>s, then upsample l(c) to an image with the same resolution as l(s), and then combine l(s) with the above The corresponding pixels of l(c) after sampling are summed separately, and the feature maps of each component are linearly fused and normalized to obtain the distribution map S _I of visual attention in the static image domain of the current texture video frame,

The size of each image of the test sequence "Ballet" and "Door Flower" is 1024×768, the luminance feature map, chrominance feature map and direction feature map of the color video frame at time t in the test sequence "Ballet" two-dimensional color video As shown in Figure 7a, Figure 7b and Figure 7c respectively; the brightness feature map, chromaticity feature map and direction feature map of the color video frame at time t in the test sequence "Door Flower" two-dimensional color video are shown in Figure 12a and Figure 12b respectively and shown in Figure 12c. In this specific embodiment, Z _S =8, that is, each pixel of the distribution map S _I of visual attention in the static image domain adopts 8-bit depth representation, and the color video frame at time t in the test sequence "Ballet" two-dimensional color video The distribution of visual attention in the static image domain is shown in Figure 8a; the distribution of visual attention in the static image domain of the color video frame at time t in the test sequence "Door Flower" two-dimensional color video is shown in Figure 13a. Here, the visual attention detection method in the static image domain may also use other known visual attention detection methods.

② Use the motion visual attention detection method to detect the motion visual attention of the current texture video frame, and obtain the distribution map of the motion visual attention of the current texture video frame, denoted as S _M , the size of the motion visual attention distribution map S _M of the current texture video frame The size is W×H and it is a grayscale image represented by _ZS bit depth. The larger the pixel value of a certain pixel in the grayscale image, the higher the relative motion attention of the human eye to the corresponding pixel of the current texture video frame, A smaller pixel value indicates that human eyes pay less attention to the relative motion of the corresponding pixel of the current texture video frame.

In this specific embodiment, the flow chart of the motion visual attention detection method is as shown in Figure 5, and the specific process of the motion visual attention detection method is:

②-1. Record the texture video frame at time t+j continuous with the current texture video frame in the texture video as F _t+j , record the texture video frame at the time tj time continuous with the current texture video frame in the texture video is F _tj , wherein, j∈(0, _NF /2], _NF is a positive integer less than 10, and _NF = 4 is taken in the specific application process of this embodiment, that is, the current texture video frame and the current texture The first two frames and the last two frames of the video frame jointly extract the motion region of the texture video.

②-2, using the known optical flow method to calculate the motion vector image of the current texture video frame and the texture video frame F _{t+j at the moment t+j} in the horizontal direction and the motion vector image of the vertical direction, and the current texture video frame and The motion vector image of the texture video frame F _tj in the horizontal direction and the motion vector image in the vertical direction of the texture video frame F tj at the moment tj, record the motion vector image of the current texture video frame and the texture video frame F t+j of the moment t _+j in the horizontal direction as The motion vector image of V _t+j ^H and the vertical direction is V _t+j ^V , and the motion vector image of the current texture video frame and the texture video frame F tj at the moment _tj in the horizontal direction is V _tj ^H and the motion vector image of the vertical direction The motion vector image is V _tj ^V , V _t+j ^H , V _t+j ^V , V _tj ^H and V _tj ^V have width W and height H.

②-3. Superimpose the absolute value of V _t+j ^H and the absolute value of V _t+j ^V to obtain the motion amplitude image of the current texture video frame and the texture video frame F _t+ j at time t+j, denoted as M _{t +j} , $m_{t+j}=\left|V_{t+j}^h\right|+\left|V_{t+j}^V\right|,$ Record the motion amplitude value of the pixel whose coordinates are (x, y) in M _t+j is m _t+j (x, y); superimpose the absolute value of V _tj ^H and the absolute value of V _tj ^V to obtain the current texture video frame and the motion amplitude image of texture video frame F _tj at time tj, denoted as M _tj , $m_{t-j}=\left|V_{t-j}^h\right|+\left|V_{t-j}^V\right|,$ Note that the motion amplitude value of the pixel whose coordinates are (x, y) in M _tj is m _tj (x, y).

否则，确定联合运动图M_j ^Δ中坐标为(x，y)的像素的像素值为0，其中，min()为取最小值函数。在此，第一阈值T₁＝1，以滤除非常微小的相机参数抖动所造成的小噪声点。②-4. Use the current texture video frame and the texture video frame F t+j at time _t+j and the texture video frame F tj at time _tj to extract the joint motion map, denoted as M _j ^Δ , and extract the joint motion map M _j ^Δ The specific process is: judging the current texture video frame and the texture video frame F t+j at the time _t+j of each pixel in the motion range image M _t+j and the current texture video frame and the texture video frame F tj at the time _tj Whether the minimum value of the motion amplitude value of the pixel corresponding to the coordinate in the motion amplitude image M _tj is greater than the set first threshold T ₁ , if yes, then determine the pixel value M of the pixel corresponding to the coordinate in the joint motion map M _j ^Δ The average of the sum of the motion amplitude values of the pixels corresponding to the coordinates in _t+j and M _tj , otherwise, the pixel value of the pixel corresponding to the coordinates in the joint motion map M _j ^Δ is determined to be 0; for M _t+j , the coordinates are (x , y) and the pixel whose coordinates are (x, y) in M _tj , determine whether min(m _t+j (x, y), m _tj (x, y)) is greater than the set first threshold T ₁ , if yes, determine the pixel value of the pixel with coordinates (x, y) in the joint motion map M _j ^Δ

Otherwise, it is determined that the pixel value of the pixel with coordinates (x, y) in the joint motion map M _j ^Δ is 0, where min() is a minimum value function. Here, the first threshold T ₁ =1 to filter out small noise points caused by very slight camera parameter shakes.

2.-5. The weighted joint motion map obtained from the weighted joint motion map of the current texture video frame is obtained by the weighted joint motion map of each moment from 1 moment to _NF /2 moment with the t moment in time, denoted as M, and the number of the current texture video frame The pixel value of the pixel whose coordinates are (x, y) in the weighted joint motion map M is m(x, y), $m(x,\mathrm{the\; y})=\underset{j=1}{\overset{N_f/2}{\mathrm{\&Sigma;}}}{\mathrm{\&zeta;}}_j{m}_j^{\mathrm{\&Delta;}}(x,\mathrm{the\; y}),$ Among them, m _j ^Δ (x, y) represents the pixel value of the pixel whose coordinates are (x, y) in the joint motion map M _j ^Δ at time j at a distance from time t in time, ζ _j is the weighting coefficient, and the weighting coefficient ζ _meet $\underset{j=1}{\overset{N_f/2}{\mathrm{\&Sigma;}}}{\mathrm{\&zeta;}}_j=1.$

In the video, the moving object is the main area of interest. However, due to the different types of motion, people's attention is different. The motion types of the video are mainly divided into the following two types. The first type is the case of still camera shooting. , the background is still, and the moving object is the main object of interest; the second type, for the case of a moving camera shooting, the background moves globally, while the moving object and the camera remain relatively still or appear in the background. It is still an object of interest; according to the above analysis, the people's motion attention area mainly comes from the motion properties of the object that are different from the motion properties of the background environment, and it is an area with a large motion contrast. Therefore, the following steps can be used to obtain motion visual attention.

②-6. Decompose the weighted joint motion map M of the current texture video frame into a Gaussian pyramid, decompose it into n _L layers of weighted joint motion maps, and record the i-th layer weighted joint motion map obtained after the weighted joint motion map M is decomposed by the Gaussian pyramid is M(i), the width and height of the i-th layer weighted joint motion map M(i) are W/2 ⁱ and H/2 ⁱ respectively, where n _L is a positive integer less than 20, i∈[0,n _L -1], the 0th layer is the bottom layer, the nth _L -1 layer is the highest layer, W is the width of the current texture video frame, and H is the height of the current texture video frame; in the specific application process of the present embodiment, n The value of _L is 9.

s，c ∈[0，n_L-1]，s＝c+δ，δ＝{-3，-2，-1，1，2，3}，

为归一化至

区间的归一化函数，符号“||”为绝对值运算符号，M(c)为第c层加权联合运动图，M(s)为第s层加权联合运动图，符号

为M(c)与M(s)进行跨层级作差运算符，如果c＜s，则将M(s)上采样至与M(c)具有相同分辨率的图像上，然后将M(c)的各个像素与上采样后的M(s)相对应像素分别进行作差，如果c＞s，则将M(c)上采样至与M(s)具有相同分辨率的图像上，然后将M(s)的各个像素与上采样后的M(c)相对应像素分别进行作差，符号为M(c)与M(s)进行跨层级相加运算符，如果c＜s，则将M(s)上采样至与M(c)具有相同分辨率的图像上，然后将M(c)的各个像素与上采样后的M(s)相对应像素分别进行求和，如果c＞s，则将M(c)上采样至与M(s)具有相同分辨率的图像上，然后将M(s)的各个像素与上采样后的M(c)相对应像素分别进行求和。②-7. Utilize the n-layer n _L weighted joint motion map of the weighted joint motion map M of the current texture video frame to extract the distribution map S _M of the motion visual attention of the current texture video frame, and record the coordinates in S _M as (x, y ) pixel value of the pixel s _m (x, y), S _M = F _M , where,

s, c ∈ [0, n _L -1], s = c + δ, δ = {-3, -2, -1, 1, 2, 3},

to normalize to

The normalization function of the interval, the symbol "||" is the absolute value operation symbol, M(c) is the weighted joint motion map of the c-th layer, M(s) is the weighted joint motion map of the s-th layer, the symbol

Perform a cross-level difference operator for M(c) and M(s), if c<s, then upsample M(s) to an image with the same resolution as M(c), and then M(c ) and the corresponding pixels of the upsampled M(s) are respectively differentiated. If c>s, then M(c) is upsampled to an image with the same resolution as M(s), and then The difference between each pixel of M(s) and the corresponding pixel of M(c) after upsampling is respectively performed, and the sign Perform a cross-level addition operator for M(c) and M(s), if c<s, then upsample M(s) to an image with the same resolution as M(c), and then M(c ) and the corresponding pixels of the upsampled M(s) are summed separately, if c>s, then M(c) is upsampled to an image with the same resolution as M(s), and then Each pixel of M(s) is summed with the corresponding pixel of M(c) after upsampling.

The distribution diagram of motion visual attention obtained after processing the color video frame at time t in the test sequence "Ballet" two-dimensional color video is shown in Figure 8b; the color video frame at time t in the test sequence "Door Flower" two-dimensional color video The distribution diagram of motion visual attention obtained after the video frames are processed in this step is shown in Fig. 13b.

范围的灰度值表示深度视频帧中的各个像素所表示的拍摄到的对象到拍摄相机的相对距离，灰度值0对应最大深度，灰度值

对应最小深度，将深度视频中各时刻的深度视频帧的尺寸大小均设置为W×H，W为深度视频中各时刻的深度视频帧的宽，H为深度视频中各时刻的深度视频帧的高，记深度视频中t时刻的深度视频帧为D_t，定义深度视频中t时刻的深度视频帧D_t为当前深度视频帧，采用深度视觉注意检测方法检测当前深度视频帧与当前纹理视频帧联合展现的三维视频图像的深度视觉注意，得到三维视频图像的深度视觉注意的分布图，记为S_D，三维视频图像的深度视觉注意的分布图S_D的尺寸大小为W×H且其为Z_S比特深度表示的灰度图，该灰度图中某一像素的像素值越大表示人眼对当前纹理视频帧的对应像素的相对深度注意程度越高，像素值越小表示人眼对当前纹理视频帧的相对深度注意程度越低。本实施例中，深度视频帧的每个像素由Z_D＝8比特深度表示，视觉注意分布图的每个像素由Z_S＝8比特深度表示。③Defining the depth video frame corresponding to the texture video at each moment is a grayscale image represented by the Z _D bit depth, and its 0 to

The grayscale value of the range indicates the relative distance from the captured object represented by each pixel in the depth video frame to the shooting camera. The grayscale value 0 corresponds to the maximum depth, and the grayscale value

Corresponding to the minimum depth, the size of the depth video frame at each moment in the depth video is set to W×H, W is the width of the depth video frame at each moment in the depth video, and H is the width of the depth video frame at each moment in the depth video High, record the depth video frame at time t in the depth video as D _t , define the depth video frame D _t at time t in the depth video as the current depth video frame, and use the depth visual attention detection method to detect the current depth video frame and the current texture video frame The depth visual attention of the 3D video images jointly displayed, the distribution map of the depth visual attention of the 3D video images is obtained, denoted as _SD , the size of the distribution map _SD of the depth visual attention of the 3D video images is W×H and it is The grayscale image represented by Z _S bit depth, the larger the pixel value of a pixel in the grayscale image, the higher the human eye’s attention to the relative depth of the corresponding pixel of the current texture video frame, and the smaller the pixel value, the human eye’s attention to the corresponding pixel of the current texture video frame. The lower the relative depth attention of the current texture video frame. In this embodiment, each pixel of the depth video frame is represented by Z _D =8 bit depth, and each pixel of the visual attention distribution map is represented by Z _S =8 bit depth.

The unique stereoscopic effect is the main feature that distinguishes stereoscopic video from traditional single-channel video. For the visual attention of stereoscopic video, the sense of depth mainly affects the user's visual attention through two aspects. Scenes (or objects) are generally more interesting than scenes (or objects) far away from the camera array; on the other hand, the depth discontinuity region provides the user with a strong depth contrast. In this specific embodiment, the flow chart of the deep visual attention detection method is shown in Figure 6, and the specific process of the deep visual attention detection method is:

③-1. Decompose the current depth video frame into a Gaussian pyramid, and decompose it into n _L -layer depth video frames. After the Gaussian pyramid decomposition of the current depth video frame, obtain the i-th layer depth video frame as D(i), and the i-th layer depth video frame The width and height of frame D(i) are W/2 ⁱ and H/2 ⁱ respectively, where n _L is a positive integer less than 20, i∈[0, n _L -1], and the 0th layer is the bottom layer, The maximum resolution is D(0)=D _t , the n _L -1th layer is the highest layer, and the resolution is the smallest, W is the width of the current depth video frame, and H is the height of the current depth video frame.

s，c ∈[0，n_L-1]，s＝c+δ，δ＝{-3，-2，-1，1，2，3}，为归一化至区间的归一化函数，符号“||”为绝对值运算符号，D(c)为第c层深度视频帧，D(s)为第s层深度视频帧，符号

为D(c)与D(s)进行跨层级作差运算符，如果c＜s，则将D(s)上采样至与D(c)具有相同分辨率的图像上，然后将D(c)的各个像素与上采样后的D(s)相对应像素分别进行作差，如果c＞s，则将D(c)上采样至与D(s)具有相同分辨率的图像上，然后将D(s)的各个像素与上采样后的D(c)相对应像素分别进行作差，符号

为D(c)与D(s)进行跨层级相加运算符，如果c＜s，则将D(s)上采样至与D(c)具有相同分辨率的图像上，然后将D(c)的各个像素与上采样后的D(s)相对应像素分别进行求和，如果c＞s，则将D(c)上采样至与D(s)具有相同分辨率的图像上，然后将D(s)的各个像素与上采样后的D(c)相对应像素分别进行求和。③-2. Utilize the n _L depth video frames of the current depth video frame to extract the depth feature map of the current depth video frame, denoted as F _D , in,

s, c ∈ [0, n _L -1], s = c + δ, δ = {-3, -2, -1, 1, 2, 3}, to normalize to The normalization function of the interval, the symbol "||" is the absolute value operation symbol, D(c) is the depth video frame of the c-th layer, D(s) is the depth video frame of the s-th layer, the symbol

Perform a cross-level difference operator for D(c) and D(s), if c<s, then upsample D(s) to an image with the same resolution as D(c), and then use D(c ) and the corresponding pixels of the upsampled D(s) are respectively differentiated, if c>s, then D(c) is upsampled to an image with the same resolution as D(s), and then Each pixel of D(s) is different from the corresponding pixel of D(c) after upsampling, and the symbol

Perform a cross-level addition operator for D(c) and D(s), if c<s, then upsample D(s) to an image with the same resolution as D(c), and then D(c ) and the corresponding pixels of the upsampled D(s) are summed separately, if c>s, then D(c) is upsampled to an image with the same resolution as D(s), and then Each pixel of D(s) is summed with the corresponding pixel of D(c) after upsampling.

③-3. The depth edge area with large depth difference gives the user a stronger sense of depth, so the depth strong edge area in the current depth video frame is another important area of interest for depth visual attention, so the well-known 0 degree, π/4 degree, π/2 degree and 3π/4 degree direction Gabor filter performs convolution operation on the current depth video frame to extract 0 degree, π/4 degree, π/2 degree and 3π/4 degree The four direction components of the direction are obtained to obtain the four direction component maps of the current depth video frame, and the four direction component maps are respectively represented as O ₀ ^D , O _π/4 ^D , O _π/2 ^D and O _3π/4 ^D ; The O ₀ ^D direction component map, O _π/4 ^D direction component map, O _π/2 ^D direction component map and O _3π/4 ^D direction component map of the current depth video frame are respectively decomposed into a Gaussian pyramid, each decomposed into n _L layers Direction component map, record the direction component map in the θ-degree direction decomposed by the Gaussian pyramid, the i-th layer direction component map is O _θ ^D (i), the width and height of O _θ ^D (i) are respectively W/2 ⁱ and H/2 ⁱ , where, θ∈{0, π/4, π/2, 3π/4}i∈[0, n _L -1], the 0th layer is the bottom layer, $o_{\mathrm{\&theta;}}^{\mathrm{D.}}\left(0\right)=o_{\mathrm{\&theta;}}^{\mathrm{D.}},$ The n _L -1th layer is the highest layer, W is the width of the current depth video frame, and H is the height of the current depth video frame.

为归一化至

区间的归一化函数，符号“||”为绝对值运算符号，O_θ ^D(c)为θ度方向的方向分量图的第c层方向分量图，O_θ ^D(s)为θ度方向的方向分量图的第s层方向分量图，符号

为O_θ ^D(c)与O_θ ^D(s)进行跨层级作差运算符，如果c＜s，则将O_θ ^D(s)上采样至与O_θ ^D(c)具有相同分辨率的图像上，然后将O_θ ^D(c)的各个像素与上采样后的O_θ ^D(s)相对应像素分别进行作差，如果c＞s，则将O_θ ^D(c)上采样至与O_θ ^D(s)具有相同分辨率的图像上，然后将O_θ ^D(s)的各个像素与上采样后的O_θ ^D(c)相对应像素分别进行作差，符号

为O_θ ^D(c)与O_θ ^D(s)进行跨层级相加运算符，如果c＜s，则将O_θ ^D(s)上采样至与O_θ ^D(c)具有相同分辨率的图像上，然后将O_θ ^D(c)的各个像素与上采样后的O_θ ^D(s)相对应像素分别进行求和，如果c＞s，则将O_θ ^D(c)上采样至与O_θ ^D(s)具有相同分辨率的图像上，然后将O_θ ^D(s)的各个像素与上采样后的O_θ ^D(c)相对应像素分别进行求和。3.-4, utilize the _nL layer direction component map of the direction component map of each degree direction of the current depth video frame to extract the preliminary depth direction feature map of the current depth video frame, denoted as F ' _DO , ${\stackrel{\&OverBar;}{f}}_{\mathrm{do}}^{\&prime;}=\frac{1}{4}\underset{\mathrm{\&theta;}\&Element;\{0,\frac{\mathrm{\&pi;}}{4},\frac{\mathrm{\&pi;}}{2},\frac{3\mathrm{\&pi;}}{4}\}}{\mathrm{\&Sigma;}}{\stackrel{\&OverBar;}{f}}_{o_{\mathrm{\&theta;}}},$ in, s, c ∈ [0, n _L -1], s = c + δ, δ = {-3, -2, -1, 1, 2, 3},

to normalize to

The normalization function of the interval, the symbol "||" is the absolute value operation symbol, O _θ ^D (c) is the c-th layer direction component map of the direction component map in the θ degree direction, O _θ ^D (s) is the θ degree direction The s-th layer direction component map of the direction component map, symbol

Perform a cross-level difference operator for O _θ ^D (c) and O _θ ^D (s), if c<s, then upsample O _θ ^D (s) to the same resolution as O _θ ^D (c) On the image, then make a difference between each pixel of O _θ ^D (c) and the corresponding pixel of the up-sampled O _θ ^D (s), if c>s, then up-sample O _θ ^D (c) to the same O _θ ^D (s) has the same resolution on the image, and then make a difference between each pixel of O _θ ^D (s) and the corresponding pixel of O _θ ^D (c) after upsampling, the symbol

Perform a cross-level addition operator for O _θ ^D (c) and O _θ ^D (s), if c < s, then upsample O _θ ^D (s) to the same resolution as O _θ ^D (c) On the image, each pixel of O _θ ^D (c) is summed with the corresponding pixel of the upsampled O _θ ^D (s), if c>s, then O _θ ^D (c) is upsampled to the same value as O _θ ^D (s) has the same resolution on the image, and then each pixel of O _θ ^D (s) is summed with the corresponding pixel of the upsampled O _θ ^D (c).

③-5. Using the known morphological dilation algorithm, take the block of size w ₁ ×h ₁ as the basic dilation unit to perform n ₁ dilation operations on the preliminary depth direction feature map F′ _DO of the current depth video frame to obtain the current depth video Depth direction feature map of a frame, denoted as F _DO . In this embodiment, for the "Ballet" and "Doorflower" test sequences, the size of each image in the test sequence is 1024×768, and the basic unit of morphological expansion adopts 8×8 blocks, that is, w ₁ ×h ₁ = 8×8, the number of expansions n ₁ =2.

记S′_D中坐标为(x，y)的像素的像素值为s′_d(x，y)，其中，

为归一化至

区间的归一化函数。③-6. Utilize the depth feature map F _D and the depth direction feature map F _DO of the current depth video frame to obtain the distribution map of the preliminary depth visual attention of the current depth video frame, denoted as S′ _D ,

Note that the pixel value of the pixel whose coordinates are (x, y) in S′ _D is s′ _d (x, y), where,

to normalize to

Normalization function for intervals.

③-7. For the left and right boundary areas of the image, the left image boundary of the left viewpoint image has no corresponding region in the right viewpoint, so it cannot form a stereoscopic effect in the human brain; similarly, for the right It is also difficult to form a three-dimensional effect at the image boundary. Therefore, in the stereoscopic video, the stereoscopic effect provided by the left and right boundary regions of the image is weak or even has no stereoscopic effect, and it is a non-stereoscopic vision attention region, so the present invention pays attention to the distribution map S′ _D of the boundary region of the preliminary depth vision of the current depth video frame To suppress, use the distribution map S′ _D of the preliminary depth visual attention of the current depth video frame to obtain the distribution map S _D of the depth visual attention of the 3D video image jointly presented by the current depth video frame and the current texture video frame, record in _SD The pixel value of the pixel whose coordinates are (x, y) is s _d (x, y), s _d (x, y) = s′ _d (x, y)·g(x, y), where, $g(x,\mathrm{the\; y})=\left\{\begin{array}{ccc}0.2& \mathrm{if}& x<b\left|\right|\mathrm{the\; y}<b\left|\right|x>W-b\left|\right|\mathrm{the\; y}>h-b\\ 0.4& \mathrm{else\; if}& x<2b\left|\right|\mathrm{the\; y}<2b\left|\right|x>W-2b\left|\right|\mathrm{the\; y}>h-2b\\ 0.6& \mathrm{else\; if}& x<3b\left|\right|\mathrm{the\; y}<3b\left|\right|x>W-3b\left|\right|\mathrm{the\; y}>h-3\mathrm{<figure-callout\; id="1"\; label="b"\; filenames="A2009101525200039H1.png"\; state="\{\{state\}\}">b</figure-callout>}\\ 1& & \mathrm{else}\end{array}\right.,$ W is the width of the current depth video frame, H is the height of the current depth video frame, b is the set second threshold, and the symbol "||" is an "or" operator. Here, the second threshold b takes a value of 16. The function g(x, y) can also be other two-dimensional functions for suppressing image edge regions, such as two-dimensional Gaussian functions whose template size is the same as the texture video frame size.

Figure 8c shows the depth visual attention distribution diagram of the 3D video image jointly presented by the color video frame and the corresponding depth video frame at time t in the test sequence "Ballet" 2D color video; Figure 13c shows the distribution diagram of the test sequence "DoorFlower "The distribution map of the depth visual attention of the 3D video image jointly presented by the color video frame and the corresponding depth video frame at time t in the 2D color video.

④Adopt the visual attention fusion method based on depth perception to combine the static image domain visual attention distribution map S _I of the current texture video frame, the motion visual attention distribution map S _M of the current texture video frame, the current depth video frame and the current depth video frame The depth visual attention distribution map S _D of the 3D video image jointly displayed with the current texture video frame is fused to extract the 3D visual attention distribution map that conforms to the stereoscopic perception of the human eye, denoted as S, the size of the 3D visual attention distribution map S The size is W×H and it is a grayscale image represented by Z _S bit depth. The larger the pixel value of a certain pixel in the grayscale image, the 3D video image jointly presented by the human eye to the current depth video frame and the current texture video frame The higher the relative attention degree of the corresponding pixel of , the smaller the pixel value is, the lower the relative attention degree of the human eye is to the 3D video image jointly presented by the current depth video frame and the current texture video frame.

In the traditional single channel, moving objects are more likely to attract the attention of the viewer than static objects. For all static objects, areas with bright colors, areas with large color or brightness contrast, and areas with large differences in texture directions etc. are more likely to attract the attention of the viewer; in stereoscopic video, the visual attention distribution of the human eye is not only affected by the movement visual attention and static image domain visual attention, but also by the unique stereoscopic effect provided by the stereoscopic video to the user; this The three-dimensional effect mainly comes from the slight positional deviation of the scene seen by our left and right eyes, which is called parallax. This small deviation is automatically synthesized by the brain into a stereoscopic image with depth, forming stereoscopic vision. The relative distance information of the object reflected in the stereoscopic sense is another important factor that directly affects our attention choice. In the stereoscopic video, the objects contained in the depth discontinuous area or the area with greater depth contrast can give the user a stronger depth difference, which has a stronger sense of three-dimensional or depth, and is one of the areas of interest to the user; On the other hand, the viewer is more interested in the foreground area close to the shooting camera (or video viewer) than the area far away from the shooting camera (or video viewer), so the foreground area is usually a stereoscopic video viewer. The important potential area of the region of interest, based on the above analysis, determines that the factors that affect the three-dimensional visual attention of the human eye include four factors: static image domain visual attention, motion visual attention, depth visual attention, and depth. Therefore, in this specific embodiment, based on the depth The specific process of the perceptual visual attention fusion method is as follows:

④-1. Carry out scale transformation to the current depth video frame by Q(d(x, y))=d(x, y)+γ, wherein, γ is the value in Coefficients within the range, d(x, y) represents the pixel value of the pixel whose coordinates are (x, y) in the current depth video frame, Q(d(x, y)) represents the coordinates in the current depth video frame after scale transformation The pixel value of the pixel at (x, y).

为归一化至

区间的归一化函数，s_D(x，y)、s_M(x，y)和s_I(x，y)分别表示S_D、S_M以及S_I中坐标为(x，y)的像素的像素值，Θ_ab(x，y)为视觉注意的相关值，Θ_ab(x，y)＝min(s_a(x，y)，s_b(x，y))，min()为取最小值函数，C_ab为相关系数，相关系数满足条件： $\underset{a,b\&Element;\{D,M,I\},a\&NotEqual;b}{\mathrm{\&Sigma;}}{C}_{\mathrm{ab}}=1,$ 0≤C_ab＜1，相关系数C_DM表示S_D与S_M的相关度，相关系数C_DI表示S_D与S_I的相关度，相关系数C_IM表示S_I与S_M的相关度，a，b ∈{D，M，I}且a≠b。④-2. Using the scale-transformed current depth video frame, the current depth video frame, and the depth visual attention distribution map _SD of the 3D video image jointly presented by the current depth video frame and the current texture video frame, and the motion of the current texture video frame The distribution map S _M of visual attention and the distribution map S _I of visual attention in the static image domain of the current texture video frame, the distribution map S of three-dimensional visual attention is obtained, and the coordinates in the distribution map S of three-dimensional visual attention are (x, y) The pixel value of the pixel is s(x, y), Among them, K _D , K _M and K _I are the weighting coefficients of S _D , S _M and S _I respectively, and the weighting coefficients meet the conditions: $\underset{a\&Element;\{\mathrm{D.},m,I\}}{\mathrm{\&Sigma;}}{K}_a=1,$ 0≤K _a ≤1,

to normalize to

The normalization function of the interval, s _D (x, y), s _M (x, y) and s _I (x, y) represent the pixel with coordinates (x, y) in S _D , S _M and S _I respectively , Θ _ab (x, y) is the relevant value of visual attention, Θ _ab (x, y)=min(s _a (x, y), s _b (x, y)), min() is the The minimum value function, C _ab is the correlation coefficient, and the correlation coefficient satisfies the conditions: $\underset{a,b\&Element;\{\mathrm{D.},m,I\},a\&NotEqual;b}{\mathrm{\&Sigma;}}{C}_{\mathrm{ab}}=1,$ 0≤C _ab <1, the correlation coefficient C _DM represents the correlation between _SD and S _M , the correlation coefficient C _DI represents the correlation between SD and S _I , and the correlation coefficient C _IM represents the correlation between S _{I and} S _M , a , b ∈ {D, M, I} and a≠b.

Motion visual attention, static image domain visual attention and depth visual attention all play important roles in conjunction with people's visual attention, however motion visual attention is the most important content in video visual attention, followed by brightness, color and orientation of the image domain Visual attention in the static image domain, again it is depth visual attention, so in this embodiment, each visual attention distribution map is represented by Z _S =8 bit depth, in this specific embodiment take K _D =0.15, K _M ＝0.4 and K _I ＝0.35, the correlation between depth visual attention and motion visual attention is small, the correlation between depth visual attention and static image domain visual attention is also small, and the correlation between static image domain visual attention and motion visual attention is relatively small is large, so the correlation coefficients C _DM , C _DI and C _IM are set to 0.2, 0.2 and 0.6 respectively, and the scale transformation coefficient γ represents the depth of the scene in the texture video scene. The smaller the γ, the greater the depth of the scene, giving the viewer more On the contrary, the larger the γ, the smaller the depth of the scene, and the weaker the depth sense given to the viewer. For the "Ballet" and "Door Flower" test sequences, since the scene has a smaller depth of field, the scale transformation coefficient γ is set as 50. The distribution diagram of the three-dimensional visual attention extracted from the texture video frame and the corresponding depth video frame at the time t of the "Ballet" test sequence is shown in Figure 9. The texture video frame and the corresponding depth video frame at the time t of the "Door Flower" test sequence The distribution map of 3D visual attention obtained by extracting the depth video frame is shown in Fig. 14.

⑤Thresholding and macroblocking post-processing are performed on the distribution map S of 3D visual attention to obtain the final region of interest that conforms to the stereoscopic perception of the human eye in the current texture video frame.

In this specific embodiment, the specific process of performing thresholding and macroblock post-processing on the distribution map S of 3D visual attention is as follows:

⑤-1. The pixel value of the pixel whose coordinates are (x, y) is s(x, y) in the distribution map S that remembers the three-dimensional visual attention, defines the third threshold T _S , $T_S=k_T\&Center\; Dot;\underset{\mathrm{the\; y}=0}{\overset{h-1}{\mathrm{\&Sigma;}}}\underset{x=0}{\overset{W-1}{\mathrm{\&Sigma;}}}\mathrm{the\; s}(x,\mathrm{the\; y})/(W\&times;h),$ Wherein, W is the width of the distribution map S of three-dimensional visual attention, H is the height of the distribution map S of three-dimensional visual attention, k _T ∈ (0, 3), and k _T can be taken as 1.5; create a new preliminary binary mask image, and judge whether s(x, y) ≥ T _S is established, and if so, mark the pixel with coordinates (x, y) in the preliminary binary mask image as the pixel of interest , otherwise, mark the pixel at (x, y) coordinate in the preliminary binary mask image as a non-interest pixel.

⑤-2. Divide the preliminary binary mask image into (W/w ₂ )×(H/h ₂ ) blocks with a size of w ₂ ×h ₂ , and the blocks do not overlap each other, mark horizontally The block with coordinate u and ordinate v is B _{u, v} , where u ∈ [0, W/w ₂ -1], v ∈ [0, H/h ₂ -1], according to the preliminary binary mask Each block in the image determines whether the pixel in the corresponding block in the current texture video frame is a pixel of interest or a non-interest pixel. For block B _{u, v} , determine the pixel marked as a pixel of interest in block B _{u, v} Whether the number of is greater than the set fourth threshold T _b , wherein, 0≤T _b ≤w ₂ ×h ₂ , if yes, all pixels in the block corresponding to the block B _{u, v} in the current texture video frame Mark as a pixel of interest, and use the block corresponding to the block B _{u, v} as the region of interest block, otherwise, mark all the pixels in the block corresponding to the block B _{u, v} in the current texture video frame as non-interest pixels, And the block corresponding to block B _{u, v} is regarded as the non-interest area block, obtains the initial interest area mask image of the current texture video frame, and the preliminary interest area mask image is composed of the interest area block and the non-interest area block composition.

In this embodiment, the size of each image in the test sequence "Ballet" and "Door Flower" is 1024×768, so the size w ₂ ×h ₂ of the block _{Bu, v} can be set to 16×16, usually pixels Regions with a small number are not likely to attract the interest of the viewer, so the fourth threshold T _b is set to 50 here.

⑤-3. Since the area of interest and the area of non-interest is usually not changed abruptly, but changes slowly, there is a transition zone, so the present invention sets _NR level between the area of interest and the area of non-interest transition region of interest. Mark all the pixels in the non-region of interest block nearest to the region of interest block in the preliminary region of interest mask image as the _NRth level transition region of interest, and update the preliminary region of interest mask image; then, set In the updated preliminary ROI mask image, all pixels in the non-ROI block closest to the NR- _th transition ROI are marked as _NR -1th transition ROIs, and the preliminary ROI is updated recursively Region mask image; repeat the recursive above-mentioned process again, until marking to the first-level transition region of interest; finally get the final region of interest mask image of the current texture video frame, the final region of interest mask image is determined by the region of interest Region block, _NR level transition region of interest and non-region of interest block. In this specific embodiment, the value of _NR is 2, that is, two levels of transition ROIs are set.

Figure 10a shows the final ROI mask image of the texture video frame at time t of the test sequence "Ballet"; Figure 15a shows the final ROI mask image of the texture video frame at time t of the test sequence "Door Flower" Region mask image. In Fig. 10a and Fig. 15a, the black region represents the region of interest, the gray region is the transition region of interest, and the white region is the non-interest region.

⑤-4. Record the pixel value of the pixel whose coordinates are (x, y) in the final region of interest mask image r(x, y), and place the non-interest region block in the final region of interest mask image Set the pixel values of all pixels in r(x, y)=255, and set the pixel values of all pixels in the N _R- level transition ROI in the final ROI mask image as $r(x,\mathrm{the\; y})=\frac{e}{N_R+1}\&times;f(x,\mathrm{the\; y}),$ Set the pixel values of all pixels in the ROI block in the final ROI mask image to r(x, y)=f(x, y), to obtain the ROI of the current texture video frame, where e Indicates the series of the transition region of interest, e∈[1, N _R ], f(x, y) indicates the pixel value of the pixel whose coordinates are (x, y) in the current texture video frame.

Fig. 10b shows the region of interest of the texture video frame at time t of the test sequence "Ballet"; Fig. 15b shows the region of interest of the texture video frame of the test sequence "Door Flower" at time t. The region of interest in Figure 10b and Figure 15b has the same pixel value as the texture video frame at time t, displaying colored texture content, the transition region of interest is a dark gray region displayed by reducing the brightness, and the smooth white region is a masked area with the region of interest The non-interest region corresponding to the white region of the model image. As a comparison of the extraction effects, Figure 11a and Figure 16a respectively show the regions of interest of the texture video frames of the test sequences "Ballet" and "Door Flower" at time t, which are traditionally extracted only based on visual attention cues in the static image domain, and the background cannot be removed. Noise area with rich texture; Figure 11b and Figure 16b show the area of interest of the texture video frame of the test sequence "Ballet" and "Door Flower" at time t, which is traditionally extracted only based on motion visual attention cues. For the "Ballet" sequence, The ROI extraction method based only on motion visual attention cues cannot fully extract the man who moves very slowly, and the background noise caused by moving shadows is serious; for the "Door Flower" sequence, only based on the ROI extraction method based on motion visual attention regions of motion without taking into account texture complexity and the sense of depth provided by stereo vision. Figure 11c and Figure 16c show the region of interest of the texture video frame at time t of the test sequences "Ballet" and "Door Flower" based on the combination of static image domain visual attention and motion visual attention cues, although the method combines static and motion vision information, however textured regions and motion noise in the background environment cannot be effectively suppressed.

It can be seen from the comparison experiment between Fig. 10a, Fig. 10b and Fig. 11a, Fig. 11b, Fig. 11c, Fig. 15a, Fig. 15b and Fig. Visual attention, motion visual attention and depth visual attention can effectively suppress the inherent singleness and inaccuracy of each visual attention extraction, solve the noise problem caused by complex background in visual attention in the static image domain, and solve the problem that motion visual attention cannot extract local The region of interest with small motion and motion range can improve the calculation accuracy, enhance the stability of the algorithm, and can extract the region of interest from the background with complex texture and the moving environment. In addition, the region of interest obtained through the present invention not only conforms to the visual characteristics of human eyes on static texture video frames and the visual characteristics of human eyes on moving objects, but also conforms to the strong sense of depth or short distance in stereo vision. The depth perception characteristics of the object of interest conform to the semantic characteristics of human stereo vision.

⑥Repeat steps ①～⑤ until all the texture video frames in the texture video are processed, and the video ROI of the texture video is obtained.

In this specific embodiment, the distribution map S _I of the static image domain visual attention of the current texture video frame, the distribution map S _M of the motion visual attention of the current texture video frame, the distribution map S _D of the depth visual attention of the three-dimensional video image, The distribution map S of three-dimensional visual attention is a grayscale image represented by Z _S bit depth, and the depth video frames at each moment in the depth video corresponding to the texture video are grayscale images represented by Z _D bit depth, and here the grayscale images are all 256 colors are used, represented by 8-bit depth, therefore, Z _S =8, Z _D =8, of course, other bit depths can also be used to represent grayscale images in practical applications, such as 16-bit depth, if 16-bit If the depth represents the grayscale image, the accuracy will be higher.

Claims (8)

1. A method for extracting a video interesting region based on visual attention is characterized by comprising the following steps:

firstly, defining a two-dimensional color video as a texture video, defining the size of texture video frames at each moment in the texture video to be W multiplied by H, W being the width of the texture video frames at each moment in the texture video, H being the height of the texture video frames at each moment in the texture video, and recording the texture video frame at t moment in the texture video as F_tDefining a texture video frame F at time t in the texture video_tFor the current texture video frame, the well-known static state is adoptedThe image visual attention detection method detects the visual attention of the static image domain of the current texture video frame to obtain a distribution diagram of the visual attention of the static image domain of the current texture video frame, which is marked as S_IThe static image domain visual attention distribution map S of the current texture video frame_IHas a dimension of W × H and is Z_SA grayscale map of bit depth representations;

secondly, detecting the motion visual attention of the current texture video frame by adopting a motion visual attention detection method to obtain a distribution diagram of the motion visual attention of the current texture video frame, and recording the distribution diagram as S_MDistribution map S of motion visual attention of current texture video frame_MHas a dimension of W × H and is Z_SA grayscale map of bit depth representations;

defining depth video frame of each time in depth video corresponding to texture video as Z_DSetting the size of a depth video frame at each moment in a depth video to be W multiplied by H, wherein W is the width of the depth video frame at each moment in the depth video, H is the height of the depth video frame at each moment in the depth video, and D is the depth video frame at t moment in the depth video_tDefining a depth video frame D at time t in the depth video_tFor the current depth video frame, detecting the depth visual attention of the three-dimensional video image jointly displayed by the current depth video frame and the current texture video frame by adopting a depth visual attention detection method to obtain a depth visual attention distribution diagram of the three-dimensional video image, which is marked as S_DDistribution diagram S of depth visual attention of three-dimensional video image_DHas a dimension of W × H and is Z_SA grayscale map of bit depth representations;

fourthly, adopting a visual attention fusion method based on depth perception to visually pay attention to the static image domain of the current texture video frame_IDistribution diagram S of motion visual attention of current texture video frame_MAnd a current depth video frame and a depth visual attention distribution diagram S of a three-dimensional video image jointly displayed by the current depth video frame and the current texture video frame_DFusing to extract a distribution diagram of three-dimensional visual attention corresponding to human eye stereoscopic perception, which is recorded as SThe size of S is W × H and it is Z_SA grayscale map of bit depth representations;

performing thresholding and macro block post-processing on the distribution graph S of the three-dimensional visual attention to obtain a final region of interest which accords with human eye three-dimensional perception of the current texture video frame;

sixthly, repeating the steps from the first step to the fifth step until all texture video frames in the texture video are processed, and obtaining the video interesting area of the texture video.

2. The method for extracting a video interesting region based on visual attention according to claim 1, wherein the specific process of the moving visual attention detection method in the step (II) is as follows:

(II-1) recording the texture video frame at the time t + j continuous with the current texture video frame in the texture video as F_t+jRecording the texture video frame at the t-j time point which is continuous with the current texture video frame in the texture video as F_t-jWherein j ∈ (0, N)_F/2]，N_FIs a positive integer less than 10;

2, calculating the current texture video frame and the texture video frame F at the moment t + j by adopting a known optical flow method_t+jMotion vector image in horizontal direction and motion vector image in vertical direction, and texture video frame F of current texture video frame and time t-j_t-jRecording the current texture video frame and the texture video frame F at the time of t + j in the motion vector image in the horizontal direction and the motion vector image in the vertical direction_t+jThe motion vector image in the horizontal direction is V_t+j ^HAnd the motion vector image in the vertical direction is V_t+j ^VRecording the current texture video frame and the texture video frame F at the moment t-j_t-jThe motion vector image in the horizontal direction is V_t-j ^HAnd the motion vector image in the vertical direction is V_t-j ^V，V_t+j ^H、V_t+j ^V、V_t+j ^HAnd V_t-j ^HW for width and H for height;

② -3, mixing V_t+j ^HAbsolute value of and V_t+j ^VThe absolute value of the current texture video frame is superposed to obtain the texture video frame F of the current texture video frame and the texture video frame F at the moment of t + j_t+jMotion amplitude image of, noted as M_t+j， $M_{t+j}=\left|V_{t+j}^H\right|+\left|V_{t+j}^V\right|,$ Memory M_t+jThe motion amplitude value of the pixel with the middle coordinate (x, y) is m_t+j(x, y); will V_t-j ^HAbsolute value of and V_t-j ^VThe absolute value of the texture video frame is superposed to obtain the texture video frame F of the current texture video frame and the t-j moment_t-jMotion amplitude image of, noted as M_t-j， $M_{t-j}=\left|V_{t-j}^H\right|+\left|V_{t-j}^V\right|,$ Memory M_t-jThe motion amplitude value of the pixel with the middle coordinate (x, y) is m_t-j(x，y)；

② 4, utilizing current texture video frame and texture video frame F at t + j moment_t+jAnd texture video frame F at time t-j_t-jExtracting a joint motion map, denoted as M_j ^ΔExtracting the joint movement map M_j ^ΔThe specific process comprises the following steps: judging the current texture video frame and the texture video frame F at the moment t + j_t+jMotion amplitude image M_t+jEach pixel in (a) and the current texture video frame and the texture video frame F at time t-j_t-jMotion amplitude image M_t-jWhether the minimum value in the motion amplitude values of the pixels of the corresponding coordinates is larger than a set first threshold value T or not₁And if so, determining a joint motion map M_j ^ΔThe pixel value of the pixel of the corresponding coordinate is M_t+jAnd M_t-jAverage of the sum of motion amplitude values of pixels of the corresponding coordinates, otherwise, determining the joint motion map M_j ^ΔThe pixel value of the pixel of the corresponding coordinate is 0; for M_t+jPixel with (x, y) middle coordinate and M_t-jThe pixel with the middle coordinate of (x, y) is judged to be min (m)_t+j(x，y)，m_t-j(x, y)) is greater than a set first threshold value T₁And if so, determining a joint motion map M_j ^ΔThe pixel value of the pixel with the middle coordinate (x, y) is

Otherwise, determining the joint motion map M_j ^ΔThe pixel value of the pixel with the middle coordinate (x, y) is 0, wherein min () is a minimum function;

② -5, the distance from the t time to the 1 time to the N time_FWeighted superposition of the combined motion images at each moment of the/2 moment is carried out to obtain a weighted combined motion image of the current texture video frame, the weighted combined motion image is marked as M, the pixel value of a pixel with coordinates (x, y) in the weighted combined motion image M of the current texture video frame is marked as M (x, y),

wherein m is_j ^Δ(x, y) represents the joint motion map M at a time j away from time t_j ^ΔPixel value, ζ, of a pixel having a middle coordinate of (x, y)_jAs a weighting coefficient, a weighting coefficient ζ_iSatisfy the requirement of

② 6, carrying out Gaussian pyramid decomposition on the weighted joint motion picture M of the current texture video frame to decompose into n_LAnd (3) a layer weighted joint motion picture, wherein the ith layer weighted joint motion picture obtained after M Gaussian pyramid decomposition of the weighted joint motion picture of the current texture video frame is recorded as M (i), and the width and the height of the ith layer weighted joint motion picture M (i) are respectively W/2ⁱAnd H/2ⁱWherein n is_LIs a positive integer less than 20, i ∈ [0, n ∈ [ ]_L-1]W is the width of the current texture video frame, and H is the height of the current texture video frame;

② -7, utilizing the weighted joint motion map of the current texture video frame to n_LLayer weighted joint motion map, extracting the motion visual attention distribution map S of the current texture video frame_MRemember S_MThe pixel value of the pixel with the middle coordinate (x, y) is s_m(x，y)，S_M＝F_MWherein

s，c∈[0，n_L-1]，s＝c+δ，δ＝{-3，-2，-1，1，2，3}，

is normalized to

The normalized function of interval, the symbol "|" is absolute value operation symbol, M (c) is the c-th layer weighted joint motion diagram, M(s) is the s-th layer weighted joint motion diagram, symbol

Performing cross-level difference operation on M (c) and M(s), if c < s, up-sampling M(s) to the image with the same resolution as that of M (c), and then matching each pixel of M (c) with the up-sampled corresponding pixel of M(s)Respectively performing difference on the pixels, if c is more than s, up-sampling M (c) to the image with the same resolution as that of M(s), and then respectively performing difference on each pixel of M(s) and the corresponding pixel of the up-sampled M (c), and symbolizingPerforming a cross-level addition operator for M (c) and M(s), if c < s, upsampling M(s) to an image with the same resolution as M (c), then summing each pixel of M (c) with the corresponding pixel of upsampled M(s), respectively, if c > s, upsampling M (c) to an image with the same resolution as M(s), and then summing each pixel of M(s) with the corresponding pixel of upsampled M (c), respectively.

3. The method as claimed in claim 2, wherein the first threshold T set in step (II) -4 is set₁＝1。

4. The method for extracting video interesting regions based on visual attention according to claim 1 or 2, wherein the depth visual attention detection method in the third step comprises the following specific processes:

thirdly-1, carrying out Gaussian pyramid decomposition on the current depth video frame to obtain n_LThe layer depth video frame is recorded as a layer i depth video frame obtained after the Gaussian pyramid decomposition of the current depth video frame is D (i), and the width and the height of the layer i depth video frame D (i) are respectively W/2ⁱAnd H/2ⁱWherein n is_LIs a positive integer less than 20, i ∈ [0, n ∈ [ ]_L-1]W is the width of the current depth video frame, and H is the height of the current depth video frame;

③ 2, utilizing n of current depth video frame_LLayer depth video frame, extracting depth characteristic map of current depth video frame, and recording asWherein,

s，c∈[0，n_L-1]，s＝c+δ，δ＝{-3，-2，-1，1，2，3}，

is normalized to

The normalized function of the interval, the symbol "|" is the absolute value operation symbol, D (c) is the c-th layer depth video frame, D(s) is the s-th layer depth video frame, the symbol

Performing cross-level difference operation on D (c) and D(s), if c < s, upsampling D(s) to an image with the same resolution as D (c), then respectively performing difference on each pixel of D (c) and the corresponding pixel of the upsampled D(s), if c > s, upsampling D (c) to an image with the same resolution as D(s), then respectively performing difference on each pixel of D(s) and the corresponding pixel of the upsampled D (c), and symbolizing

Performing a cross-layer addition operator for D (c) and D(s), if c < s, upsampling D(s) to an image with the same resolution as D (c), then summing each pixel of D (c) with the corresponding pixel of upsampled D(s), respectively, if c > s, upsampling D (c) to an image with the same resolution as D(s), and then summing each pixel of D(s) with the corresponding pixel of upsampled D (c), respectively;

thirdly, carrying out convolution operation on the current depth video frame by adopting known Gabor filters with 0 degree, pi/4 degree, pi/2 degree and 3 pi/4 degree directions to extract four direction components of the 0 degree, pi/4 degree, pi/2 degree and 3 pi/4 degree directions to obtain four direction component graphs of the current depth video frame, wherein the four direction component graphs are respectively represented as O₀ ^D、O_π/4 ^D、O_π/2 ^DAnd O_3π/4 ^D(ii) a O for current depth video frame₀ ^DDirectional component diagram, O_π/4 ^DDirectional component diagram, O_π/2 ^DDirectional component diagram and O_3π/4 ^DThe direction component diagram is respectively decomposed into n by Gaussian pyramid_LThe layer direction component diagram is marked, and the ith layer direction component diagram obtained by decomposing the direction component diagram in the theta degree direction by a Gaussian pyramid is O_θ ^D(i)，O_θ ^D(i) Is W/2 in width and height respectivelyⁱAnd H/2ⁱWhere θ ∈ {0, π/4, π/2, 3 π/4} i ∈ [0, n ∈_L-1]W is the width of the current depth video frame, and H is the height of the current depth video frame;

thirdly-4, utilizing n of the direction component diagram of each degree direction of the current depth video frame_LExtracting a primary depth direction feature map of the current depth video frame, and recording the primary depth direction feature map as F'_DO， <math> <mrow> <msubsup> <mover> <mi>F</mi> <mo>&OverBar;</mo> </mover> <mi>DO</mi> <mo>&prime;</mo> </msubsup> <mo>=</mo> <mfrac> <mn>1</mn> <mn>4</mn> </mfrac> <munder> <mi>&Sigma;</mi> <mrow> <mi>&theta;</mi> <mo>&Element;</mo> <mo>{</mo> <mn>0</mn> <mo>,</mo> <mfrac> <mi>&pi;</mi> <mn>4</mn> </mfrac> <mo>,</mo> <mfrac> <mi>&pi;</mi> <mn>2</mn> </mfrac> <mo>,</mo> <mfrac> <mrow> <mn>3</mn> <mi>&pi;</mi> </mrow> <mn>4</mn> </mfrac> <mo>}</mo> </mrow> </munder> <msub> <mover> <mi>F</mi> <mo>&OverBar;</mo> </mover> <msub> <mi>O</mi> <mi>&theta;</mi> </msub> </msub> <mo>,</mo> </mrow> </math> Wherein,

s，c∈[0，n_L-1]，s＝c+δ，δ＝{-3，-2，-1，1，2，3}，is normalized toNormalized function of interval, symbol "|" is absolute value operation symbol, O_θ ^D(c) A c-th layer direction component diagram which is a direction component diagram of a theta degree direction, O_θ ^D(s) an s-th layer direction component diagram of a theta degree direction, symbol

Is O_θ ^D(c) And O_θ ^D(s) perform a cross-level differencing operator, if c < s, then O_θ ^D(s) upsampling to and O_θ ^D(c) On the image with the same resolution, and then O_θ ^D(c) And the up-sampled O_θ ^D(s) performing difference on corresponding pixels respectively, and if c is more than s, performing O_θ ^D(c) Up-sampling to and O_θ ^D(s) on an image with the same resolution, and then adding O_θ ^D(s) each pixel with up-sampled O_θ ^D(c) Performing difference and sign on corresponding pixels

Is O_θ ^D(c) And O_θ ^D(s) performing a cross-level addition operator, if c < s, adding O_θ ^D(s) upsampling to and O_θ ^D(c) Images with the same resolutionThen O is introduced_θ ^D(c) And the up-sampled O_θ ^D(s) the corresponding pixels are summed separately, and if c > s, O is added_θ ^D(c) Up-sampling to and O_θ ^D(s) on an image with the same resolution, and then adding O_θ ^D(s) each pixel with up-sampled O_θ ^D(c) Summing corresponding pixels respectively;

③ 5, adopting the known morphological dilation algorithm to obtain the size w₁×h₁Is a preliminary depth direction feature map F 'of the basic expansion unit to the current depth video frame'_DOCarry out n₁Performing secondary expansion operation to obtain a depth direction characteristic diagram of the current depth video frame, and recording the characteristic diagram as F_DO；

Thirdly-6, utilizing the depth characteristic image F of the current depth video frame_DAnd depth direction feature map F_DOObtaining a distribution map, noted as S ', of the preliminary depth visual attention of the current depth video frame'_D，

Note S'_DThe pixel value of the pixel with the middle coordinate of (x, y) is s'_d(x, y) wherein,is normalized to

A normalization function of the interval;

-7, profile S 'using preliminary depth visual attention of current depth video frame'_DObtaining a depth visual attention distribution diagram S of the three-dimensional video image jointly displayed by the current depth video frame and the current texture video frame_DRemember S_DThe pixel value of the pixel with the middle coordinate (x, y) is s_d(x，y)，s_d(x，y)＝s′_d(x, y) · g (x, y), wherein, $g(x,y)=\left\{\begin{array}{ccc}0.2& \mathrm{if}& x<b\left|\right|y<b\left|\right|x>W-b\left|\right|y>H-b\\ 0.4& \mathrm{elseif}& x<2b\left|\right|y<2b\left|\right|x>W-2b\left|\right|y>H-2b\\ 0.6& \mathrm{elseif}& x<3b\left|\right|y<3b\left|\right|x>W-3b\left|\right|y>H-3b\\ 1& & \mathrm{else}\end{array}\right.,$ w is the width of the current depth video frame, H is the height of the current depth video frame, b is a set second threshold, and the symbol "|" is an "or" operator.

5. The method according to claim 4, wherein w in-5 is w₁＝8，h₁＝8，n₁The second threshold b set in the step (c) -7 is 16.

6. The method for extracting a video interesting region based on visual attention according to claim 1, wherein the visual attention fusion method based on depth perception in the step (iv) comprises the following specific processes:

and 1, carrying out scale transformation on the current depth video frame by Q (d (x, y)) ═ d (x, y) + gamma, wherein the gamma is a value in

Coefficients within a range, d (x, y) represents a pixel value of a pixel with coordinates (x, y) in the current depth video frame, and Q (d (x, y)) represents a pixel value of a pixel with coordinates (x, y) in the current depth video frame after the scaling;

fourthly-2, performing combined display by utilizing the current depth video frame after the scale transformation, the current depth video frame and the current texture video frameDistribution diagram S of depth visual attention of three-dimensional video image_DDistribution diagram S of motion visual attention of current texture video frame_MAnd a histogram S of the static image domain visual attention of the current texture video frame_IAcquiring a three-dimensional visual attention distribution graph S, wherein the pixel value of a pixel with coordinates (x, y) in the three-dimensional visual attention distribution graph S is S (x, y),

wherein, K_D、K_MAnd K_IRespectively S_D、S_MAnd S_IThe weighting coefficient satisfies the condition: <math> <mrow> <munder> <mi>&Sigma;</mi> <mrow> <mi>a</mi> <mo>&Element;</mo> <mo>{</mo> <mi>D</mi> <mo>,</mo> <mi>M</mi> <mo>,</mo> <mi>I</mi> <mo>}</mo> </mrow> </munder> <msub> <mi>K</mi> <mi>a</mi> </msub> <mo>=</mo> <mn>1</mn> <mo>,</mo> </mrow> </math> 0≤K_a≤1，

is normalized to

Normalized function of interval, s_D(x，y)、s_M(x, y) and s_I(x, y) each represents S_D、S_MAnd S_IThe pixel value, Θ, of a pixel having the middle coordinate (x, y)_ab(x, y) is the correlation value of visual attention, [ theta ]_ab(x，y)＝min(s_a(x，y)，s_b(x, y)), min () is the minimum function, C_abAs the correlation coefficient, the correlation coefficient satisfies the condition: <math> <mrow> <munder> <mi>&Sigma;</mi> <mrow> <mi>a</mi> <mo>,</mo> <mi>b</mi> <mo>&Element;</mo> <mo>{</mo> <mi>D</mi> <mo>,</mo> <mi>M</mi> <mo>,</mo> <mi>I</mi> <mo>}</mo> <mo>,</mo> <mi>a</mi> <mo>&NotEqual;</mo> <mi>b</mi> </mrow> </munder> <msub> <mi>C</mi> <mi>ab</mi> </msub> <mo>=</mo> <mn>1</mn> <mo>,</mo> </mrow> </math> 0≤C_ab< 1, correlation coefficient C_DMDenotes S_DAnd S_MDegree of correlation, coefficient of correlation C_DIDenotes S_DAnd S_IDegree of correlation, coefficient of correlation C_IMDenotes S_IAnd S_MA, b ∈ { D, M, I } and a ≠ b.

7. The method for extracting a video region of interest based on visual attention as claimed in claim 6, wherein the specific process of thresholding and macro-block post-processing the distribution map S of three-dimensional visual attention in the fifth step is as follows:

-1, recording the pixel value of the pixel with coordinate (x, y) in the distribution graph S of the three-dimensional visual attention as S (x, y), defining a third threshold T_S， <math> <mrow> <msub> <mi>T</mi> <mi>S</mi> </msub> <mo>=</mo> <msub> <mi>k</mi> <mi>T</mi> </msub> <mo>&CenterDot;</mo> <munderover> <mi>&Sigma;</mi> <mrow> <mi>y</mi> <mo>=</mo> <mn>0</mn> </mrow> <mrow> <mi>H</mi> <mo>-</mo> <mn>1</mn> </mrow> </munderover> <munderover> <mi>&Sigma;</mi> <mrow> <mi>x</mi> <mo>=</mo> <mn>0</mn> </mrow> <mrow> <mi>W</mi> <mo>-</mo> <mn>1</mn> </mrow> </munderover> <mi>s</mi> <mrow> <mo>(</mo> <mi>x</mi> <mo>,</mo> <mi>y</mi> <mo>)</mo> </mrow> <mo>/</mo> <mrow> <mo>(</mo> <mi>W</mi> <mo>&times;</mo> <mi>H</mi> <mo>)</mo> </mrow> <mo>,</mo> </mrow> </math> Wherein W is the width of the three-dimensional visual attention profile S, H is the height of the three-dimensional visual attention profile S, and k_TE (0, 3); newly building a preliminary binary mask image, and judging that s (x, y) is more than or equal to T_SIf so, marking the pixel with the coordinate (x, y) in the preliminary binary mask image as the pixel of interest, otherwise, marking the pixel with the coordinate (x, y) in the preliminary binary mask image as the pixel of non-interest;

fifthly, dividing the preliminary binary mask image into (W/W)₂)×(H/h₂) Size of w₂×h₂The blocks are not overlapped, and the block with the abscissa of u and the block with the ordinate of v is marked as B_u，vWherein u ∈ [0, W/W ]₂-1]，v∈[0，H/h₂-1]Determining whether the pixel in each corresponding block in the current texture video frame is the pixel of interest or the non-interest pixel according to each block in the preliminary binary mask image, and regarding the block B_u，vJudgment Block B_u，vWhether the number of pixels marked as the pixel of interest is larger than a set fourth threshold value T or not_bWherein, 0 is less than or equal to T_b≤w₂×h₂If yes, the current texture video frame is compared with the block B_u，vAll pixels in the corresponding block are marked as pixels of interest, and block B is marked as the pixel of interest_u，vTaking the corresponding block as the region of interest block, otherwise, comparing the current texture video frame with the block B_u，vAll pixels in the corresponding block are marked as non-interesting pixels and block B is marked_u，vThe corresponding block is used as a non-interested area block to obtain a preliminary interested area mask image of the current texture video frame, wherein the preliminary interested area mask image consists of an interested area block and a non-interested area block;

fifthly, 3, a non-interested region block B which is most adjacent to the interested region block in the preliminary interested region mask image_u，vAll pixels in (1) are labeled as Nth_RLevel transition interested areas and updating the mask images of the primary interested areas; then, the updated preliminary region of interest mask imageNeutral and N_RNon-interested region block B nearest to stage transition interested region_u，vAll pixels in (1) are labeled as Nth_R-level 1 transition region of interest, recursively updating the preliminary region of interest mask image; repeating the recursion process until the region of interest is marked to the level 1 transition region of interest; finally, obtaining a final interested region mask image of the current texture video frame, wherein the final interested region mask image consists of an interested region block and N_RA level transition region of interest and a region of non-interest block;

fifthly, recording the pixel value of a pixel with the coordinate of (x, y) in the final region-of-interest mask image as r (x, y), setting the pixel values of all pixels in a region block which is not the region of interest in the final region-of-interest mask image as r (x, y) 255, and setting N in the final region-of-interest mask image as N_RSetting the pixel values of all pixels in the region of interest of the level transition to <math> <mrow> <mi>r</mi> <mrow> <mo>(</mo> <mi>x</mi> <mo>,</mo> <mi>y</mi> <mo>)</mo> </mrow> <mo>=</mo> <mfrac> <mi>e</mi> <mrow> <msub> <mi>N</mi> <mi>R</mi> </msub> <mo>+</mo> <mn>1</mn> </mrow> </mfrac> <mo>&times;</mo> <mi>f</mi> <mrow> <mo>(</mo> <mi>x</mi> <mo>,</mo> <mi>y</mi> <mo>)</mo> </mrow> <mo>,</mo> </mrow> </math> Setting the pixel values of all pixels in the region of interest block in the final region of interest mask image as r (x, y) ═ f (x, y), obtaining the region of interest of the current texture video frame, wherein e represents the progression of the transition region of interest, and e belongs to [1, N ]_R]And f (x, y) represents the pixel value of the pixel with coordinates (x, y) in the current texture video frame.

8. The method according to claim 7, wherein w in the step (c) -2 is w₂＝16，h₂＝16，Set fourth threshold value T_b＝50。

Images