CN103051888B - Image processing method and image acquisition device for generating dynamic images - Google Patents

Abstract

An image processing method for generating dynamic images and an image acquisition device thereof are provided. The image processing method comprises the following steps. First, a plurality of images are continuously acquired, one of the images is a reference image, and the rest of the images are to-be-processed images. Then, global motion estimation and global motion correction are performed on the image to be processed according to the reference image, so as to generate a plurality of corrected images. And comparing pixel points in each corrected image respectively, marking a local action area with pixel difference value variation, and defining a mask covering all the local action areas. And finally, playing the reference image by a display module of the image acquisition device, and updating the playing content of the reference image in sequence according to the position and the size of the mask of each corrected image.

Description

Image processing method and image acquisition device for generating dynamic images

technical field

The invention relates to an image processing method, in particular to an image processing method for playing dynamic images on an image acquisition device.

Background technique

Graphics Interchange Format (GIF) is a bitmap graphics file format that can only express color images in 8 bits (that is, 256 colors). The GIF format image file supports repeated playback of multiple images, which is suitable for integrating different photos or pictures into the same image file for playback, so as to achieve the playback effect of dynamic images.

When making a GIF format image file, a full-color image stored in 24 bits must first be converted into a color image stored in 8 bits (that is, 256 colors) before it can be saved as a GIF format image file. When converting a 24-bit full-color image to an 8-bit color image, different conversion modes may be used, resulting in discontinuous color gamut of each image after conversion, which may cause the playback effect of the dynamic image to be unsatisfactory. Therefore, If the playback of dynamic images is to achieve certain special effects, it must rely on professional image processing skills.

In addition, most of the GIF format image files are viewed on computers at present, and general cameras do not support GIF format image files. Complete images to continuously update the screen, and then achieve the effect of dynamic image playback. However, for the special playback effect that emphasizes that most of the objects in the image are still and only a few objects change, it is difficult to express the above-mentioned emphasis on partial transient image changes by updating the picture continuously with one complete image special effects.

Contents of the invention

In view of this, the present invention provides an image processing method for generating dynamic images and its image acquisition device, which can locally update the image content on the image acquisition device with a display module, so as to produce a special playback effect that emphasizes local transient image changes .

The invention proposes an image processing method for generating dynamic images, which is suitable for an image acquisition device with a display module. The image processing method includes the following steps. First, a plurality of images are continuously acquired, one of which is a reference image, and the rest are images to be processed. Then, global motion estimation and global motion correction are performed on the image to be processed according to the reference image, so as to generate a plurality of corrected images. In addition, pixel comparisons are performed in each corrected image, local motion areas with pixel difference changes are marked, and a mask covering all local motion areas is defined. Finally, the reference image is played by the display module of the image acquisition device, and the playback content of the reference image is sequentially updated according to the position and size of the mask of each corrected image.

In an embodiment of the present invention, the above-mentioned steps of performing pixel point comparison in each of the corrected images, marking the local motion areas with pixel difference changes, and defining a mask covering the local motion areas also include: In the time sequence of acquisition, the pixel value difference of each pixel is compared between each corrected image and its corresponding previous image, so as to generate an image difference value of each pixel. And judge whether the image difference is greater than a threshold value, if so, mark this pixel as a local action area. Then, a rectangular area covering all marked local action areas in each corrected image is circled as a mask of each corrected image.

In an embodiment of the present invention, after the above-mentioned step of circling all the rectangular areas marked as local action areas in each corrected image as a mask for each corrected image, it also includes All the local action areas within the range of the rectangular area in , and the corresponding areas in the reference image, are color-corrected by filter processing. And store the reference image and the image content of the rectangular area of each corrected image after color correction, so as to provide to the display module for playing.

In an embodiment of the present invention, the image acquisition device is used to obtain a grayscale image under a shooting condition, the average brightness value and standard deviation of the grayscale image are counted, and the grayscale image is set according to the average brightness value and standard deviation. Threshold value under shooting conditions.

In an embodiment of the present invention, after the step of generating the corrected image, the step of generating the corrected image further includes using a dynamic image compression method to estimate the area movement of each corrected image, so as to generate a plurality of area motion vectors of each corrected image . And compare the pixel value of each pixel point in the area where the area motion vector exists in each corrected image, so as to mark the local motion area with pixel difference change.

In an embodiment of the present invention, the above-mentioned dynamic image compression method includes H.264 compression method, Moving Picture Experts Group 2/4 (Moving Picture Experts Group 2/4, MPEG-2/4) compression method or dynamic joint image experts Group (Motion Joint Photographic Coding Expert Group, MJPEG) compression method.

The present invention further proposes an image acquisition device for generating dynamic images, which includes an image acquisition module, a movement estimation and correction module, an image comparison module, and a display module. Wherein, the image acquisition module is used to continuously acquire multiple images, one of which is a reference image, and the rest of the images are multiple images to be processed. The movement estimation and correction module is coupled to the image acquisition module, and performs global movement estimation and global movement correction on the image to be processed according to the reference image, so as to generate a plurality of corrected images. The image comparison module is coupled to the image acquisition module and the movement amount estimation and correction module, respectively compares each corrected image, marks the local action area with pixel difference change, and defines the area covering all local action areas A mask. The display module is coupled to the image comparison module for playing the reference image, and sequentially updates the playback content of the reference image according to the position and size of the mask of each corrected image.

In an embodiment of the present invention, the above-mentioned image comparison module compares the pixel value difference between each pixel point of each corrected image and each pixel point of the corresponding previous image according to the time sequence of image acquisition. , so as to generate the image difference value of each pixel. In addition, the image comparison module judges whether the image difference is greater than the threshold value, if so, the image comparison module marks this pixel as a local action area, and circles all the corrected images covering all marked as local action areas The rectangular area of , as a mask for each rectified image.

In an embodiment of the present invention, the image acquisition device for generating dynamic images further includes a filter processing module and a storage module. Wherein, the filter processing module is coupled to the image comparison module, and performs filter processing on all local motion regions within the range of the rectangular region in each corrected image and corresponding regions in the reference image. The storage module is coupled to the filter processing module, and is used for storing the reference image and image content of each rectified image after filter processing.

In an embodiment of the present invention, the above-mentioned image comparison module receives a grayscale image acquired by the image acquisition device under a shooting condition. The image comparison module counts the average brightness value and standard deviation of the gray scale image, and sets the threshold value under this shooting condition according to the average brightness value and standard deviation.

In an embodiment of the present invention, the image acquisition device for generating dynamic images further includes a compression module coupled to the image comparison module. The compression module performs regional motion estimation on each corrected image, so as to generate a plurality of regional motion vectors for each corrected image. The image comparison module compares the pixel value of each pixel in the area where the area motion vector exists, so as to mark the local motion area with pixel difference change.

In an embodiment of the present invention, the compression module is one of the H.264 compression module, animation expert group 2/4 compression module, and dynamic joint image expert group compression module.

Based on the above, the image processing method for generating dynamic images and its image acquisition device provided by the present invention can first display the first complete reference image on the image acquisition device with a display module, and then according to the previously recorded The content and position of the local image area can continuously and quickly update the area content locally, so as to achieve a dynamic image playback effect that emphasizes local short-term image changes.

In order to make the above-mentioned features and advantages of the present invention more comprehensible, the following specific embodiments are described in detail with reference to the accompanying drawings.

Description of drawings

FIG. 1 is a block diagram of an image acquisition device for generating dynamic images according to an embodiment of the present invention.

FIG. 2 is a flowchart of an image processing method for generating dynamic images according to an embodiment of the present invention.

FIG. 3 is a schematic diagram of a reference image and an image to be processed displayed according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of a rectangular area of a reference image and a corrected image displayed according to an embodiment of the present invention.

FIG. 5 is a block diagram of an image acquisition device for generating dynamic images according to another embodiment of the present invention.

FIG. 6 is a flow chart of an image processing method for generating dynamic images according to another embodiment of the present invention.

FIG. 7 is a block diagram of an image acquisition device for generating dynamic images according to yet another embodiment of the present invention.

Reference signs:

100, 500, 700: image acquisition device

110: Image acquisition module

120:Movement estimation and correction module

130: Image comparison module

140: display module

302: Reference Image

304, 306, 308, 310: images to be processed

304’, 306’, 308’, 310’: Corrected image

550: Filter processing module

560: Storage Module

770: Compression Mod

A～D, a～d: Rectangular area

S210-S240: each step of the image processing method for generating dynamic images

S610-S660: each step of the image processing method for generating dynamic images

Detailed ways

FIG. 1 is a block diagram of an image acquisition device for generating dynamic images according to an embodiment of the present invention. The image acquisition device for generating dynamic images of the present invention is, for example, a general consumer camera, a digital camera, a single-eye camera, or a mobile phone with an image acquisition function, a smart phone, and the like. Referring to FIG. 1 , the image acquisition device 100 for generating dynamic images includes an image acquisition module 110 , a movement estimation and correction module 120 , an image comparison module 130 and a display module 140 . Its functions are described as follows:

The image acquisition module 110 includes a lens, a photosensitive element, an aperture, and the like. The lens is, for example, a standard lens, a wide-angle lens, a zoom lens, and the like. The photosensitive element is, for example, a Charge Coupled Device (CCD), a Complementary Metal-Oxide Semiconductor (CMOS) element or other elements, and the lens, photosensitive element or combination thereof are not limited herein. The image capturing module 110 is used for continuously capturing multiple images.

The motion estimation and correction module 120 is coupled to the image acquisition module 110. For example, a G-sensor or a gyroscope can be used to perform global motion estimation (global motion estimation), and then according to the global motion The estimated result is subjected to global motion correction to remove the overall variation between images caused by the shaking of the image capturing module 110 during shooting.

The image comparison module 130 is coupled to the image acquisition module 110 and the movement estimation and correction module 120, and compares the pixel values of the acquired images respectively, marks and issues local motion areas with pixel difference changes, and defines coverage A mask for all local motion regions.

The display module 140 is coupled to the image comparison module 130 , such as a liquid crystal display (Liquid Crystal Display, LCD), a light-emitting diode (Light-Emitting Diode, LED) display, etc., and can be used to play images continuously.

FIG. 2 is a flowchart of an image processing method for generating dynamic images according to an embodiment of the present invention. Please refer to FIG. 2 , the method of this embodiment is applicable to the device in FIG. 1 , and the detailed steps of the image processing method for generating dynamic images in this embodiment will be described below in conjunction with each module in FIG. 1 .

First, as shown in step S210 , the image acquisition module 110 is used to continuously acquire a plurality of images, one of which is a reference image, and the rest are images to be processed. For example, FIG. 3 is a schematic diagram of a reference image and an image to be processed displayed according to an embodiment of the present invention. Please refer to FIG. 3 , according to the order of acquisition time of the image acquisition module 110 , the first acquired image is set as the reference image 302 , and the second to fifth acquired images are images to be processed 304 , 306 , and 308 respectively. , 310. In order to facilitate the description of the present invention, this embodiment takes acquisition of five images as an example for illustration, but the present invention is not limited thereto.

Next, in step S220, the motion estimation and correction module 120 performs global motion estimation on the to-be-processed images 304, 306, 308, and 310 based on the reference image 302 to generate a global motion vector, and then uses the global motion vector to perform global motion correction. , so as to generate a plurality of corrected images. Since the image acquisition module 110 adopts a rapid continuous shooting mode for a subject within a short period of time, it is used to capture local slight changes of the subject. The continuously shot images may have differences between the consecutive images due to the shake of the image capturing device 100 and the local movement of the subject itself. Among them, the local movement of the subject must be preserved, but the overall geometric deformation between images must be removed. Therefore, in this embodiment, the motion estimation and correction module 120 is used to perform geometric correction (registration) on the to-be-processed images 304, 306, 308, 310 relative to the reference image 302, so as to remove the overall variation between the images and generate corresponding Corrected images 304', 306', 308', 310'.

Next, as shown in step S230 , the image comparison module 130 compares each corrected image, marks local motion areas with pixel difference changes, and defines a mask covering all local motion areas. Specifically, the image comparison module 130 compares the difference between the pixel values of each pixel between each corrected image and its corresponding previous image according to the time sequence of image acquisition, so as to generate an image of each pixel difference.

For example, the image comparison module 130 first compares the pixel value difference between each pixel point of the corrected image 304' and the reference image 302, so as to generate an image difference value for each pixel point. Next, the image comparison module 130 judges whether the image difference is greater than a threshold value (threshold), if so, it means that the object at the pixel position may change, which means that the subject or an object in the image has a movement , so the pixel position greater than the threshold value is marked as the local action area. It should be noted here that the threshold value can be set in advance by using the image comparison module 130 to receive a grayscale image obtained by the image acquisition module 110 under a shooting condition, for example, a grayscale image obtained under a shooting condition of ISO100 step diagram. The image comparison module 130 counts the average brightness value and standard deviation of the grayscale image, and sets a threshold value based on the average brightness value and standard deviation. Therefore, under different shooting conditions, such as ISO100, ISO200, ISO400, ISO800, etc., different threshold values can be set.

After the image difference comparison is done for each pixel in the corrected image 304', all local action areas can be marked. Next, the image comparison module 130, for example, selects a minimum rectangular area covering all marked local action areas in the corrected image 304', as a mask of the corrected image 304'. Wherein, the shape and size of the mask can be set by the user according to actual needs, and the shape and size are not limited here. Similarly, after the image comparison module 130 respectively performs the above steps on the corrected images 306', 308', 310', a mask of the corrected images 306', 308', 310' can be respectively obtained. FIG. 4 is a schematic diagram of a rectangular area of a reference image and a corrected image displayed according to an embodiment of the present invention. Referring to FIG. 4 , in this embodiment, it is assumed that the rectangular areas a, b, c, and d are the mask positions of the corrected images 304', 306', 308', and 310', respectively.

Finally, as shown in step S240, the display module 140 of the image acquisition device 100 that generates the dynamic image first plays the reference image 302, and then updates the reference image according to the image content of the mask position (rectangular area a) of the corrected image 304' 302 the playback content in the rectangular area A; then update the playback content of the reference image 302 in the rectangular area B according to the image content of the mask position (rectangular area b) of the corrected image 306'; then follow the corrected image 308 'Update the playback content of the reference image 302 in the rectangular area C with the image content at the mask position (rectangular area c); finally update the reference image based on the image content at the mask position (rectangular area d) of the corrected image 310' 302 Play content in the rectangular area D.

By first displaying the first complete reference image 302, and then according to the previously recorded mask positions covering all marked local action areas, the image content played by the display module 140 can be continuously and locally updated, so as to achieve The playback effect of motion pictures. In addition, the above-mentioned image content updated from the reference image 302 to the mask position of the corrected image 310' can be regarded as a loop playback, and the user can also set the display module 140 to repeat the loop playback for more than one time, so as to achieve emphasis on local areas. A dynamic image playback effect with brief image changes.

Another embodiment is given below as an example in which the present invention can indeed be implemented. FIG. 5 is a block diagram of an image acquisition device for generating dynamic images according to another embodiment of the present invention. Please refer to FIG. 5 , the image acquisition device 500 for generating dynamic images not only includes the image acquisition module 110 , the movement amount estimation and correction module 120 , the image comparison module 130 and the display module 140 , but also includes a filter processing module. 550 and storage module 560. Wherein, the filter processing module 550 is coupled to the image comparison module 130 . The storage module 560 is coupled to the filter processing module 550 and the display module 140 .

FIG. 6 is a flow chart of an image processing method for generating dynamic images according to another embodiment of the present invention. The operation of the image acquisition device 500 for generating dynamic images will be described below with reference to FIG. 6 . Please refer to Figure 5 and Figure 6 at the same time.

The image acquisition module 110 is used to continuously acquire a plurality of images, one of which is a reference image, and the rest are images to be processed (step S610 ). The motion estimation and correction module 120 estimates the global motion of the image to be processed according to the reference image to generate a global motion vector, and then uses the global motion vector to perform global motion correction to generate multiple corrected images (step S620). The image comparison module 130 compares each corrected image, marks the local motion areas with pixel difference changes, and defines a mask covering all the local motion areas (step S630 ). After the step S630, color correction is also performed by using a filter processing method for all local motion regions within the range of the rectangular region in each corrected image and corresponding regions in the reference image (step S640).

With reference to FIG. 4 , it can be seen from the previous embodiment that the image content of the mask position (rectangular area a) of the corrected image 304 ′ will be used to update the corresponding area (rectangular area A) of the reference image 302 playback content. In the present invention, in order to highlight local short-term image changes, the continuity of image color is also an important consideration. Therefore, in order to reduce the problem of discontinuity in the outline and color of the edge of the mask after partial update, the filter processing module 550 can be further used The edge area of the image content at the mask position (rectangular area a) of the corrected image 304 ′ is finely adjusted with the reference image 302 . The filter processing module 550 may, for example, adopt a bidirectional filter (bilateral filter) for correction, so as to fully reflect the contour of the local motion region with pixel difference changes. Similarly, the filter processing module 550 performs the same processing on the corrected images 306', 308', and 310' respectively. After the above correction, the problem of color discontinuity between images can be reduced when the mask of each corrected image is updated.

Next, the storage module 560 stores a complete reference image and the image contents of the rectangular areas of each corrected image after color correction (step S650 ), and provides them to the display module 140 for playback. Compared with the current general camera that must completely store each image, the present invention can save storage space. Finally, the display module 140 first plays the reference image, and sequentially updates the playback content of the reference image according to the stored rectangular areas of the corrected images (step S660 ). Other detailed processes of this embodiment have been described in the above embodiments, and will not be repeated here.

FIG. 7 is a block diagram of an image acquisition device for generating dynamic images according to yet another embodiment of the present invention. Please refer to FIG. 7 , different from the previous embodiments, the image acquisition device 700 for generating dynamic images further includes a moving amount estimation and correction module 120, an image comparison module 130, a filter processing module 550, and a storage module. Compression module 770 of group 560 . The compression module 770 is, for example, one of the H.26x compression module, the animation expert group 2/4 compression module, the motion joint video expert group compression module, and the like. In this embodiment, the compression module 770 mainly includes two functions.

One is to receive the corrected image generated by the motion estimation and correction module 120, and use the local motion estimation function in the compression module 770 to find the local motion vector of each corrected image relative to the reference image , and then send this information to the image comparison module 130 . Next, the image comparison module 130 compares the pixel difference of each pixel for the block with the region motion vector. Accordingly, the image comparison module 130 only needs to compare the pixels in the block where the region motion vector exists, instead of comparing every pixel in the entire image, thereby saving the overall calculation time.

Furthermore, since the storage module 560 only stores a complete reference image and stores the rectangular regions of each corrected image after color correction. That is, except for the first reference image, the other subsequent images only need to keep the selected local motion area, and further define a mask that can cover the minimum local motion area according to the screened local motion area. In an embodiment, the position of the mask and the content of the image can be progressively stored in the reference image in the form of attached information. The attached information is, for example, a metadata format. In another embodiment, the image content of each mask can be partially fused with the previous image in advance to generate a new image. After sequential processing, these newly generated images can be combined with the reference image Compress in the compression module 770 together. Accordingly, the compressed image files can be played by various players that meet the compression standards.

To sum up, the present invention firstly displays the first complete reference image, and then according to the pre-recorded content and position of the local image area, it can continuously and quickly update the area content locally, which can realize local emphasis on ordinary cameras. A dynamic image playback effect with brief image changes. The present invention is not limited to the playback of GIF format image files, so it can effectively solve the situation of discontinuous color gamut in the prior art and achieve good playback effect. Furthermore, compared with the current general camera that must completely store each image for playing dynamic images, the present invention only needs to store one complete image, thus saving storage space.

Although the present invention has been disclosed above with the embodiments, it is not intended to limit the present invention, and any person of ordinary skill in the art may make some changes and modifications without departing from the spirit and scope of the present invention.

Claims (12)

1. produce an image treatment method for dynamic image, be applicable to a video capturing device with a display module, comprise:

The multiple image of continuous acquisition, those images one of them be one with reference to image, remaining those image is multiple pending images;

According to this reference image to those pending images carry out one universe amount of movement estimate and a universe amount of movement corrects, use generation multiple correction after image;

After each this correction, carry out pixel comparison in image respectively, mark an activities region according to the pixel value difference of image last image corresponding thereto after each this correction, and a shade in this activities region is contained in definition; And

This display module by this video capturing device plays this reference image, and sequentially upgrades the play content of this reference image according to the position of this shade of image after each this correction and size.

2. the image treatment method of generation dynamic image according to claim 1, wherein after each this correction, carry out pixel comparison in image respectively, mark provides this activities region of pixel value difference change, and the step that a shade in this activities region is contained in definition comprises:

Respectively after this correction, image compares the difference of the pixel value of each pixel respectively with last image according to the time sequencing of image capturing, uses the The image difference of each pixel of generation;

Judge whether this image difference is greater than a threshold value, if so, mark this pixel for this activities region; And

All rectangular areas being labeled as this activities region are contained in image, using this shade of image after each this correction after respectively this correction selected by circle.

3. the image treatment method of generation dynamic image according to claim 2, wherein after respectively this correction selected by circle, contain all this rectangular areas being labeled as this activities region in image, after the step of this shade of image after each this correction, also comprise:

The region corresponding with this reference image to this activities regions all belonged in image after each this correction in the scope of this rectangular area, adopts a filtering facture to carry out colour correction; And

Store the presentation content of this rectangular area of image after this reference image and respectively this correction after carrying out colour correction, play to be supplied to this display module.

4. the image treatment method of generation dynamic image according to claim 2, wherein:

Utilize this video capturing device to obtain a gray-scale figure under a shooting condition, add up an average brightness value and a standard deviation of this gray-scale figure, and set this threshold value under this shooting condition according to this average brightness value and this standard deviation.

5. the image treatment method of generation dynamic image according to claim 1, wherein also comprises after producing those and correcting the step of rear image:

Utilize a dynamic image compression method to carry out a region amount of movement to image after each this correction respectively to estimate, use generation respectively multiple regions motion-vector of image after this correction; And

The pixel value region that there are those region motion-vectors in image after each this correction being carried out to each pixel compares, and uses this activities region that mark provides pixel value difference change.

6. the image treatment method of generation dynamic image according to claim 5, wherein this dynamic image compression method comprises H.264 compression method, mpeg group 2/4 compression method or dynamic syndicated image expert group compression method.

7. produce a video capturing device for dynamic image, comprising:

One image capturing module, continuously obtains multiple image, those images one of them be one with reference to image, remaining those image is multiple pending images;

One amount of movement estimate with correct module, be coupled to this image capturing module, according to this reference image to those pending images carry out one universe amount of movement estimate and a universe amount of movement correct, use generation multiple correction after image;

One image comparison module, be coupled to this image capturing module and this amount of movement estimate and correct module, respectively image after each this correction is compared, mark an activities region according to the pixel value difference of image last image corresponding thereto after each this correction, and a shade in this activities region is contained in definition; And

One display module, is coupled to this image comparison module, shows this reference image, and foundation respectively after this correction the position of this shade of image and size sequentially upgrade the displaying contents of this reference image.

8. the video capturing device of generation dynamic image according to claim 7, wherein:

This image comparison module according to the time sequencing of image capturing respectively to the difference of each pixel of image after each this correction with each pixel compared pixels value of last corresponding image, use the The image difference producing each pixel, this image comparison module judges whether this image difference is greater than a threshold value, if, this this pixel of image comparison module marking is this activities region, and contain all rectangular areas being labeled as this activities region in image after respectively this correction selected by circle, using this shade of image after each this correction.

9. the video capturing device of generation dynamic image according to claim 8, wherein also comprises:

One filtering process module, is coupled to this image comparison module, and filtering process is carried out in the region corresponding with this reference image to this activities regions all belonged in image after each this correction in the scope of this rectangular area; And

One stores module, is coupled to this filtering process module, stores this reference image and the presentation content of this rectangular area of image after respectively this correction after filtering process is carried out in storage.

10. the video capturing device of generation dynamic image according to claim 8, wherein:

This image comparison module receives the gray-scale figure that this video capturing device obtains under a shooting condition, this image comparison module adds up an average brightness value and a standard deviation of this gray-scale figure, and sets this threshold value under this shooting condition according to this average brightness value and this standard deviation.

The video capturing device of 11. generation dynamic images according to claim 7, wherein also comprises:

One compression module, be coupled to this image comparison module, carry out a region amount of movement to image after each this correction to estimate, use generation respectively multiple regions motion-vector of image after this correction, this image comparison module compares the pixel value that each pixel is carried out in the region that there are those region motion-vectors, uses this activities region that mark provides pixel value difference change.

The video capturing device of 12. generation dynamic images according to claim 11, wherein:

This compression module for H.264 compressing module, mpeg group 2/4 compresses module, dynamic syndicated image expert group compression module one of them.