TW201531744A - Image processing device - Google Patents

Abstract

An image processing device including an image sensor array, an image pre-processing unit, a depth information generator, and a focusing unit is provided. The image sensor array is configured to take multiple images of two objects (e.g., a first object and a second object). The image pre-processing unit is configured to process the images to generate two shift images associated with the two objects. The depth information generator is configured to generate depth information according to the two shift images. The depth information includes distance information associated with the first object. The focusing unit is configured to generate a pair of focused images that have the first object focused thereon according to the depth information and the two shift images.

Description

Image processing device

The present invention relates to an image processing apparatus, and particularly relates to depth information that can generate images, and can generate large-sized two-dimensional (2D) images and stereoscopic three-dimensional images (stereoscopic 3D video) suitable for human eyes according to depth information. An image processing device.

Traditional cameras use a single lens and a single image sensor, and use a voice coil motor (VCM) to drive the lens back and forth to achieve autofocus and depth information for two-dimensional (2D) images and three-dimensional (3D) Prepare for the production of images. However, the voice coil motor is slow, consumes power, and emits noise. These flaws make the function of generating deep information more time and power to complete. Image input using multiple cameras or image matrices cannot overcome the problem of obtaining large-size 2D images, how to handle multiple complex image inputs, and how to instantly generate stereoscopic 3D images suitable for human eyes.

Camera users want to be able to take large 2D images, such as 10 megapixels. Images from multiple cameras or image matrices can make depth information easier, but multiple cameras or image matrices Image output sizes are usually small, such as one million pixels per frame, and these multiple small images need to have the same focus plane. How to produce clear large-size 2D images will be a challenge.

When multiple input images have different imaging planes combined with optical zoom, how to instantly generate stereoscopic 3D images suitable for human eyes can cause more complex challenges, especially when the user is interested in objects. When moving around, it will make it easier to use images from multiple cameras or image matrices into the camera.

The invention provides an image processing device, which combines multiple image sensors or image matrix to input images of different focus planes, and uses digital image processing technology to simultaneously and instantly generate large-sized two-dimensional images and generate stereoscopic three-dimensional images suitable for human eyes. It has the effect of fast and power saving. The above image processing apparatus also includes a plurality of application functions of depth information.

The image processing apparatus of the present invention includes an image sensing array, an image preprocessing unit, a depth generator, and a focusing unit. The image sensing array includes a plurality of image sensors for capturing a plurality of images for the first object and the second object. The image pre-processing unit is coupled to the image sensing array for receiving the plurality of images, and processing the plurality of images to generate a first displacement image and a second displacement image related to the first object and the second object. The depth generator is coupled to the image pre-processing unit for generating depth information according to the first displacement image and the second displacement image. In-depth information The first distance information related to the first object is included. The focus unit is coupled to the image pre-processing unit and the depth generator for generating a first pair of focus images according to the depth information, the first displacement image and the second displacement image, wherein the first pair of focus images are focused on the first object.

The above described features and advantages of the invention will be apparent from the following description.

100‧‧‧Image processing device

105‧‧‧Image sensing array

110‧‧‧ lens array

120‧‧‧sensor array

125‧‧•Image pre-processing unit

130‧‧‧Image Processing Pipeline

140‧‧‧Image Analyzer

150‧‧‧Two-dimensional image synthesizer

160‧‧‧ Focus unit

170‧‧‧ storage unit

180‧‧‧Deep Generator

310‧‧‧Lens distortion correction unit

320‧‧‧Synchronous processing unit

330‧‧‧To the noise unit

340‧‧‧ Parallax Correction Unit

350‧‧‧Image Correction Unit

415, 425‧‧‧ cut box

510‧‧‧Color Space Converter

520‧‧‧Background model unit

530‧‧‧object outline unit

540‧‧‧Offset estimator

550‧‧‧post processor

560‧‧‧Offset depth converter

570‧‧‧Infrared transceiver

610~650‧‧‧Deep information area

810‧‧‧Image background

815, 825‧‧‧Infrared reflected light spots

820‧‧‧Image prospects

910~930‧‧‧ objects

1000‧‧‧Image processing device

1020‧‧‧3D image synthesizer

1030‧‧‧ Shadow Point Detector

1040‧‧‧Display unit

1130, 1140‧‧‧ objects

1135, 1137, 1145, 1147‧‧ ‧ shadow points

1200‧‧‧Image processing device

M1, M2‧‧‧ Shadowing position information

R1, L1, R2, L2‧‧‧ displacement images

R3, L3‧‧ ‧ focus image

FIG. 1 is a schematic diagram of an image processing apparatus according to an embodiment of the invention.

2 and 3 are schematic diagrams of an image processing pipeline in accordance with an embodiment of the present invention.

4 is a schematic diagram of parallax correction in accordance with an embodiment of the present invention.

Figure 5 is a schematic illustration of a depth generator in accordance with an embodiment of the present invention.

6A and 6B are schematic diagrams of depth information according to an embodiment of the invention.

FIG. 7 is a schematic diagram of a depth generator in accordance with another embodiment of the present invention.

Figure 8 is a schematic illustration of an infrared reflected spot in accordance with an embodiment of the present invention.

9A-9D are schematic diagrams of a focused image in accordance with an embodiment of the present invention.

FIG. 10 is a schematic diagram of an image processing apparatus according to another embodiment of the present invention.

Figure 11 is a schematic illustration of a shadow point in accordance with an embodiment of the present invention.

FIG. 12 is a schematic diagram of an image processing apparatus according to another embodiment of the present invention. Figure.

FIG. 1 is a schematic diagram of an image processing apparatus 100 in accordance with an embodiment of the invention. The image processing device 100 can be a digital camera, a digital camera, or an electronic device having camera and/or camera functions, such as a personal digital assistant (PDA), a smart phone, or a tablet. The image processing apparatus 100 includes an image sensing array 105, an image preprocessing unit 125, a plurality of image processing pipelines 130, an image analyzer 140, a focusing unit 160, a depth generator 180, a two-dimensional (2D) image synthesizer 150, and storage. Unit 170. The image sensing array 105 includes a lens array 110 and a sensor array 120. The sensor array 120 is coupled to the lens array 110. The image pre-processing unit 125 is coupled to the sensor array 120. The image processing pipeline 130 is coupled to the pre-processing unit 125. The image analyzer 140, the focusing unit 160, and the depth generator 180 are coupled. The image processing pipeline 130 is coupled to the focusing unit 160. The storage unit 170 is coupled to the two-dimensional image synthesizer 150.

The sensor array 120 can include a plurality of image sensors (eg, a plurality of cameras) for capturing a plurality of images for one or more objects in the same scene, and outputting the plurality of images to the image pre-processing unit 125 . In the sensor array 120, the focal length of each image sensor can be fixed or variable, and each image sensor can use a fixed lens or a liquid lens ( Liquid lens), crystal lens, or rnicro-lens array. Focus distance of the image sensor in the sensor array 120 Can be the same or different. In other words, the plurality of images captured by the sensor array 120 may have the same focal plane. Alternatively, the plurality of images captured by the sensor array 120 may have a plurality of different focus planes.

The lens array 110 may include a plurality of optical zoom lenses that are in one-to-one correspondence with a plurality of image sensors of the sensor array 120. Each optical zoom lens is disposed in front of the corresponding sensor. These optical zoom lenses can zoom in on distant objects in the above images to improve the layering of distant objects.

The image pre-processing unit 125 can receive the plurality of images captured by the sensor array 120 and process the plurality of images to generate two displacement images R1 and L1 associated with the one or more objects. The displacement images R1 and L1 can be assumed to correspond to images seen by the right and left eyes of the user, respectively.

In more detail, the image pre-processing unit 125 can correct the image geometric plane of the plurality of images captured by the sensor array 120, and classify the plurality of images according to the physical relative positions of the plurality of images and the focus plane. Corresponding to the two image groups of the right eye and the left eye. The image pre-processing unit 125 can then combine multiple images in the first image group into a single image multi-frame super resolution according to the optical characteristics of each image in the first image group to generate a corresponding right eye. Displacement image R1. Similarly, the image pre-processing unit 125 can synthesize a plurality of images in the second image group into a single image according to the optical characteristics of each image in the second image group to generate a displacement image corresponding to the left eye. .

The single image magnification output described above is to combine multiple smaller images into one Larger images, such as the image of two five million pixels, are combined into a single image of ten million pixels. The plurality of image sensors of the sensor array 120 can capture a plurality of images at the same time (ie, synchronous), and then the image pre-processing unit 125 can generate a higher resolution displacement using the single image magnification output technology and the plurality of images. Images R1 and L1.

In addition, the image pre-processing unit 125 can also support a high dynamic range (HDR) technology. The traditional high dynamic range technique is to capture multiple images at different times using different exposure values at different times, and then combine an image to achieve a wider range of light and dark in a single image. The plurality of image sensors of the sensor array 120 can capture a plurality of images using different exposure values at the same time, and then the image pre-processing unit 125 can use a high dynamic range technology to generate a wider range of light and dark according to the plurality of images. Displace images R1 and L1. The above multiple images are taken at the same time, which is not only more efficient, but also better in effect, for example, the influence of the user's hand shake can be minimized.

Image processing pipeline 130 is shown in FIG. In this embodiment, the image processing apparatus 100 can include at least two image processing pipelines 130. One image processing pipeline 130 can receive the displacement image R1 by the image pre-processing unit 125, and the other image processing pipeline 130 can be processed by the image pre-processing. Unit 125 receives the displacement image L1. Each image processing pipeline 130 includes a plurality of image processing units coupled in series, and sequentially performs multi-stage image processing on the received displacement image R1 (or L1), and outputs corresponding to the displacement image R1 (or L1). The displacement image R2 (or L2). Further, the first image processing unit in each image processing pipeline 130 receives the displacement image. R1 (or L1) is used as an input, and each of the remaining image processing units receives the output of the previous image processing unit as an input. These image processing units perform a series of image processing on the shifted images R1 (and L1). In the following description, two displacement images respectively received by the two image processing pipelines 130 are represented by R1 and L1, and two displacement images respectively output by the two image processing pipelines 130 are represented by R2 and L2. The two image processing pipelines 130 output the displacement images R2 and L2 to the image analyzer 140, the focusing unit 160, and the depth generator 180.

For example, each image processing pipeline 130 may be as shown in FIG. 3 , wherein the image processing unit may include a lens distortion correction unit 310 coupled in series, and a synchronization processing unit. 320. A de-noise unit 330, a parallax calibration unit 340, and an image rectification unit 350. The following description takes the image processing pipeline 130 above the FIG. 2 as an example. For the description of the image processing pipeline 130 in the lower part of FIG. 2, it is only necessary to replace the displacement images R1 and R2 with the displacement images L1 and L2, respectively.

The lens distortion correcting unit 310 corrects the fisheye phenomenon in the displacement image R1, that is, a phenomenon in which the straight line is slightly bent after being photographed.

The synchronization processing unit 320 corrects and reduces the difference between the displaced images R1 and L1, which may include the shooting time, exposure, color, white balance, and focus of the image sensor of the sensor array 120. One or more differences in the plane.

The noise removing unit 330 can filter out noise in the displacement image R1, including brightness. Noise and color noise.

The parallax correcting unit 340 corresponding to the displacement image R1 determines a cropping frame and its position in the displacement image R1, and cuts out the portion other than the cutting frame and retains the cutting side on the displacement image R1. The part within the box. Similarly, the parallax correction unit 340 corresponding to the displacement image L1 determines another cropping frame and its position in the displacement image L1, and cuts out the portion other than the cropping frame on the displacement image L1 and Keep the parts within the crop box. In order to create the parallax effect required between the two displacement images L1 and R1, the parallax correction unit 340 arranges the cropping frame at different positions in the displacement image R1 and the displacement image L1 so that the horizon of each displacement image There is a slight difference in (view).

For example, as shown in FIG. 4, in this example, two of the two image processing pipelines 130 determine the cropping blocks 415 and 425 and the two cropping boxes on the displacement images L1 and R1, respectively. position. The positions of the two cropping blocks 415, 425 are different. If placed in the same image, there will be a small distance between the two cropping blocks 415, 425, which is based on multiple image sensing of the sensor array 120. The distance between the multiple lenses of the device is determined. Therefore, each parallax correcting unit 340 determines the position of the cutting frame and the cutting frame in the displacement image according to the distance between the plurality of lenses described above to create a parallax effect.

The plurality of lenses of the plurality of image sensors of the sensor array 120 should be mounted on the same plane, and the mounting angles of the plurality of lenses should be the same. For example, the upper of the field of view of each lens must point in the same direction, and there should be no deviation in the angle of rotation. However, in the manufacturing process, the mounting position of each lens is not necessarily the same. On the plane, the deviation of the mounting angle is also inevitable. The image correcting unit 350 can correct the distortion caused by the deviation of the above-described mounting position and/or mounting angle from the displacement image R1. For example, an affine transform can be used to correct the above distortion.

FIG. 5 illustrates further details of depth generator 180. The depth generator 180 includes a color space converter 510, a background modeling unit 520, an object contour unit 530, an offset estimator 540, a post processor 550, and an offset depth converter 560. The color space converter 510 is coupled to each of the image processing pipelines 130. The background model unit 520 and the offset estimator 540 are each coupled to a color space converter 510. The object contour unit 530 is coupled to the background model unit 520. The post processor 550 is coupled. The object contour unit 530 and the offset estimator 540 are coupled between the post processor 550 and the focusing unit 160.

The depth generator 180 generates depth information (e.g., a depth map) based on the displacement images R2 and L2. This depth information includes distance information relating to each object captured by the sensor array 120. For example, the distance information may be the distance between the corresponding object and the lens of the image sensor, and the distance may also be referred to as a depth or depth value.

6A and 6B are schematic diagrams of depth information according to an embodiment of the invention. FIG. 6A is a scene taken by the sensor array 120, in which there are a plurality of objects (for example, a plurality of dolls), and the depth information generated by the depth generator 180 corresponding to the scene is as shown in FIG. 6B. The depth information may be a two-dimensional matrix composed of depth values, wherein each depth value is a depth value of an object at the same position among the displacement images R2 and L2. The depth value of Figure 6B has been converted to a corresponding gray scale to facilitate display. The object of Figure 6A can be divided into five The levels correspond to the five regions 610-650 of Figure 6B, respectively, wherein the darker the gray area, the further the distance.

Each unit of the depth generator 180 is explained below. The color space converter 510 can convert the displacement images R2 and L2 from the first color space to the second color space. The first color space described above does not include a luminance component, such as RGB or CMYK; and the second color space includes a luminance component and at least one color component, such as YUV or YCbCr. The sensor array 120 of the present embodiment captures images using the RGB color space, and the color space converter 510 can be omitted if the sensor array 120 can capture images using a color space containing luminance components such as YUV.

The offset estimator 540 can generate offset information (eg, a disparity map) according to the luminance components of the displacement images R2 and L2 (eg, the Y component of the YUV color space), the offset information including each object between the displacement images R2 and L2. The offset. The offset refers to the difference between different positions in the different images of the same point of the same object. Objects that are closer to the lens will have a larger offset, so the depth information can be derived using the offset plus the spacing of the lens of the image sensor. The offset estimator 540 can detect and estimate the offset of the object to generate offset information. The offset information can be a two-dimensional matrix similar to that of Figure 6B, except that the depth values are replaced by offsets.

The background model unit 520 can distinguish the foreground and background of the displaced image from at least one color component of the displacement image R2 or L2 (eg, the U component and the V component of the YUV color space). The above prospects refer to the parts that the user is more likely to be interested in, while the background is the less important part. Because each of the sensor arrays 120 The field of view of the image sensor is not much different, so the background model unit 520 can extract only the foreground or background information of one of the images R2 and L2.

In the embodiment of FIG. 5, the background model unit 520 is also coupled to the offset estimator 540. The background model unit 520 can estimate the object depth based on the offset information generated by the offset estimator 540, and then distinguish the foreground from the background based on the depth.

The object contour unit 530 can extract an object contour among the foreground, and the post processor 550 can modify the offset information according to the object contour, and particularly modify the object contour information in the offset information.

Post processor 550 is responsible for modifying the offset information generated by offset estimator 540. The post-processor 550 can modify the foreground object contour in the offset information according to the object contour extracted by the object contour unit 530 in the foreground, that is, the protrusion and the roughness of the foreground object contour in the offset information, and the offset The foreground object outline in the information is smoothed.

In addition, the post processor 550 can patch the foreground and background anomalies in the offset information. The offset information can be a two-dimensional matrix similar to a two-dimensional image, for example, where there are three or five adjacent points around each point of the matrix edge, and eight points in the interior of the matrix are surrounded by adjacent points. If there is a difference between the offset of a point and any adjacent point that is greater than a threshold, the point is treated as an abnormal point, and the post-processor 550 averages the offset of all the immediate points of the point. The value replaces the offset of this point.

The offset depth converter 560 can convert the offset information modified by the post processor 550 into depth information for use by the focusing unit 160.

FIG. 7 illustrates further details of depth generator 180 in accordance with another embodiment of the present invention. The depth generator 180 of this embodiment further includes an infrared transceiver 570 coupled to the background model unit 520. The infrared transceiver 570 can emit infrared rays and sense the reflected spots of the infrared rays. For example, FIG. 8 illustrates an infrared image sensed by the infrared transceiver 570 of the embodiment. The background 810 has a plurality of reflected spots 815, and the foreground 820 has a plurality of reflected spots 825. Because the distance of the foreground is relatively close, the reflected spot 825 of the foreground 820 will be relatively large and relatively bright, while the reflected spot 815 of the background 810 will be relatively small and relatively dark. The background model unit 520 can distinguish the foreground and background described above from the above differences in reflected light spots.

The focusing unit 160 can generate the focused images R3 and L3 of the objects specified or interested by the two users according to the depth information, the displacement images R2 and L2, and/or the lens parameters of each of the image sensors. The R3 and L3 are simultaneously focused on the same object. The lens parameters described above include a focal length and a point spread function of the corresponding lens of the image sensor. The lens parameters of each image sensor can be the same or different.

9A to 9D are an example of the above-described focused image. FIG. 9A is a scene taken by the image processing apparatus 100, in which there are three objects 910-930. In this example, the focusing unit 160 generates three pairs of in-focus images R3 and L3 according to the depth information generated by the depth generator 180, and respectively focuses on the objects 910-930. Focusing on the focused images R3 and L3 of the object 910, as shown in FIG. 9B, the object 910 is the clearest, and the objects 920, 930 are relatively blurred. Focusing on the focused images R3 and L3 of the object 920, as shown in FIG. 9C, the object 920 is the clearest, and the objects 910 and 930 are relatively blurred. Correct Focusing on the focused images R3 and L3 of the object 930, as shown in FIG. 9D, the object 930 is the clearest, and the objects 910, 920 are relatively blurred.

The theoretical basis for the focus unit 160 to produce a focused image that focuses on any object is the paper "A New Sense for Depth of Field" by APPentland in IEEE Transactions on Pattern Analysis and Machine Intelligence, 9(4): 523-531, 1987. . This paper describes the relationship between image depth information, focus plane depth, and lens parameters. The focus plane depth and lens parameters are known at the hardware design of the image processing apparatus 100, and the depth information is from the depth generator 180.

The image analyzer 140 provides a smart auto focus function. In more detail, image analyzer 140 may identify the location of one or more objects, such as a human face, or a region having features, in displacement images R2 and/or L2 to produce a position relative to one or more of the objects described above. Information, wherein the focusing unit 160 generates one or more pairs of in-focus images relative to the one or more objects based on the position information described above. In addition to being automatically recognized by image analyzer 140, the above-described focusing object can also be specified by the user.

For example, in the example of FIGS. 9A-9D, the image analyzer 140 can identify the objects 910-930 in the displacement images R2 and L2, and transmit the position information of the objects 910-930 to the focusing unit 160. Therefore, the focusing unit 160 can respectively focus on the objects 910-930 according to the received position information to generate three pairs of focus images R3 and L3 as shown in FIGS. 9B to 9D.

The image processing apparatus 100 can capture still images or dynamic movies, wherein A movie is a combination of still images that are continuously shot. In an embodiment of the invention, the image sensing array 105 can continuously capture a plurality of images, and the image analyzer 140 can continuously track one or more objects, such as a face or a moving object, in the displacement images R2 and L2, and Position information of these objects is provided for the focusing unit 160 to generate a focused image. In addition to being automatically recognized by image analyzer 140, the above-described focusing object can also be specified by the user. For example, if a pedestrian walks from the back of the shooting scene to the front, the user can designate the pedestrian as the focusing object, and the focusing unit 160 can continuously track the focus so that the pedestrian moves with the focus no matter where he goes.

Returning to FIG. 1, the two-dimensional image synthesizer 150 can receive the focused images R3 and L3 by the focusing unit 160, and perform two-dimensional image interpolation based on the focused images R3 and L3 to generate a two-dimensional synthetic image. The resolution of the synthesized image may be greater than or equal to the resolution of the focused images R3 and L3. The storage unit 170 can receive and store the synthesized image, and can also store one or more of the depth map, the depth of focus, and the lens parameters. If the image processing apparatus 100 captures a dynamic movie, the storage unit 170 may encode the continuous composite image as a movie storage.

FIG. 10 is a schematic diagram of an image processing apparatus 1000 in accordance with another embodiment of the present invention. The image processing device 1000 includes an image sensing array 105, an image preprocessing unit 125, a plurality of image processing pipelines 130, a image analyzer 140, a focusing unit 160, a depth generator 180, a three-dimensional (3D) image synthesizer 1020, and a shadow point ( The occlusion detector 1030, the display unit 1040, and the storage unit 170. The image sensing array 105, the image pre-processing unit 125, and the plurality of image processing pipelines 130, the image analyzer 140, the focusing unit 160, and the depth generator 180 are respectively the same as the corresponding elements of FIG. 1, and will not be described again. The masking point detector 1030 is coupled to the image analyzer 140, the focusing unit 160 and the depth generator 180. The three-dimensional image synthesizer 1020 is coupled to the shadow point detector 1030 and the focusing unit 160. The display unit 1040 and the storage unit 170 are coupled. A three-dimensional image synthesizer 1020.

The shadow point detector 1030 can receive the focus images R3 and L3 from the focusing unit 160, and receive the position information relative to the object output by the image analyzer 140 and the depth information output by the depth generator 180, and according to the received The focus images R3 and L3, the position information and the depth information generate the mask point position information M1 and M2 corresponding to the focus images R3 and L3. The obscuration point is the part of the stereoscopic three-dimensional image that is obscured by the object and is only seen by one of the eyes of the human eye, that is, the part of the scene captured by the sensor array 120 that is only captured by a part of the image sensor. For example, in the example shown in FIG. 11, the focus images R3 and L3 have two objects 1130 and 1140 therein. The focus image L3 includes shadow points 1135 and 1145, and the focus image R3 includes mask points 1137 and 1147. Since the image sensors of the sensor array 120 are mounted at different positions, the shadow points also appear at different positions. The closer the shadow point is to the image sensor lens, the more obvious it will be.

Correcting the shadow point allows the user to see a more realistic and comfortable 3D image. The 3D image synthesizer 1020 can shift the object in the focus image R3 or L3 by a distance according to the shading point position information M1 or M2 to modify the edge of the object to correct the obscuration point of the object.

As described above, the three-dimensional image synthesizer 1020 can be based on the concealing point of the object. The position information M1 and M2 correct the shadow points in the focus images R3 and L3, and the three-dimensional image synthesizer 1020 can generate a three-dimensional composite image according to the focus images R3 and L3 and the at least one conceal point position information M1 or M2. Through the image content analysis and object tracking of the image analyzer 140, the time for detecting and processing the shadow point can be reduced, and the calculation amount of the shadow point correction can be changed in time to instantly generate a stereoscopic three-dimensional image suitable for human eyes.

The display unit 1040 receives the three-dimensional synthesized image from the three-dimensional image synthesizer 1020, and displays the three-dimensional synthesized image in a stereoscopic three-dimensional manner. The stereoscopic effect of a stereoscopic three-dimensional image refers to the extent to which the user appears to protrude from the screen or into the screen. The image processing device 1000 can provide a setting option to set the degree of convexity or indentation of the stereoscopic effect. The 3D image synthesizer 1020 can adjust the display unit 1040 according to the setting option and the screen size and resolution of the display unit 1040. Three-dimensional. The storage unit 170 can receive and store the three-dimensional composite image output by the three-dimensional image synthesizer 1020.

FIG. 12 is a schematic diagram of an image processing apparatus 1200 according to another embodiment of the present invention. The image processing device 1200 is a combination of the image processing device 100 of FIG. 1 and the image processing device 1000 of FIG. Therefore, the image processing device 1200 has all the functions of the image processing device 100 and the image processing device 1000. Furthermore, in the image processing device 1200, the 3D image synthesizer 1020 and the 2D image synthesizer 150 can simultaneously receive the in-focus images R3 and L3 by the focusing unit 160, thereby respectively generating a depth information and a high value for the captured object. a two-dimensional synthetic image of resolution and a three-dimensional synthetic image, and the storage unit 170 can receive and store the two-dimensional image Synthesize the image and the three-dimensional composite image.

In summary, the image processing apparatuses 100, 1000, and 1200 in the above embodiments use image processing technology to focus on images when shooting images without using a voice coil motor, so it is quieter and faster than the conventional scheme using a voice coil motor. More power saving. The image processing device 100, 1000, 1200 can refocus each object in the image by focusing the image after the image has been captured, thereby avoiding focus or tracking errors caused by human shooting. The image processing device 100, 1000, 1200 can synthesize a plurality of images simultaneously captured by the sensor array by using a high dynamic range technology to extend the brightness and darkness of the image, and can synthesize multiple images simultaneously captured by the sensor array by using a single image magnification output technology. Produces large-sized 2D images. The image processing device 1000, 1200 can detect and correct the shadow point in the image, and can adjust the three-dimensional sense displayed by the display unit. In summary, the image processing apparatuses 100, 1000, and 1200 in the embodiments of the present invention can provide two-dimensional images and stereoscopic three-dimensional images that are more suitable for human eyes to view.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

180‧‧‧Deep Generator

510‧‧‧Color Space Converter

520‧‧‧Background model unit

530‧‧‧object outline unit

540‧‧‧Offset estimator

550‧‧‧post processor

560‧‧‧Offset depth converter

R2, L2‧‧‧ displacement image

Claims (17)

An image processing device includes: an image sensing array, comprising: a plurality of image sensors for capturing a plurality of images for a first object and a second object; and an image pre-processing unit coupled to the image sensing array Receiving the plurality of images, and processing the plurality of images to generate a first displacement image and a second displacement image related to the first object and the second object; a depth generator coupled The image pre-processing unit is configured to generate a depth information according to the first displacement image and the second displacement image, wherein the depth information includes a first distance information related to the first object; and a focusing unit, The image processing unit and the depth generator are configured to generate a first pair of focus images according to the depth information, the first displacement image and the second displacement image, wherein the first pair of focus images are focused on the First object. The image processing device of claim 1, wherein the depth information further includes a second distance information related to the second object, and the focusing unit is further configured to: according to the depth information, the first displacement image And generating, by the second displacement image, a second pair of focus images, wherein the second pair of focus images is focused on the second object. The image processing device of claim 1, wherein the focusing unit generates the first pair of focus images according to lens parameters of each of the image sensors, and the lens parameters of each of the image sensors include the image sensing The focal length of the lens and the point spread function. The image processing device of claim 1, wherein the image The sensing array further includes: a lens array including a plurality of optical zoom lenses, wherein the plurality of optical zoom lenses are disposed in front of the plurality of image sensors. The image processing device of claim 1, wherein the depth generator comprises: an offset estimator coupled to the image pre-processing unit for the first displacement image and the second displacement image Generating an offset information, where the offset information includes an offset between the first object and the second object between the first displacement image and the second displacement image; and an offset depth converter coupled to the An offset estimator for converting the offset information into the depth information. The image processing device of claim 5, wherein the depth generator further comprises: a post processor coupled between the offset estimator and the offset depth converter for repairing the bias Move the anomaly in the message. The image processing device of claim 5, wherein the depth generator further comprises: a background model unit coupled to the image pre-processing unit for distinguishing the first displacement image or the second displacement image a foreground and a background; and an object contour unit coupled to the background model unit for extracting an object contour in the foreground. The image processing device of claim 7, wherein the background The model unit is also coupled to the offset estimator and distinguishes the foreground from the background based on the offset information. The image processing device of claim 7, wherein the depth generator further comprises: an infrared transceiver coupled to the background model unit for emitting an infrared ray to sense a reflected spot of the infrared ray, wherein The background model unit distinguishes the foreground from the background based on the reflected light spots described above. The image processing device of claim 7, wherein the depth generator further comprises: a color space converter coupled to the image pre-processing unit, the offset estimator, and the background model unit, Converting the first displacement image and the second displacement image from a first color space to a second color space, wherein the first color space does not include a brightness component, and the second color space includes a brightness component and at least a color component, the offset estimator generates the offset information according to the brightness component of the first displacement image and the second displacement image, the background model unit according to the at least the first displacement image or the second displacement image A color component distinguishes the foreground from the background. The image processing device of claim 1, wherein the image pre-processing unit is configured to classify the plurality of images into a first image group and a second image group, and according to the first image group The optical characteristics of each of the images are collected by the plurality of images in the first image group to generate the first displacement image, and the plurality of second image groups are assembled according to optical characteristics of each image in the second image group Image The second displacement image is generated. The image processing device of claim 1, wherein the plurality of images captured by the image sensing array are captured by the plurality of image sensors at different times at the same time, and the image is captured. The pre-processing unit generates the first displacement image and the second displacement image according to the plurality of images by using a high dynamic range technique. The image processing device of claim 1, further comprising: an image analyzer coupled to the image pre-processing unit and the focusing unit for use in the first displacement image or the second displacement image Identifying a position of the first object to generate position information relative to the first object, wherein the focusing unit further generates the first pair of in-focus images according to the position information. The image processing device of claim 1, further comprising: at least two image processing pipelines coupled between the image pre-processing unit, the focusing unit, and the depth generator for respectively receiving The first displacement image and the second displacement image, wherein each of the image processing pipelines includes: a synchronization processing unit that corrects a difference between the first displacement image and the second displacement image, wherein the difference includes at least a shooting time, One of the differences in exposure, color, white balance, and focus plane. The image processing device of claim 1, further comprising: an image synthesizer coupled to the focusing unit for generating a composite image according to the first pair of in-focus images. The image processing device of claim 15, wherein the image processing device The image is a two-dimensional image, and the image synthesizer performs a two-dimensional image complementing point according to the first pair of focused images to generate the synthesized image, and the resolution of the synthesized image is greater than the resolution of the first pair of focused images. The image processing device of claim 15, wherein the composite image is a three-dimensional image, and the image processing device further comprises: a shadow point detector coupled to the depth generator, the focusing unit and the image a synthesizer configured to generate, according to the depth information and the first pair of in-focus images, shading point position information corresponding to the first pair of in-focus images, wherein the image synthesizer is based on the first pair of in-focus images and the shading point position Information to generate the three-dimensional image; and a display unit coupled to the image synthesizer to display the three-dimensional image in a stereoscopic three-dimensional manner.