JP7659450B2 - Imaging device and imaging method - Google Patents

Description

The present invention relates to an imaging device and imaging method, and in particular to an imaging device and imaging method that perform super-resolution imaging using compressed sensing.

Super-resolution technology is a technology that creates high-pixel images from low-pixel images, and is used in various digital devices to support the increasing resolution of 4K and 8K technologies as they develop. There are many different methods, but a common one is to create high-resolution image data by preparing multiple low-resolution images and aligning them at the sub-pixel level. For example, if you have four images that are sampled by shifting a 32 x 32 pixel image vertically and horizontally by half a pixel, you can create an image with 64 x 64 pixels. As the number of pixels increases, the pixel size of cameras (image sensors) has become so fine that it is restricted by the diffraction limit of light, and so super-resolution technology has also developed in imaging in recent years.

As mentioned above, super-resolution requires multiple low-resolution images depending on the degree of resolution, so the frame rate drops during imaging. For this reason, a method has been proposed to generate a high-resolution image super-resolved from a small number of low-resolution images using compressive sensing. As it is known that information can be compressed by processing such as DCT (Discrete Cosine Transform) and DWT (Discrete Wavelet Transform), compressed sensing can be applied because the image can be transformed into a sparse space using some basis. In general, simultaneous equations cannot be solved unless there are more conditional equations than unknowns, whereas in compressed sensing, even if the number of conditional equations is small, the solution can be estimated under the assumption that the equations are sparse. In super-resolution of images, if the image is treated as a one-dimensional signal, the unknown is the "number of pixels in the high-resolution image," and the number of conditional equations can be expressed as the product of the "number of pixels in the low-resolution image" and the "number of low-resolution images encoded under different conditions." The solution is the "pixel value of the high-resolution image," but because it can be converted into sparse values, "DCT components" and "DWT components," using the DCT and DWT mentioned above, the solution can be found using compressed sensing. Finally, by performing an inverse DCT or inverse DWT on the estimated solution, a high-resolution image can be reconstructed.

Various proposals have been made to improve the performance of super-resolution processing, for example, a method of randomizing the distribution of light sensitivity of pixels has been proposed (Non-Patent Document 1). In Non-Patent Document 1, focusing on the fact that the resolution in super-resolution depends on the pixel shape of the camera, toner powder finer than the pixel size is randomly sprinkled on the sensor to encode the subject, 20 low-resolution images (120 x 100) are taken while changing the relative positional relationship between the subject and the camera, and a high-resolution image (1600 x 1200) is generated by compressed sensing using sparsity by DWT. In this method, the size of the toner powder and the amount of shift in the position of the camera determine the number of pixels of the high-resolution image to be reconstructed.

It has also been proposed to display random patterns on a DMD (digital micromirror) to encode the image incident on the DMD and achieve super-resolution (Non-Patent Documents 2 and 3). In Non-Patent Document 2, super-resolution is achieved by 64 times both vertically and horizontally. Although 4096 patterns (number of shots) would normally be required, compressed sensing reconstructs the image with 1600 or 2700 patterns. In Non-Patent Document 3, super-resolution is achieved by 4 times both vertically and horizontally, and the reconstructed images are compared for cases where the number of patterns ranges from 1 to 16. In these methods, the number of pixels of the DMD (resolution of the encoded pattern) determines the number of pixels of the high-resolution image.

Tomotaka Sasao, Shinsaku Hiura, Kosuke Sato, "Super-resolution based on random coding of pixel shape", IEICE Transactions on Computer Vision, Vol. J96-D No. 8, (2013), pp. 1778-1789 D. Takhar et al. “A New Compressive Imaging Camera Architecture using Optical-Domain Compression,” Electronic Imaging, (2006) J. Flake et al. “Experimental study of super-resolution using a compressive sensing architecture,” Proceedings of SPIE, (2014)

However, in the method of Non-Patent Document 1, it is difficult to arrange fine toner powder at any position on a pixel, and it is necessary to check in advance how the toner powder is arranged relative to the camera pixels. Since the encoding pattern is created by randomly sprinkling toner powder, there are cases where pixels are completely covered with toner powder or toner powder is concentrated in a certain place. Therefore, even if the camera position is changed to acquire multiple images, a certain part of the light of the subject cannot be received in all images, and the image quality of the high-resolution image obtained by super-resolution may be degraded or information may be lost. In addition, since the encoding is performed using toner powder and the encoding pattern is fixed and cannot be modulated, it is necessary to physically move the camera position multiple times to change the encoding conditions. This takes time to acquire the encoded image, which becomes a bottleneck in increasing speed. In addition, there is a problem that the encoding is performed in a form in which light is blocked by toner powder, and the efficiency of light utilization decreases according to the amount of toner powder.

In the methods of Non-Patent Document 2 and Non-Patent Document 3, encoding with any pattern is possible by using a DMD. A DMD is a two-dimensional array of micromirrors, and each mirror can be controlled to be either ON or OFF at high speed, but only the light of ON pixels (or OFF pixels) of the encoded light is captured by the camera. Since only one side of the light is captured, the light utilization efficiency is the same as when toner powder is used, and one low-resolution image is captured for each pattern of the DMD, so it takes a long time to capture the low-resolution image. In addition, in Non-Patent Document 1, there are two types of high-resolution information used for super-resolution: [1] toner powder (encoding pattern) finer than the pixel size and [2] the amount of camera position shift, whereas in Non-Patent Document 2 and Non-Patent Document 3, only the encoding pattern of the DMD is used, which is an issue.

Furthermore, even if a DMD is used as in Non-Patent Document 2 or Non-Patent Document 3, if each coding pattern is a random pattern, there is a possibility that the OFF state of a particular mirror will overlap in multiple patterns, as with the pixel completely covered with toner powder in Non-Patent Document 1. In this case, there is also the problem that information about the object to be imaged cannot be obtained in all low-resolution images, and errors (pixels that cannot be restored) occur in the reconstruction of the image, as shown in Figure 19.

Therefore, in consideration of the above problems, the object of the present invention is to provide an imaging device and imaging method that can optically encode a subject image in any pattern, quickly obtain a low-resolution encoded image, and generate a high-resolution image without errors in image reconstruction.

In order to solve the above problems, the imaging device according to the present invention is an imaging device that acquires a low-resolution image and generates a high-resolution image, and is characterized by comprising: an optical modulator that spatially intensity-modulates an image of an object to be imaged based on a coding pattern; a first imaging unit that acquires an image of the object to be imaged that has been coded with the coding pattern as a low-resolution image having a lower resolution than the coding pattern; a second imaging unit that acquires an image of the object to be imaged that has been coded with a pattern that is an inversion of the coding pattern as a low-resolution image having a lower resolution than the coding pattern; and an image generation unit that generates an image of the object to be imaged that has a higher resolution than the low-resolution image by compressed sensing based on at least the low-resolution image, resampling information of the first imaging unit and the second imaging unit, and the coding pattern.

It is also desirable that the imaging device acquires the low-resolution images of the first imaging unit and the second imaging unit in a state in which the relative positional relationship between the encoded image of the object to be imaged and the pixels of the imaging units is different from each other.

Furthermore, it is desirable that in the imaging device, one of the first imaging section and the second imaging section shifts the relative positional relationship between the encoded image of the imaging object and the pixels of the imaging section in units smaller than the pitch of the encoding pattern.

Furthermore, it is desirable that the imaging device controls the relative positional relationship between the encoded image of the object to be imaged and the pixels of the imaging units using a stage attached to at least one of the first imaging unit and the second imaging unit.

It is also desirable that the imaging device controls the relative positional relationship between the encoded image of the imaging object and the pixels of the imaging unit using a stage attached to the optical modulator.

Furthermore, it is desirable that the imaging device controls the relative positional relationship between the encoded image of the imaging object and the pixels of the imaging section by an optical path modulation element disposed between at least one of the first imaging section and the second imaging section and the optical modulator.

It is also preferable that the imaging device converts the high-resolution image into pixel components using a two-dimensional basis function of a discrete cosine transform, a discrete wavelet transform, or a discrete Fourier transform, and the image generating unit generates a high-resolution image of the imaging object by solving a linear equation that estimates the pixel components of the high-resolution image from the low-resolution image using an optimization problem.

In order to solve the above problems, the imaging method according to the present invention is characterized by comprising: an encoding step of spatially intensity-modulating an image of an object to be imaged based on an encoding pattern; a first low-resolution image acquisition step of acquiring an image of the object to be imaged, encoded with the encoding pattern, as a low-resolution image having a resolution lower than that of the encoding pattern; a second low-resolution image acquisition step of acquiring an image of the object to be imaged, encoded with a pattern obtained by inverting the encoding pattern, as a low-resolution image having a resolution lower than that of the encoding pattern; and an image generation step of generating an image of the object to be imaged, having a higher resolution than that of the low-resolution image, by compressed sensing, from at least the low-resolution image, resampling information in the first and second low-resolution image acquisition steps, and the encoding pattern.

Furthermore, it is preferable that the imaging method is such that the first low-resolution image acquisition step and the second low-resolution image acquisition step each acquire the low-resolution image in a state in which the relative positional relationship between the encoded image of the imaging object and the pixels of the imaging unit is different from each other.

The imaging device and imaging method of the present invention can optically encode a subject image in any pattern, quickly obtain a low-resolution encoded image, and generate a high-resolution image without errors in image reconstruction.

1 is a conceptual diagram of an optical system of an imaging apparatus according to a first embodiment. 3 is a diagram illustrating the principle of light modulation by a DMD in the first embodiment. 1A to 1C are diagrams showing examples of a DMD display pattern and projected and captured images of each camera. FIG. 4 is a conceptual diagram of an optical system of an imaging device according to a modified example of the first embodiment. FIG. 11 is a diagram illustrating the principle of light modulation by a DMD in a modified example of the first embodiment. FIG. 2 is a diagram showing a linear equation for estimating a high-resolution image from a low-resolution image. FIG. 1 is a diagram showing a linear equation for estimating a high-resolution image from a low-resolution image summarized as an inverse problem. FIG. 11 is a conceptual diagram of an optical system of an imaging apparatus according to a second embodiment. FIG. 11 is a conceptual diagram of another optical system of the imaging apparatus according to the second embodiment. FIG. 13 is a diagram showing an example in which a relative positional shift is applied between two cameras with respect to an encoded image. FIG. 13 is a diagram illustrating a position shift vector. FIG. 13 is a conceptual diagram of an optical system of an imaging apparatus according to a third embodiment. FIG. 13 is a conceptual diagram of another optical system of the imaging apparatus according to the third embodiment. FIG. 13 is a conceptual diagram of yet another optical system of the imaging apparatus in the third embodiment. FIG. 13 is a diagram showing an example in which the positional relationship between the cameras and the encoded image is changed for each encoding pattern while shifting the relative positions of the two cameras with respect to the encoded image. 13A and 13B are diagrams showing how a high-resolution image is obtained from a low-resolution image to which a relative positional shift with a narrower pitch than the encoding pattern is applied in the fourth embodiment. FIG. 13 is a diagram showing an example of an image reconstructed based on a relative positional shift with a narrower pitch than the encoded pattern. FIG. 13 is a conceptual diagram of an optical system of an imaging apparatus according to a fifth embodiment. 13A and 13B are diagrams illustrating an example in which the information of an imaged object cannot be completely obtained by the coding pattern.

The following describes an embodiment of the present invention with reference to the drawings.

First Embodiment
1 shows a conceptual diagram of the optical system of an image pickup apparatus according to the first embodiment. The image pickup apparatus includes first and second cameras (image pickup elements) 11 and 12, first to third lenses 21 to 23, a DMD 30, and a control and arithmetic processing device 40.

The first lens 21 forms an image of the object to be imaged on the DMD 30.

The DMD 30 is controlled by the control and arithmetic processing device 40 and can display any coding pattern, thereby encoding the image of the object to be imaged. In other words, the DMD 30 functions as an optical modulator that spatially modulates the intensity of the image of the object to be imaged. Figure 2 is a diagram showing the principle of optical modulation by the DMD 30. When a coding pattern is displayed on the DMD 30, light reflected by pixels (mirrors) 31 in the ON state is imaged on the first camera 11, and light reflected by pixels (mirrors) 32 in the OFF state is imaged on the second camera 12.

Figure 3 shows an example of the DMD display pattern and the projected and captured images of each camera. Each pixel (mirror) of the DMD 30 is either ON or OFF, and the image is distributed to the first and second cameras depending on whether it is ON or OFF. Therefore, as shown in Figure 3, the encoded imaged object (encoded image) projected onto the first camera 11 and the encoded image projected onto the second camera 12 are encoded with an inverted pattern (negative-positive inversion). Therefore, the light imaged onto the DMD 30 is projected onto either the first camera 11 or the second camera 12, and no pixels are missing. Note that in this embodiment, a random pattern is used to control the ON/OFF of the DMD, but any pattern may be used, including orthogonal patterns such as the Hadamard pattern.

The second lens 22 projects and forms an image of the image encoded by the positive pattern of the DMD 30 on the first camera 11. The third lens 23 projects and forms an image of the image encoded by the negative pattern of the DMD 30 on the second camera 12. In the present invention, the original image is obtained at super-resolution from the images acquired by the cameras 11 and 12, and the lenses 22 and 23 transfer the encoded image of the object to be captured so that the pixel size of the encoded pattern displayed on the DMD 30 is equal to or smaller than the pixel size of the camera. In this embodiment, a lens is used as an image transfer means for transferring the encoded image of the object to be captured, but other optical elements may be used as a means for transferring the encoded image to the camera and forming an image.

The first camera (first imaging section) 11 captures an image encoded with the positive pattern of the DMD 30, which is transferred by the second lens 22. The second camera (second imaging section) 12 captures an image encoded with the negative pattern of the DMD 30, which is transferred by the third lens 23. It is possible to set as appropriate which of the first camera 11 and the second camera 12 captures the positive pattern and which of the negative patterns, respectively.

As shown in FIG. 3, the pixel size of the camera is twice as large as the pixel size of the DMD 30 (four times larger in area), and if the positional relationship of the first camera 11 to the DMD 30 and the positional relationship of the second camera 12 to the DMD 30 are relatively the same, the sum of the luminance of four pixels of the image encoded by the DMD 30 is input to each pixel of the camera, and low-resolution images can be acquired by the first and second cameras 11 and 12. This is repeated multiple times to acquire multiple encoded images while changing the pattern displayed on the DMD 30. That is, the first camera (first imaging unit) 11 and the second camera (second imaging unit) 12 acquire an image of the object to be imaged, encoded with the encoding pattern, as a low-resolution image with a resolution lower than that of the encoding pattern. Then, the first and second cameras 11 and 12 output the signals of the acquired low-resolution images to the control arithmetic processing device 40.

The control and arithmetic processing unit 40 controls the entire imaging device and performs arithmetic processing for super-resolution. Specifically, the control and arithmetic processing unit 40 controls the pattern of pixels (mirrors) of the DMD 30. The control and arithmetic processing unit 40 also performs arithmetic processing for reconstructing a high-resolution image based on image data of the low-resolution images (encoded images) obtained by the first and second cameras 11 and 12, as described below, and functions as an image generating unit that generates a high-resolution image of the imaging subject.

In this way, in the present invention, even if there is an area (pixels in an OFF state) where information about the imaged object cannot be obtained by encoding using only the first camera 11, the second camera 12 can always obtain information about the imaged object in that area. This prevents the problem of information about the imaged object not being obtained in all low-resolution images and causing errors in reconstruction (see Figure 19) when the number of encoding patterns (or the number of shots) is reduced using compressed sensing. Furthermore, two encoded images (positive and negative) can be obtained simultaneously using one DMD encoding pattern, making it possible to obtain low-resolution images faster than before.

Note that while the object to be imaged in FIG. 1 is shown as a real image, the object to be imaged may be anything that can be imaged by a camera, and may be not only a real image body but also modulated light such as interference fringes. In addition, cameras 11 and 12 and DMD 30 can handle a wide range of wavelengths from ultraviolet to near infrared, so they can also be used in regions other than visible light.

Figure 4 is a conceptual diagram of the optical system of an imaging device according to a modified example of the first embodiment. As in Figure 1, the imaging device includes first and second cameras (imaging elements) 11, 12, first to third lenses 21-23, a DMD 30, and a control and arithmetic processing device 40, but the arrangement of the cameras 11, 12 and the DMD 30 differs from that in Figure 1.

Since the DMD 30 is composed of micromirrors, the optical system may be such that the first camera 11 and the second camera 12 are arranged on the same side (lower side in the figure) of the object to be imaged, as shown in FIG. 4. In this case, the DMD 30 controls the ON/OFF operation of the pixels (mirrors) as shown in FIG. 5, and the image of the object to be imaged is encoded. In this case too, the light imaged on the DMD 30 is projected onto either the first camera 11 or the second camera 12, and no pixels are missing.

Next, the reconstruction of a high-resolution image will be explained. A super-resolved image is reconstructed by performing reconstruction calculations on the acquired encoded image using the control and arithmetic processing device 40. Here, the low-resolution images acquired by the cameras 11 and 12 are two-dimensional images (m, n) [number of pixels m×n], the number of patterns displayed on the DMD 30 is p, and the super-resolved high-resolution image is two-dimensional images (M, N) [number of pixels M×N, M>m, N>n].

In the reconstruction calculation, the optical system is applied to the formula as follows:
(Low-resolution encoded image Y) = (camera resampling R) x (encoding pattern D) x (super-resolved high-resolution image O)
In addition, by DCT,
(Acquired low-resolution encoded image Y) = (camera resampling R) x (encoding pattern D) x (DCT basis B) x (DCT component X of super-resolved image)
can be converted to:

6 is a diagram showing a linear equation for estimating a high-resolution image from a low-resolution encoded image. When an image is converted into a one-dimensional signal (components a _0,0 to a _m-1,n-1 of a two-dimensional image are arranged in a row), the low-resolution encoded image Y becomes a matrix of m×n rows and one column {Y ₁ , ..., Y _m×n }. The super-resolved high-resolution image O is expressed as a product of a DCT base B of M×N rows and M×N columns and a DCT component X of the super-resolved image of M×N rows and one column, {X ₁ , ..., X _M×N }. The encoding pattern D is a one-dimensional signal obtained by arranging all pixels displayed on the DMD on the diagonal of a matrix of M×N rows and M×N columns, with effective pixels (reflecting mirrors) being 1 and ineffective pixels (non-reflecting mirrors) being 0. The process in which an encoded high-resolution image (M×N) is obtained as a low-resolution image (m×n) by coarse pixels of a camera is expressed as a matrix of m×n rows and M×N columns as resampling R by the camera (for example, when 2×2 pixels of an encoded image correspond to one pixel of a camera, four elements of 0.25 are arranged in each of m×n rows). Therefore, the conversion between a low-resolution image and a high-resolution image (DCT components of an image) can be expressed as a linear matrix product as shown in FIG. 6.

This is repeated for p coding patterns to create a simultaneous equation. Since the first and second cameras can each acquire one low-resolution image with one coding pattern, a total of 2p low-resolution coded images Y can be acquired. Similarly, a total of 2p coding patterns, p negatives and p positives, are used for coding pattern D. Therefore, for each coding pattern D={D ₁ , ..., D _2p }, low-resolution coded images Y={Y ₁ , ..., Y _m×n×2p } acquired by each of the first and second cameras 11 and 12 are simultaneously solved. In addition, if the three two-dimensional vectors R, D, and B are grouped together as a super-resolution vector A (=RDB), the optical system can be expressed by a linear equation of Y=AX, and Y and A are known, and the inverse problem is to find X={X ₁ , ..., X _M×N } from these two elements (FIG. 7). Generally, in a simultaneous equation with M×N unknowns (i.e., a simultaneous equation with M×N unknowns), M×N or more conditional expressions are required. However, by using compressed sensing, if the solution X is sparse (many of the components are close to 0), it is possible to obtain the conditional expressions even if the number of unknowns is less than the number of unknowns. In this embodiment, the image is made into a sparse solution X by DCT, so this is applied. In this embodiment, the DCT basis (two-dimensional basis function of discrete cosine transform) is used as an image transform basis for transforming an image into pixel components that are sparse values, but other image transform bases such as DWT basis (two-dimensional basis function of discrete wavelet transform) and DFT basis (two-dimensional basis function of discrete Fourier transform) may be used.

Specifically, in compressed sensing, a solution X is obtained by solving the optimization problem of the following equation (1), called LASSO (Least Absolute Shrinkage and Selection Operator) regression, using the ADMM (Alternating Direction Method of Multipliers) method. There are several algorithms for solving optimization problems, and methods other than ADMM, such as Newton's method, quasi-Newton method, coordinate descent method, ISTA (Iterative Shrinkage-thresholding algorithm), and FISTA (Fast Iterative Shrinkage-thresholding algorithm), may also be used.

By solving this and finally performing inverse DCT on the estimated solution X, a super-resolved high-resolution image can be reconstructed.

Second Embodiment
In the first embodiment, image data is acquired while the positional relationship between each pixel (encoded image) of the DMD 30 and each pixel of the first and second cameras remains constant, but in the second embodiment, the position of one of the cameras is shifted to capture the encoded image. In this embodiment, in addition to the resolution of the encoded pattern, the amount of positional shift can be used as spatial high-resolution information.

Figure 8 is a conceptual diagram of the optical system of an imaging device in the second embodiment. The imaging device comprises first and second cameras (imaging elements) 11, 12, first to third lenses 21 to 23, a DMD 30, a stage 52, and a control and arithmetic processing device 40. Compared to the first embodiment shown in Figure 1, the difference is that the stage 52 is attached to the second camera 12, and the stage 52 is controlled by the control and arithmetic processing device 40. The other configuration is the same as in the first embodiment, so a description will be omitted.

The stage 52 is attached to the second camera 12, and spatially shifts the position of the encoded image incident on the second camera 12 by a specified amount in response to a command from the control and arithmetic processing device 40. Note that although the stage is attached to the second camera 12 in FIG. 8, the stage may be attached to the first camera 11, and the first camera 11 may be shifted.

Figure 9 is a conceptual diagram of another optical system of the imaging device of the second embodiment. The imaging device includes first and second cameras (imaging elements) 11 and 12, first to third lenses 21 to 23, a DMD 30, an optical path modulation element 62, and a control and arithmetic processing device 40.

As shown in FIG. 9, a light path modulation element 62 may be inserted between the DMD 30 and the second camera 12, and the light path may be modulated for each pattern based on a command from the control arithmetic processing device 40 to shift the encoded image, thereby enabling the device to capture encoded images with different spatial resolutions. Here, the light path modulation element 62 can be realized by using an actuator that tilts a transparent material such as glass to shift the light path, or a transmissive liquid crystal modulation element in which a refractive index change occurs due to an electric field. Alternatively, a manual stage equipped with a micrometer or the like may be used without using the control arithmetic processing device 40.

Figure 10 shows an example of relative positional shift between two cameras with respect to the encoded image. The left side shows the positional relationship between the pixels of the first camera 11 and the encoded image, and the right side shows the positional relationship between the pixels of the second camera 12 and the encoded image.

In the first camera 11, the positional relationship is adjusted so that the 16 x 16 pixels of the coding pattern (coded image) fit within the pixels of the first camera 11. Meanwhile, in the second camera 12, one pixel of the second camera 12 corresponds to a 16 x 16 pixel coding pattern in a similar manner, but examples are given of cases in which the position of the coding pattern is shifted by 1, 2, 4, or 8 pixels in each of the vertical and horizontal directions relative to the positional relationship between the first camera 11 and the coded image to capture the coded image.

In an optical system including such a position shift, the linear expression of the optical system used for the reconstruction calculation is given by
(Low-resolution encoded image Y)=(camera resampling R)×(position shift amount S)×(encoding pattern D)×(super-resolved high-resolution image O)
In addition, by DCT,
(Acquired low-resolution encoded image Y) = (Resampling by camera R) x (Position shift amount S) x (Encoding pattern D) x (DCT basis B) x (DCT component X of super-resolved image)
can be converted to:

The vector S indicating the amount of position shift will be briefly explained using FIG. 11. For example, assume that the coding pattern (coded image) is shifted by one pixel each in the X and Y directions (to the lower right). In two dimensions, each pixel (a1, a2, a3, ..., c1, c2, c3, ...) of the coded image is shifted by one pixel each in the X and Y directions after the movement, but when this is expressed as a one-dimensional signal, the row position of each pixel shifts within a column after the movement (see the left diagram in FIG. 11). The position shift vector S that realizes this shift is a matrix in which the element of the column corresponding to the pixel position before the movement in the row corresponding to the row of the pixel position after the movement is 1 (other elements in the same row are 0) (see the right diagram in FIG. 11). Such a position shift vector S can be used as the position shift amount S in the above formula.

As in the case where there is no positional shift, if we use the super-resolution vector A (=RSDB) which is a combination of the vectors R, S, D, and B in the above equation, the optical system can be expressed by a linear equation Y = AX, which becomes an inverse problem of estimating X from Y and A, and by performing inverse DCT on the estimated X, a super-resolved high-resolution image can be reconstructed.

(Verification of the first and second embodiments)
The super-resolution in the first and second embodiments was verified using actual images. An image of (M, N) = (96, 96) was prepared as the original image (= image to be restored by super-resolution), and downsampled to (m, n) = (6, 6). In other words, the super-resolution is 16 times larger vertically and horizontally, and in general super-resolution by pixel shifting, 16 x 16 = 256 low-resolution images are required. In this embodiment, the super-resolution simulation was performed under the condition that the number of patterns p = 128.

Image quality was evaluated using PSNR (Peak Signal To Noise Ratio). PSNR is calculated using the following formulas (2) and (3) using MSE (Mean Squared Error), and the higher the value, the closer the image reconstructed is to the original image. In the formulas below, MAX is the maximum pixel value in original image I, u and v are the number of pixels in the vertical and horizontal directions of original image I and super-resolution image I', respectively, and i and j indicate any pixel position within the image.

A random coding pattern of 96 x 96 pixels was prepared, and an image was generated from low-resolution images acquired with 128 different patterns. First, an image was reconstructed from 128 low-resolution images using only the first camera 11, resulting in a PSNR of 25.8 dB.

Next, super-resolution was performed from a total of 256 low-resolution images acquired by the first camera 11 and the second camera 12. In this case, the positional relationship of the first camera 11 was adjusted so that the 16 × 16 pixels of the coding pattern fit within the pixels of the first camera 11, while the second camera 12 shifted the position of the coding pattern by 0, 1, 2, 4, and 8 pixels in each of the vertical and horizontal directions to capture the coding image (see FIG. 10).
The results are shown in Table 1.

As the shift amount increases, the image quality improves. When there is no shift amount in the second camera 12 (0 pixels), the number of encoded images that can be acquired doubles to 256, which slightly improves the image quality, but adding the shift of the second camera 12 in addition to that improves the image quality significantly. This is because the arrangement with the relative positions shifted allows the optical system to encode the positional shift amount as spatial high-resolution information in addition to the resolution of the encoding pattern. In particular, the highest image quality is achieved when the shift amount between the first camera 11 and the second camera 12 is set to 8 pixels of the encoding pattern, which is half the value of the camera pixel. When the positional shift amount is 1 pixel, the high-resolution information due to the positional shift contributes only to the peripheral part of the camera pixel, but when it is 8 pixels, the high-resolution information due to the positional shift can be encoded over the entire camera pixel area.

Third Embodiment
The third embodiment is a case where a position shift for each pattern between the encoded image and the camera is added to the encoding as high resolution information at the time of encoding.

Figure 12 is a conceptual diagram of the optical system of an imaging device in the third embodiment. The imaging device comprises first and second cameras (imaging elements) 11, 12, first to third lenses 21-23, a DMD 30, stages 51, 52, and a control and arithmetic processing device 40. Compared to the first embodiment shown in Figure 1, the difference is that a stage 51 is attached to the first camera 11, and a stage 52 is attached to the second camera 12, and the stages 51, 52 are controlled by the control and arithmetic processing device 40. The other configuration is the same as in the first embodiment, so a description will be omitted.

As shown in FIG. 12, stages 51 and 52 allow the camera pixels to be shifted vertically and horizontally as desired, and the device can capture encoded images with different spatial resolutions (different relative positional relationships) by changing the camera position for each pattern.

Also, FIG. 13 is a conceptual diagram of another optical system of an imaging device in the third embodiment. The imaging device comprises first and second cameras (imaging elements) 11, 12, first to third lenses 21 to 23, a DMD 30, a stage 53, and a control and arithmetic processing device 40. Compared to the first embodiment shown in FIG. 1, the difference is that the stage 53 is attached to the DMD 30, and the stage 53 is controlled by the control and arithmetic processing device 40. The other configuration is the same as in the first embodiment, so a description will be omitted.

As shown in FIG. 13, the stage 53 allows the DMD (optical modulator) to be shifted to any position, and by changing the position of the DMD 30 for each pattern, the coded images captured by each camera can be captured with different spatial resolutions.

Figure 14 is a conceptual diagram of yet another optical system of an imaging device in the third embodiment. The imaging device includes first and second cameras (imaging elements) 11, 12, first to third lenses 21-23, a DMD 30, light path modulation elements 61, 62, and a control and arithmetic processing device 40. Compared to the first embodiment shown in Figure 1, the difference is that the light path modulation element 61 is placed between the DMD 30 and the first camera 11, and the light path modulation element 62 is placed between the DMD 30 and the second camera 12, and the light path modulation elements 61, 62 are controlled by the control and arithmetic processing device 40. The other configurations are the same as those of the first embodiment, so explanations will be omitted.

As shown in FIG. 14, optical path modulation elements 61 and 62 may be inserted to modulate the optical path from the DMD 30 to the cameras 11 and 12 for each pattern, thereby shifting the encoded image, thereby making it possible to create a device that can capture encoded images with different spatial resolutions.

In response to a command from the control arithmetic processing device 40, the position of each stage 51 to 53 or the light path modulation elements 61, 62 is shifted by a specified amount. Using this position shift amount, an optimization problem that linearly represents the optical system is set in the same way as in the second embodiment, and an inverse problem is solved. Then, a high-resolution image can be reconstructed by performing inverse DCT on the estimated solution.

In this embodiment, the stage and other components are controlled by the control and arithmetic processing device 40, but a manual stage equipped with a micrometer or the like may be used without using the control and arithmetic processing device 40.

(Verification of the third embodiment)
The third embodiment, in which the position is shifted for each encoding pattern using an optical path modulation element, was verified by simulation in the same manner as the first and second embodiments. Super-resolution was performed 16 times vertically and horizontally in the same manner as the first and second embodiments. When acquiring a low-resolution image using 128 encoding patterns, the first camera 11 and the second camera 12 are shifted to different positions for each pattern to capture the image. The spatial resolution of the acquired encoded image changes depending on the pitch of the position shift.

Figure 15 is a diagram showing an example in which the positional relationship between the camera and the encoded image is changed for each encoding pattern while a certain relative positional shift is applied between the two cameras with respect to the encoded image. As shown in Figure 15, in the state before the position shift (i.e., the first encoding pattern), the first camera 11 is arranged so that the 16x16 pixels of the encoding pattern are in the same position as the 1x1 camera pixels, whereas the second camera 12 is arranged so that the encoding pattern is shifted by one pixel both vertically and horizontally. In addition, when acquiring low-resolution images by changing the positional relationship between the camera and the encoded image for each encoding pattern, the amount of relative positional shift between the first camera 11 and the second camera 12 with respect to the encoded image is always the same (always one pixel shift vertically and horizontally in Figure 15).

A random coding pattern of 96 x 96 pixels was prepared, and an image was generated from low-resolution images acquired using 128 different patterns. First, an image was reconstructed from 128 low-resolution images using only the first camera 11, resulting in a PSNR of 25.0 dB.

In contrast, we verified the case in which the shift pitch (the relative positional deviation between the first and second cameras) was set to 0, 1, 2, 4, or 8 pixels of the encoding pattern, and the first camera 11 and the second camera 12 were used simultaneously while randomly moving the position for each encoding pattern to obtain a total of 256 encoded images.
The results are shown in Table 2.

As shown in Table 2, the image quality of the reconstructed image improved as the relative positional shift between the first and second cameras increased.

In this embodiment, the coding pattern may be fixed (the pattern of the DMD 30 may be fixed) as in Non-Patent Document 1 using toner powder, and 128 low-resolution images may be acquired by shifting the position. A high-resolution image can be reconstructed from the low-resolution images using information based on the amount of position shift.

(Fourth embodiment)
In the first to third embodiments, the encoding pattern is shifted in pixel units, so that the resolution of the reconstructed image is equivalent to the resolution of the encoding pattern. However, if the relative positional shift amount between the two cameras is made finer than the encoding pattern, resolution information with a higher frequency than the encoding pattern is encoded, and the resolution can be increased to be higher than the encoding pattern. In the fourth embodiment, the relative positional shift amount between the first camera 11 and the second camera 12 is made smaller (narrower) than the pixel pitch of the encoding pattern.

The following simulation was performed to confirm that it is possible to achieve a higher resolution than the coding pattern. The resolution of the random coding pattern with 96 x 96 pixels was reduced to 48 x 48 symbols, with 2 x 2 pixels per symbol (same display state). First, when this coded image is acquired using only the first camera 11 and reconstructed, although the reconstructed image has 96 x 96 pixels, the effective image resolution is 48 x 48 (same pixel value in 2 x 2 pixel units).

Next, two cameras were used, and as shown in Figure 16, the second camera 12 shifted the position of the encoded image by applying a shift amount (relative positional deviation) to the first camera 11 relative to the position of the encoded image in units smaller than the pitch of the encoding pattern, thereby obtaining a low-resolution image.

In this embodiment, a coding pattern with a resolution of 48x48 symbols was used, with 2x2 pixels of the DMD as one symbol, and the first camera 11 was positioned so that an 8x8 symbol coding pattern fits within one pixel of the camera, while the second camera 12 was positioned so that the coding pattern was shifted by one DMD pixel (1/2 pitch of the coding pattern) relative to the camera, and verification was performed with 128 coding patterns.

Figure 17 shows an example of an image reconstructed based on a relative positional shift with a narrower pitch than the coding pattern. (a) shows a case where a low-resolution coding pattern is used and reconstruction is performed without a relative positional shift with respect to the coding image in the first camera 11 and the second camera 12, and (b) shows a case where a low-resolution coding pattern is used and reconstruction is performed by shifting the coding image relative to the coding image by 1/2 the pitch of the coding pattern in the second camera 12. Both reconstructed images are reconstructed with a resolution of 96 x 96 pixels. As shown in Figure 17(a), when there is no positional shift with respect to the coding image in the first camera 11 and the second camera 12, the reconstructed image cannot have a resolution higher than the resolution contained in the coding image (48 x 48 symbols), and it can be seen that even if the number of pixels is (96 x 96), the effective resolution is reduced. On the other hand, referring to Figure 17(b), it can be seen that an image with a resolution higher than the coding pattern can be reconstructed by shifting the positions of the first camera 11 and the second camera 12 relatively in units smaller than the pixel pitch of the coding pattern.

Note that this embodiment is an example in which a relative position shift that is finer than the coding pattern is applied to the second embodiment, but it may also be applied to a system in which the camera position is shifted for each coding pattern, such as the third embodiment.

Fifth Embodiment
18 shows a conceptual diagram of the optical system of the imaging device of the fifth embodiment. In the first to fourth embodiments, a DMD is used as the optical modulator, but in this embodiment, a reflective liquid crystal optical modulator is used. The imaging device includes first and second cameras (imaging elements) 11 and 12, first to third lenses 21 to 23, a PBS (Polarizing Beam Splitter) 81, a BS (Beam Splitter) 82, a reflective liquid crystal optical modulator 90, and a control and arithmetic processing device 40.

The first lens 21 transmits light (image) reflected (or emitted) from the object to be imaged. Half of the light is reflected downward in the figure by the PBS 82, but the other half travels toward the PBS 81. Of that light, the S-polarized light is reflected downward in the figure by the PBS 81, and only the P-polarized light passes through the PBS 81 and forms an image on the reflective liquid crystal light modulator 90.

The reflective liquid crystal light modulator 90 is controlled by the control and arithmetic processing device 40 and displays an arbitrary coding pattern. The P-polarized incident light is modulated to S-polarized light when it is irradiated to the ON part of the coding pattern, while the light irradiated to the OFF part of the coding pattern is reflected as P-polarized light. In other words, the reflective liquid crystal light modulator 90 is a light modulator that spatially modulates the intensity of the image of the object being imaged.

The PBS 81 transmits P-polarized light and reflects S-polarized light from the reflective liquid crystal light modulator 90 in the upward direction in the figure, where it passes through the second lens 22 and heads toward the first camera 11. That is, of the P-polarized incident light, the light that is irradiated onto the ON portion of the coding pattern of the reflective liquid crystal light modulator 90 is modulated to S-polarized light and reflected by the PBS 81.

The P polarized light reflected by the reflective liquid crystal light modulator 90 passes through the PBS 81 and is separated into two by the BS 82. One light passes through the BS 82, while the other light is reflected by the BS 82 in the upward direction in the figure, passes through the third lens 23 and heads toward the second camera 12. In other words, of the P polarized incident light, the light irradiated onto the OFF part of the coding pattern of the reflective liquid crystal light modulator 90 is reflected while remaining P polarized, so it passes through the PBS 81 and is reflected by the BS 82.

The first camera 11 captures an image reflected by the PBS 81. That is, the first camera 11 can capture an encoded image (positive image) that is irradiated to and reflected from the ON parts of the encoding pattern. The second camera 12 captures an image reflected by the BS 82. That is, the second camera 12 can capture an encoded image (negative image) that is irradiated to and reflected from the OFF parts of the encoding pattern. The first and second cameras 11, 12 each output a signal of the acquired low-resolution image to the control arithmetic processing device 40.

The control and arithmetic processing device 40 controls the entire imaging device and performs arithmetic processing for super-resolution. Specifically, the control and arithmetic processing device 40 controls the pixel pattern of the reflective liquid crystal light modulator 90. In addition, the control and arithmetic processing device 40 performs arithmetic processing to reconstruct a high-resolution image based on image data of a low-resolution image (encoded image) obtained by the first and second cameras 11 and 12.

In this embodiment, when shifting the coding pattern, a stage can be attached to the first camera 11, the second camera 12, or the reflective liquid crystal light modulator 90, as in the second to fourth embodiments.

In each of the above embodiments, the configuration and operation of the imaging device have been described, but the present invention is not limited thereto, and may be configured as an imaging method for generating a high-resolution image. That is, the present invention may be configured as an imaging method including an encoding step of spatially intensity-modulating an image of an object to be imaged based on an encoding pattern, a low-resolution image acquisition step of acquiring an image of the object to be imaged, encoded with the positive and negative patterns of the encoding pattern, as a low-resolution image having a lower resolution than the encoding pattern, and an image generation step of generating an image of the object to be imaged with a higher resolution than the low-resolution image by compressed sensing.

The above-mentioned embodiments have been described as representative examples, but it will be apparent to those skilled in the art that many modifications and substitutions are possible within the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited by the above-mentioned embodiments, and various modifications or changes are possible without departing from the scope of the claims. For example, the functions contained in each block, step, etc. described in the embodiments can be rearranged so as not to cause logical inconsistencies, and multiple constituent blocks, steps, etc. can be combined into one or divided.

11 First camera 12 Second camera 21 First lens 22 Second lens 23 Third lens 30 DMD
40 Control arithmetic processing device 51 to 53 Stage 61, 62 Light path modulation element 81 PBS
82 B.S.
90 Reflective liquid crystal light modulator

Claims (9)

1. An imaging device for acquiring a low-resolution image and generating a high-resolution image, comprising:
an optical modulator that spatially intensity-modulates an image of an object based on a coding pattern;
a first imaging unit that acquires an image of the imaging object coded with the coding pattern as a low-resolution image having a resolution lower than that of the coding pattern;
a second imaging unit that acquires an image of the object to be imaged, the image being coded with a pattern obtained by inverting the coding pattern, as a low-resolution image having a resolution lower than that of the coding pattern;
an image generating unit that generates an image of the image capture object having a higher resolution than the low-resolution image by compressed sensing based on at least the low-resolution image, resampling information of the first imaging unit and the second imaging unit, and the coding pattern;
An imaging device comprising: 2. The imaging device according to claim 1,
An imaging device, characterized in that the first imaging unit and the second imaging unit each acquire the low-resolution image in a state in which the relative positional relationship between the encoded image of the object to be imaged and the pixels of the imaging units is different from each other. 3. The imaging device according to claim 2,
An imaging device, characterized in that one of the first imaging unit and the second imaging unit shifts the relative positional relationship between the encoded image of the object to be imaged and the pixels of the imaging unit in units smaller than the pitch of the encoding pattern. 4. The imaging device according to claim 1,
An imaging device, characterized in that a relative positional relationship between an encoded image of the object to be imaged and pixels of the imaging units is controlled by a stage attached to at least one of the first imaging unit and the second imaging unit. 4. The imaging device according to claim 1,
An imaging apparatus, characterized in that a relative positional relationship between the encoded image of the object to be imaged and pixels of an imaging section is controlled by a stage attached to the optical modulator. 4. The imaging device according to claim 1,
An imaging device, characterized in that a relative positional relationship between an encoded image of the object to be imaged and pixels of the imaging unit is controlled by an optical path modulation element arranged between at least one of the first imaging unit and the second imaging unit and the optical modulator. 7. The imaging device according to claim 1,
The high-resolution image is transformed into pixel components by a two-dimensional basis function of a discrete cosine transform, a discrete wavelet transform, or a discrete Fourier transform;
The imaging device, characterized in that the image generation unit generates a high-resolution image of the object to be imaged by solving a linear equation that estimates pixel components of the high-resolution image from the low-resolution image using an optimization problem. an encoding step of spatially intensity modulating an image of an object based on a coding pattern;
a first low-resolution image acquisition step of acquiring an image of the imaging object coded with the coding pattern as a low-resolution image having a resolution lower than that of the coding pattern;
a second low-resolution image acquisition step of acquiring an image of the object coded with a pattern obtained by inverting the coding pattern as a low-resolution image having a resolution lower than that of the coding pattern;
an image generating step of generating an image of the image capture object having a higher resolution than the low-resolution image by compressed sensing from at least the low-resolution image, resampling information in the first and second low-resolution image acquisition steps, and the coding pattern;
An imaging method comprising: 9. The imaging method according to claim 8,
an imaging method, characterized in that the first low-resolution image acquisition step and the second low-resolution image acquisition step each acquire the low-resolution image in a state in which the relative positional relationship between the encoded image of the object to be imaged and the pixels of the imaging unit is different from each other.

Images