JP2013055483A - Image processor and program - Google Patents

Abstract

To suppress block noise as needed.
An image processing apparatus includes: a code amount prediction unit that predicts a code amount generated from a plurality of images constituting a moving image; and an image having a code amount that exceeds a predetermined reference value among the plurality of images. A smoothing processing unit that applies a smoothing process, and a compression unit that compresses a moving image including an image to which the smoothing process is applied. The program predicts a code amount generated from each of a plurality of images constituting the moving image, and applies a smoothing process to an image of which the code amount exceeds a predetermined reference value among the plurality of images. And causing the computer to execute a step of compressing the moving image including the image to which the smoothing process is applied.
[Selection] Figure 2

Description

  The present invention relates to an image processing apparatus and a program.

The adjacent two image groups including the resolution change point of the image signal are maintained at a constant number of screens constituting the two image groups, and the resolutions of the screens constituting the two image groups are maintained. There is known a technique for encoding by changing the code (for example, see Patent Document 1).
[Prior art documents]
[Patent Literature]
[Patent Document 1] JP-A-11-313322

  For example, when a high frequency component appears when an image is compressed while there is a certain restriction on the bit rate, block noise may appear prominently. Conventionally, there has been a problem that block noise cannot be appropriately suppressed as necessary.

  In the first aspect of the present invention, the image processing apparatus includes a code amount prediction unit that predicts a code amount generated from each of a plurality of images constituting a moving image, and a code amount of the plurality of images is determined in advance. A smoothing processing unit that applies a smoothing process to an image that exceeds a reference value, and a compression unit that compresses a moving image including the image to which the smoothing process is applied.

  In the second aspect of the present invention, the image processing apparatus includes a code amount prediction unit that predicts a code amount generated from each of a plurality of partial areas in the image, and a code amount of the plurality of partial areas is determined in advance. A smoothing processing unit that applies a smoothing process to a partial region that exceeds the reference value, and a compression unit that compresses an image in which the smoothing process is applied to the partial region.

  In the third aspect of the present invention, the program predicts a code amount generated from each of a plurality of images constituting the moving image, and an image having a code amount exceeding a predetermined reference value among the plurality of images. And causing the computer to execute a step of applying the smoothing process and a step of compressing the moving image including the image to which the smoothing process is applied.

  In the fourth aspect of the present invention, the program predicts a code amount generated from each of a plurality of partial areas in the image, and the code quantity of the plurality of partial areas exceeds a predetermined reference value. A computer is caused to execute a step of applying a smoothing process to the partial area and a step of compressing an image in which the smoothing process is applied to the partial area.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

1 shows an example of a system configuration diagram of an imaging apparatus 100. FIG. 2 shows an example of a functional block configuration that the ASIC 135 has. An example of the process which estimates code amount is shown typically. An example of the reference value is shown in a table format. The example of control of the intensity | strength of a smoothing process is shown typically. A control example of the smoothing process based on the encoding type is schematically shown. An example in which smoothing processing is applied to a partial area in a frame will be schematically shown. 2 shows a processing flow from start to finish of the imaging apparatus 100. An example of an operation flow in moving image shooting is shown. An example of an operation flow in still image shooting is shown. An example of the operation | movement flow in reproduction | regeneration processing is shown.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  FIG. 1 shows an example of a system configuration diagram of the imaging apparatus 100. The imaging apparatus 100 includes an interchangeable lens 120 and a camera body 130. The interchangeable lens 120 includes a lens mount having a lens mount contact 121, and the camera body 130 includes a camera mount having a camera mount contact 131. When the lens mount and the camera mount are engaged and the interchangeable lens 120 and the camera body 130 are integrated, the lens mount contact 121 and the camera mount contact 131 are connected. The imaging device 100 functions as a single-lens reflex camera in a state where the interchangeable lens 120 and the camera body 130 are integrated.

  The subject light passes through the lens group 122 serving as a photographing optical system along the optical axis and forms an image on the light receiving surface of the image sensor 132. The lens MPU 123 is connected to the camera MPU 133 via the lens mount contact 121 and the camera mount contact 131, and controls the interchangeable lens 120 and the camera body 130 in cooperation with each other while performing communication. For example, the lens MPU 123 cooperates with the camera MPU 133 to adjust the focus of the lens group 122 so as to focus on the subject in at least one focus adjustment region set within the angle of view. For example, the lens MPU 123 controls the focus lens motor to move the focus lens group constituting the lens group 122. Thereby, the focus of the lens group 122 is adjusted.

  The main mirror 150 reflects the subject light flux that has passed through the lens group 122. The main mirror 150 can take an oblique state where the main mirror 150 is obliquely provided in the subject light flux centered on the optical axis of the lens group 122 and a retreat state where the main mirror 150 is retracted from the subject light flux. When the main mirror 150 is in an oblique state, the reflected subject light beam is guided to the optical finder unit 156 and observed by the user. The user can set the composition of shooting through the optical finder unit 156.

  The region near the optical axis of the main mirror 150 in the oblique state is formed as a half mirror, and a part of the incident subject light beam is transmitted therethrough. The transmitted subject luminous flux is reflected by the sub mirror 151 and guided to the focus sensor 142. The focus sensor 142 has a plurality of photoelectric conversion element arrays that receive a subject light flux and output a phase difference signal. The focus sensor 142 can detect a focus state, a front pin state, and a rear pin state in each of a plurality of focus adjustment regions provided corresponding to a specific region of the subject image. The focus sensor 142 is configured to detect a deviation amount from the focus state in the front pin state and the rear pin state. The camera MPU 133 can detect the distance to the subject based on the amount of deviation from the focused state. The focus state of the lens group 122 is specified by performing a correlation operation on the phase difference signal by processing of the camera MPU 133 or the like. The target position of the focus lens included in the lens group 122 is determined based on the focus state by the control of the camera MPU 133 or the like, and the position of the focus lens is controlled toward the determined target position by the control of the camera MPU 133 or the like. As described above, the lens group 122 is adjusted in focus by detecting the focus state by the phase difference detection method. That is, when the main mirror 150 is down and in an oblique state, the focus state is detected by the phase difference detection method using the output from the focus sensor 142 to adjust the focus.

  When the main mirror 150 is retracted from the subject light beam, the sub mirror 151 is retracted from the subject light beam in conjunction with the main mirror 150. When the main mirror 150 is in the retracted state, the subject light flux that has passed through the lens group 122 is incident on the light receiving surface of the image sensor 132. The image sensor 132 captures an image using the subject light flux that has passed through the lens group 122. The imaging element 132 functions as an imaging unit.

  The imaging element 132 has a photoelectric conversion element group for imaging formed on the light receiving surface. The imaging photoelectric conversion element group receives the subject image substantially formed on the light receiving surface by the lens group 122. As the imaging device 132, for example, a solid-state imaging device such as a CMOS sensor or a CCD sensor can be applied. The imaging device 132 supplies an analog signal to the A / D converter 134 using a signal of accumulated charges generated by receiving the subject light beam as an image signal. The A / D converter 134 converts the analog image signal output from the image sensor 132 into a digital image signal and outputs it as image data. The image sensor 132 is driven by the drive unit 140 that has received an instruction from the camera MPU 133.

  The image data output from the A / D converter 134 is stored in the SDRAM 136. At least a part of the memory area of the SDRAM 136 is used as a buffer area for temporarily storing image data generated by imaging. When the image sensor 132 continuously captures an image with a subject light beam, the image data sequentially generated by the imaging is sequentially stored in the SDRAM 136. For example, image data of a plurality of still images obtained by continuously capturing still images with subject light flux is sequentially stored in the SDRAM 136. Further, the image data of a plurality of images constituting the moving image obtained by capturing the moving image with the subject luminous flux is sequentially stored in the SDRAM 136. Note that image data of an image constituting a moving image may be referred to as a frame. The SDRAM 136 functions as a frame memory.

  The ASIC 135 sequentially processes the image data stored in the SDRAM 136. The ASIC 135 is an integrated circuit in which circuits related to the image processing function are integrated into one, and functions as a part of the image processing apparatus. The ASIC 135 performs image processing by using at least a part of the memory area of the SDRAM 136 as a work area for image processing. For example, the ASIC 135 generates image data having a color of each color at each pixel from the discrete image data of the Bayer array output from the image sensor 132 by color interpolation processing. In addition, the ASIC 135 performs predetermined image processing such as color correction, gamma correction, white balance correction, and contour enhancement processing.

  The ASIC 135 converts the image data into image data of a standardized image format and outputs the image data. For example, the ASIC 135 performs image processing for generating still image data obtained by encoding still image data in an encoding format conforming to a standard such as JPEG. In addition, the ASIC 135 converts a plurality of frames into H.264. H.264, MPEG2, and Motion Jpeg are used to perform image processing for generating moving image data encoded by an encoding method compliant with standards such as Motion JPEG. The camera MPU 133 transfers image data such as still image data and moving image data generated by the ASIC 135 to an external memory 160 as an example of a nonvolatile recording medium for recording. As the external memory 160, for example, a semiconductor memory such as a flash memory can be exemplified. The camera MPU 133 may output image data to a transmission line such as a network in addition to recording image data such as moving image data and still image data in the external memory 160.

  The ASIC 135 generates image data for display in parallel with the image data processed for recording in the external memory 160. The generated image data for display is converted into a display image signal under the control of the display control unit 137 and displayed on the display unit 138 as a display device such as a liquid crystal display. Various menu items related to various settings of the imaging apparatus 100 can be displayed on the display unit 138 under the control of the display control unit 137 with or without displaying an image. The camera MPU 133 controls the display control unit 137 to display menu items on the display unit 138.

  The imaging apparatus 100 is directly or indirectly controlled by the camera MPU 133 including each element in the image processing described above. The system memory 139 is an electrically erasable / storable nonvolatile memory, and is configured by, for example, a flash ROM. The system memory 139 stores control parameters such as constants and variables required when the image capturing apparatus 100 operates, programs, and the like so that they are not lost even when the image capturing apparatus 100 is not operating. The camera MPU 133 develops constants, variables, programs, and the like in the SDRAM 136 as appropriate, and uses them for controlling the imaging apparatus 100. The ASIC 135, SDRAM 136, system memory 139, display control unit 137, camera MPU 133, and external memory 160 in the camera main body 130 are connected to each other by a connection interface 145 such as a bus.

  The operation input unit 141 receives a user operation. The operation input unit 141 includes a power button, a release button, a moving image shooting button, various operation buttons, a touch panel integrated with the display unit 138, and the like. The camera MPU 133 detects that the operation input unit 141 has been operated, and executes an operation corresponding to the operation. For example, the camera MPU 133 controls each unit of the imaging apparatus 100 to execute the release operation when the release button is operated. For example, when the imaging mode is set to the continuous shooting mode, the camera MPU 133 controls each unit of the imaging device 100 to execute continuous shooting as an example of continuous shooting. Further, the camera MPU 133 controls each unit of the imaging apparatus 100 to execute moving image shooting as an example of continuous shooting when the moving image shooting button is operated. In addition, the camera MPU 133 controls each unit of the imaging apparatus 100 so that when a touch panel incorporated in the display unit 138 is operated, the camera MPU 133 performs an operation according to the menu item and the operation content displayed on the display unit 138.

  Each element of the camera body 130 and the external memory 160 are supplied with power from the power source 170. Examples of the power source 170 include a secondary battery such as a lithium ion battery that can be attached to and detached from the camera body 130, a system power source, and the like. The camera MPU 133 detects the power supply state of the power supply 170. For example, the camera MPU 133 determines whether or not the power source 170 is a secondary battery. When the camera MPU 133 determines that the power source 170 is a secondary battery, the camera MPU 133 detects the remaining capacity of the secondary battery and controls each part of the imaging device 100 according to the remaining capacity. In addition, the camera MPU 133 controls power supply from the power supply 170 to each unit of the imaging apparatus 100. The secondary battery is an example of a battery, and the battery includes a non-rechargeable battery that cannot be substantially charged.

  FIG. 2 shows an example of a functional block configuration that the ASIC 135 has. Here, among the functional blocks of the ASIC 135, functional blocks related to compression processing and functional blocks related to decompression processing are shown. The ASIC 135 includes an input image processing unit 200, a code amount prediction unit 230, a reference value determination unit 240, a compression unit 250, a compression control unit 254, a smoothing information generation unit 252, an expansion unit 260, an expansion control unit 264, and smoothing information acquisition. A unit 262 and an output image processing unit 270. The input image processing unit 200 includes an interpolation processing unit 202, a color processing unit 204, a switching unit 210, a smoothing processing unit 220, and a contour enhancement processing unit 208. The output image processing unit 270 includes a switching unit 280, an enhancement processing unit 290, and an output image generation unit 272.

  The function and operation in the functional block related to the compression process will be described. Each frame constituting the moving image to be compressed is sequentially input from the SDRAM 136 to the input image processing unit 200 and the code amount prediction unit 230 as a compression target frame. In the input image processing unit 200, the interpolation processing unit 202 performs an interpolation process on the frame to generate a frame having a plurality of color signals for each pixel. For example, the interpolation processing unit 202 applies an interpolation process from a frame in which each pixel has a signal of a specific color, and generates a frame having a signal of all colors in each pixel. The color processing unit 204 applies color adjustment processing to the frame supplied from the interpolation processing unit 202. For example, the processing in the color processing unit 204 can be exemplified by linear matrix processing for correcting the spectral characteristics of the image sensor 132, color balance correction, and the like.

  The switching unit 210 switches the output destination of the frame supplied from the color processing unit 204 to one of the contour enhancement processing unit 208 and the smoothing processing unit 220. Specifically, the switching unit 210 switches the frame output destination based on the code amount predicted by the code amount prediction unit 230.

  The code amount prediction unit 230 predicts the code amount generated from the compression target frame. For example, the code amount prediction unit 230 predicts the code amount based on a difference amount of pixel values from neighboring pixels. In this manner, the code amount prediction unit 230 predicts the code amount generated from each of a plurality of frames constituting the moving image. The code amount prediction unit 230 supplies information indicating the predicted code amount to the switching unit 210. The switching unit 210 outputs a frame to the smoothing processing unit 220 when the predicted code amount exceeds a predetermined reference value. The switching unit 210 outputs the frame to the contour enhancement processing unit 208 when the predicted code amount is equal to or less than a predetermined reference value. In this way, the switching unit 210 functions as an image selection unit that selects a frame whose code amount exceeds the reference value among a plurality of frames. The reference value is determined by the reference value determining unit 240 and supplied to the switching unit 210. As will be described later, the reference value determination unit 240 determines the reference value based on the imaging sensitivity, the number of recording pixels, the bit rate, and the like when the frame is imaged.

  The smoothing processing unit 220 applies a smoothing process to the frame supplied from the switching unit 210. For example, the smoothing processing unit 220 applies a smoothing process to the frame by applying a low-pass filter. As described above, the smoothing processing unit 220 applies the smoothing process to a frame whose code amount exceeds a predetermined reference value among a plurality of frames. As will be described later, the smoothing processing unit 220 may apply a smoothing process to a specific partial region in the frame. As will be described later, the compression unit 250 compresses each block, but the smoothing processing unit 220 executes a smoothing process by applying a low-pass filter having a size larger than the block size of the block. Good. The frame to which the smoothing process is applied in the smoothing processing unit 220 is output to the contour enhancement processing unit 208.

  The contour enhancement processing unit 208 applies contour enhancement processing to the frame. For example, the contour emphasis processing unit 208 performs contour emphasis processing on the frame by adding, to each pixel in the frame, a difference in pixel value from surrounding pixels. The frame to which the contour enhancement processing is applied by the contour enhancement processing unit 208 is output to the compression unit 250.

  The compression unit 250 generates compressed moving image data by performing compression using an encoding method that complies with the moving image encoding standard described above. For example, the compression unit 250 applies intra-frame prediction or inter-frame prediction to an input frame. Also, conversion processing such as DCT conversion, processing such as quantization and variable length coding is applied. The compression unit 250 applies these processes in units of blocks such as macroblocks. The compression control unit 254 controls the operation of the compression unit 250. For example, the compression control unit 254 controls the encoding parameter in the compression unit 250. Adapt the bit rate of the compressed video data to the target bit rate. For example, the compression control unit 254 controls the quantization step value such that the generated code amount of each frame such as the quantization step value is within a predetermined range including the target code amount. Further, the compression control unit 254 controls the timing for transferring the compressed moving image data to the external memory 160 or the like. The moving image data compressed by the compression unit 250 is stored in the SDRAM 136. As described above, the compression unit 250 compresses a moving image including a frame to which the smoothing process is applied.

  The smoothing information generation unit 252 generates smoothing information including specific information for specifying a frame to which the smoothing process is applied. Examples of the specific information include information such as a frame number for specifying a frame to which smoothing processing is applied in a moving image. When the smoothing process is applied to a specific partial region in the frame, the specification information may include information for specifying the partial region to which the smoothing process is applied. Examples of the smoothing information include intensity information indicating the intensity of the smoothing process, filter parameters such as a low-pass filter applied in the smoothing process, and the like.

  The smoothing information generation unit 252 may generate smoothing information that is recorded as tag information that is recorded along with the moving image. The smoothing information is temporarily stored in the SDRAM 136 together with the moving image compressed by the compression unit 250 and transferred to the external memory 160 and recorded under the control of the camera MPU 133. As described above, the camera MPU 133 functions as an output unit that outputs smoothing information in association with the moving image compressed by the compression unit 250. The camera MPU 133 may output a moving image in which smoothing information is associated with an external display device such as an external monitor or an external device such as a network device.

  According to the imaging apparatus 100, a process for reducing the amount of generated code can be applied to the frame by the smoothing process performed by the smoothing process unit 220 and then supplied to the compression unit 250. For this reason, even when the bit rate is limited, the compression unit 250 does not need to significantly increase the quantization step value. For this reason, it can suppress that block noise appears notably.

  The function and operation of the functional block related to the decompression process will be described. The moving image recorded in the external memory 160 is read to the SDRAM 136 at the time of reproduction, for example. The decompression target frames are sequentially input to the decompression unit 260. The smoothing information acquisition unit 262 acquires smoothing information associated with a moving image. The smoothing information acquisition unit 262 acquires the smoothing information attached to the moving image as tag information. That is, the functional blocks of the camera MPU 133 and the smoothing information acquisition unit 262 function as an image acquisition unit that acquires a moving image associated with the smoothing information.

  The decompressing unit 260 decompresses the input frame. For example, the expansion unit 260 applies variable-length decoding, inverse quantization, and inverse transform to the input frame, applies intra-frame prediction or inter-frame prediction, and expands the input frame. The expansion control unit 264 controls the expansion process in the expansion unit 260. For example, the expansion control unit 264 causes the expansion unit 260 to perform inverse quantization using the quantization step value applied in the compression unit 250. Further, the expansion control unit 264 sequentially selects the frames to be expanded and supplies them to the expansion unit 260 sequentially.

  The switching unit 280 switches the output destination of the expanded frame to one of the output image generation unit 272 and the enhancement processing unit 290. Specifically, the switching unit 280 outputs a frame specified from the smoothing information acquired by the smoothing information acquisition unit 262 to the enhancement processing unit 290. The switching unit 280 outputs a frame other than the frame specified from the smoothing information to the output image generation unit 272.

  The enhancement processing unit 290 enhances the high frequency component of the frame. Specifically, high frequency components are emphasized using a high-pass filter. That is, the enhancement processing unit 290 emphasizes the high-frequency component of the frame specified by the smoothing information among the plurality of frames constituting the moving image. When the smoothing process is applied to a specific partial area in the frame, the smoothing processing unit 220 may emphasize the high-frequency component of the partial area specified by the smoothing information. As described above, the enhancement processing unit 290 may emphasize the high-frequency component of the partial area specified by the smoothing information among the plurality of partial areas of the image output in association with the smoothing information. Note that the enhancement processing unit 290 may apply an inverse filter such as a low-pass filter applied in the smoothing processing unit 220. The frame processed by the enhancement processing unit 290 is output to the output image generation unit 272.

  The output image generation unit 272 generates an output image. For example, the output image generation unit 272 generates a display frame to be displayed on the display unit 138. The frame generated by the output image generation unit 272 is temporarily stored in the SDRAM 136 and displayed on the display unit 138 in a predetermined display order and display rate. The output image generation unit 272 may generate a frame that is output to an external monitor or the like in addition to the display unit 138. The output image generation unit 272 may generate a frame that forms a moving image stream to be output to a transmission path such as a network.

  As described above, according to the imaging apparatus 100, in a scene with a large amount of code, it is possible to perform compression by applying a smoothing process to reduce the amount of code. At the time of reproduction, processing for enhancing high-frequency components is applied to the scene, so that the reproduced image is a so-called resolution image. That is, according to the imaging apparatus 100, it is possible to perform compression processing and encoding processing that can provide an image with a sense of resolution while suppressing block noise, even when the bit rate is limited.

  FIG. 3 schematically illustrates an example of a process for predicting the code amount. The code amount prediction unit 230 calculates, for each pixel 310 in the frame 300, a difference in pixel value f from the adjacent pixel 310. The code amount prediction unit 230 calculates the sum of absolute values of the differences of the calculated pixels 310 as an evaluation value of the code amount.

  As an example, as illustrated in the drawing, the code amount prediction unit 230 calculates a difference in pixel value from the pixel 310-1 belonging to the first row as a starting point and the pixel 310-2 on the right side. Similarly, the code amount prediction unit 230 calculates a pixel value difference between the other pixels 310-2 to N-1 and the adjacent pixel 310 on the right. As described above, the code amount prediction unit 230 determines, for each row, the pixel value f (x, y) at the position (x, y) and the pixel value f (x + 1, y) at the right adjacent position (x + 1, y). The difference is calculated. The code amount prediction unit 230 calculates the sum of absolute values of differences in each line. Also, the code amount prediction unit 230 calculates the evaluation value of the code amount by adding the absolute value sums of all the rows. The code amount prediction unit 230 uses the calculated evaluation value of the code amount as a predicted value of the generated code amount.

  Here, the evaluation value of the code amount is calculated from the sum of absolute values of the pixel values. As a method for calculating the evaluation value of the code amount, various methods can be applied in addition to the absolute value sum of the pixel values. For example, the frame 300 may be divided into blocks, the pixel information of each block may be converted into a spatial frequency component by orthogonal transform such as DCT transform, and the code amount may be calculated based on the obtained spatial frequency component. For example, a transform coefficient obtained by orthogonal transform may be encoded, and the code amount may be calculated as the code amount evaluation value. Note that the pixel information to be subjected to orthogonal transformation may be a difference signal from another frame. The pixel information to be subjected to orthogonal transformation may be the pixel value itself of the block. An evaluation value of the code amount is calculated from the inter-frame encoded frame using the difference signal as an object of orthogonal transform, and from the intra-encoded frame, the code value evaluation value of the block pixel value itself as the object of orthogonal transform May be calculated. When the difference signal is orthogonally transformed, the difference signal may be calculated after applying motion compensation, or the difference signal may be calculated without applying motion compensation. In addition, as the code amount evaluation value, the sum of the absolute values of the transform coefficients over a plurality of blocks may be applied. In addition to these, the coding complexity calculated from the pixel values of the frame 300 by various methods may be calculated as the code amount evaluation value.

  Note that block noise generated in the luminance signal is more easily recognized by human eyes than block noise generated in the color difference signal. For this reason, the evaluation value of encoding may be calculated from the luminance value. By calculating the evaluation value of the code amount from the luminance signal, smoothing processing may not be applied if the code amount of the luminance signal is small even if the code amount of the color difference signal is large. it can. For this reason, it is possible to appropriately predict whether or not block noise is easily recognized, and to apply a smoothing process as necessary.

  FIG. 4 shows an example of a reference value for determining whether or not to apply the smoothing process in a table format. This information is stored in the system memory 139. When the imaging apparatus 100 is activated, the information is expanded from the system memory 139 to the SDRAM 136 and used for the ASIC 135. This information is used in the reference value determination unit 240 to determine whether or not to apply a smoothing process to the frame. That is, the system memory 139 and the SDRAM 136 are an example of a reference value storage unit that stores a code amount reference value.

  The code amount reference value L is stored in association with the ISO sensitivity, the number of pixels, and the bit rate. Here, the number of pixels is the number of recorded pixels. For example, it is the number of pixels of the frame to be compressed by the compression unit 250.

  In this example, the code amount reference value L1 is stored in association with the ISO sensitivity 100, the number of pixels N1 × M1, and the bit rate B1. Also, a code amount reference value L3 is stored in association with the ISO sensitivity 200, the number of recorded pixels N1 × M1, and the bit rate B1. Here, it is assumed that L3 <L1. In this way, a smaller code amount reference value is stored in association with higher imaging sensitivity. The reference value determination unit 240 determines a code amount reference value using the code amount reference value stored in association with the imaging sensitivity when the frame is imaged. Specifically, the reference value determination unit 240 determines the code amount reference value stored in association with the imaging sensitivity when the frame is captured as the code amount reference value. That is, the smoothing processing unit 220 applies the smoothing process to a frame in which the code amount evaluation value exceeds the reference value stored in association with the imaging sensitivity when the frame is captured. Therefore, a smaller reference value can be applied as the ISO sensitivity at the time of imaging is higher. In this way, smoothing processing can be reliably applied when imaging is performed with imaging parameters in which high-frequency noise is easily generated and block noise is likely to occur.

  Also, a code amount reference value L2 is stored in association with the ISO sensitivity 100, the number of pixels N2 × M2, and the bit rate B1. Here, N2 × M2 is larger than N1 × M1, and it is assumed that L2 <L1. Thus, a smaller code amount reference value is stored in association with a larger number of pixels. The reference value determination unit 240 determines a reference value using the code amount reference value stored in association with the number of pixels of the image. Specifically, the reference value determination unit 240 determines the code amount reference value stored in association with the number of pixels in the frame as the code amount reference value. For this reason, a smaller reference value can be applied as the number of pixels increases. Therefore, the smoothing process can be reliably applied when recording is performed with a recording parameter in which the code amount tends to be large and block noise is likely to occur.

  Here, the value obtained by dividing the bit rate by the number of pixels is one of index values representing the compression strength in the compression unit 250. That is, a smaller code amount reference value is stored in association with a higher compression strength. For this reason, the reference value determination unit 240 can determine the reference value using the code amount reference value stored in association with the compression strength in the compression unit 250. For this reason, a smaller reference value can be applied as the compressive strength is higher. Therefore, smoothing processing can be reliably applied when compression is performed using compression parameters that are likely to cause block noise.

  A smaller reference value may be stored in association with a higher bit rate. The reference value determination unit 240 may determine the reference value using the code amount reference value stored in association with the bit rate. For this reason, a smaller reference value can be applied as the bit rate is higher. Therefore, smoothing processing can be reliably applied when compression is performed using compression parameters that are likely to cause block noise.

  As described above, the reference value determination unit 240 determines the code amount reference value stored in association with the imaging sensitivity, the number of pixels, and the bit rate when the frame is imaged as the code amount reference value. . Therefore, the smoothing processing unit 220 applies a smoothing process to a frame whose code amount evaluation value exceeds the reference value stored in association with the imaging sensitivity, the number of pixels, and the bit rate when the frame is imaged. To do. Therefore, it is possible to determine whether to apply the smoothing process by applying an appropriate reference value according to the imaging condition, the compression condition, the recording condition, and the like.

  FIG. 5 schematically shows a control example of the strength of the smoothing process. Here, frames 500-0 to 19 are shown together with the magnitude relationship of the evaluation value of the code amount with respect to the reference value L and the strength of the smoothing process. A strength of 0 indicates that no smoothing process is applied.

  In this example, it is assumed that the evaluation value of the code amount calculated from the frames obtained before the frame 500-0 is smaller than the reference value L. Therefore, the smoothing process is not applied to frames before the frame 500-0. Assume that an evaluation value having a code amount exceeding the reference value L is calculated from the frame 500-1. Here, the smoothing processing unit 220 determines the strength 4 as the target smoothing processing strength.

  In this case, the smoothing processing unit 220 determines the strength 1 as the strength of the smoothing processing that is actually applied to the frame 500-1. Then, as shown in this example, when the evaluation value of the code amount exceeding the reference value L is calculated from the subsequent frames 500-2 to 14, the smoothing processing unit 220 determines the strength of the smoothing processing to be applied, Increase gradually until the target smoothing strength is reached. Then, the input image processing unit 200 applies a smoothing process of intensity 4 to the frames 5005 to 15 after the frame 500-5.

  As described above, the smoothing processing unit 220 sets the target value for the frame 500 in which the evaluation value of the code amount exceeding the reference value is calculated next to the frame 500 in which the evaluation value of the code amount equal to or less than the reference value is calculated. Set a value smaller than the strength of the smoothing process. Then, the strength of the smoothing process is gradually increased on condition that the evaluation value of the code amount exceeding the reference value is continuously calculated. For this reason, it can prevent beforehand that the intensity | strength of a smoothing process increases rapidly.

  In this example, an evaluation value having a code amount smaller than the reference value L is calculated from the frame 500 after the frame 500-16. In this case, the smoothing processing unit 220 determines the strength 3 as the strength of the smoothing processing applied to the frame 500-16. Then, the smoothing processing unit 220 gradually decreases the strength of the smoothing processing to be applied in the subsequent frames 500-15 to 18. And the smoothing process part 220 does not apply a smoothing process with respect to the frame 500 after the frame 500-19.

  As described in the example of the control of the strength of the smoothing process in this example, the smoothing processing unit 220 is configured so that the difference in the strength of the smoothing process applied to successive frames is less than a predetermined value. The strength of the smoothing process applied to each successive frame is set. For this reason, it is possible to prevent the strength of the smoothing process from abruptly changing, and to gently change the strength. Moreover, it can suppress that the presence or absence of a smoothing process switches frequently.

  In this example, the intensity of the smoothing process is expressed by a numerical value for the purpose of easy understanding. However, it goes without saying that the strength of the smoothing process need not be controlled numerically. For example, the intensity of the smoothing process can be switched by preparing a plurality of low-pass filters having different degrees of smoothing and switching the applied low-pass filters. Examples of the low-pass filter include various types of filters such as an averaging filter and a Gaussian filter. The strength of the smoothing process can be controlled by filter parameters such as a filter size and a filter coefficient. Examples of the filter parameter include dispersion as one parameter of a Gaussian filter.

  In this example, the intensity of the smoothing process is changed for each frame for the purpose of easy-to-understand explanation. However, the intensity of the smoothing process is changed every predetermined number of frames of two or more frames. May be changed. Further, the smoothing processing unit 220 may determine the target strength of the smoothing processing based on the difference between the code amount evaluation value and the reference value. For example, the smoothing processing unit 220 may determine a larger value as the target smoothing processing strength as the difference between the code amount evaluation value and the reference value is larger. However, a predetermined fixed value may be applied as the target smoothing processing strength regardless of the difference between the code amount evaluation value and the reference value. Further, in this example, the target smoothing process strength is determined from the frame 500-1, but a target value that is the target smoothing process strength may be determined from each frame 500. Then, the smoothing processing unit 220 increases the strength of the smoothing process when the target value is larger than the current smoothing process strength, and when the target value is smaller than the current smoothing process strength, The strength of the smoothing process may be reduced.

  FIG. 6 schematically illustrates a control example of the smoothing process based on the encoding type of each frame. Here, frames 600-1 to 5 and frames 601-1 to 5 are shown together with the magnitude relation of the evaluation value of the code amount with respect to the reference value L and the presence or absence of smoothing processing. Here, it is assumed that the frame 600 and the frame 601 belong to different GOPs. “I”, “B”, and “P” shown in the frame 600 and the frame 601 represent encoding types. “I” is an I frame and represents an intra-coded frame. “B” is a B frame and represents a frame that is inter-frame-coded by bi-directional prediction or bi-prediction. “P” is a P frame and represents a frame that is inter-frame-coded by forward prediction.

  As shown in this example, when the evaluation value of the code amount exceeding the reference value L is calculated from the frame 600-1 as the I frame, the smoothing processing unit 220 applies the smoothing process to the frame 600-1. To do. In this case, the smoothing processing unit 220 also applies smoothing to frames 600-2 to 4-4 that may be interframe-coded with reference to the frame 600-1, regardless of the magnitude of the evaluation value of the code amount. Apply the process. Also, the smoothing processing unit 220 sets the evaluation value of the code amount to the frame 600-5 that may be inter-frame-coded with reference to the frame 600-4 to which the smoothing process is applied. Regardless, smoothing is applied. As described above, when the code amount exceeding the reference value L is calculated from the I frame, the smoothing processing unit 220 also applies the smoothing process to other frames belonging to the GOP including the I frame.

  Further, as shown in this example, when an evaluation value of a code amount exceeding the reference value L is calculated in a frame 601-4 encoded next to a frame 601-1 to which no smoothing process is applied, smoothing is performed. The processing unit 220 applies a smoothing process to the frame 601-4. A smoothing process is applied to the frames 601-2 and 601-3 that may be inter-frame-coded with reference to the frame 601-4 regardless of the evaluation value of the code amount. To do. Also, the smoothing processing unit 220 refers to the frame 601-4 to which the smoothing process is applied, and the frame 601-5 that may be inter-frame-encoded depends on the evaluation value of the code amount. First, smoothing processing is applied. As described above, when the smoothing processing unit 220 applies the smoothing processing to the first frame, the smoothing processing unit 220 performs the smoothing processing on the second frame compressed using the inter-frame correlation with the first frame. Apply. If the difference between the frame to which the smoothing process is applied and the frame to which the smoothing process is not applied is taken, the intensity of the difference signal may increase and the code amount may increase. However, an unnecessary increase in the amount of code can be suppressed by applying the smoothing process to the second frame that is inter-frame encoded with the first frame to which the smoothing process is applied. .

  FIG. 7 schematically illustrates an example in which the smoothing process is applied to a partial region in the frame. In this example, it is assumed that a code amount evaluation value exceeding the reference value L is calculated from the frame 700. In this case, the smoothing processing unit 220 determines whether to apply the smoothing process for each partial region 710 in the frame 700. For example, the code amount prediction unit 230 calculates an evaluation value of the code amount for each partial region 710. The smoothing processing unit 220 applies a smoothing process to the partial region 710 in which the evaluation value of the calculated code amount exceeds a predetermined reference value.

  Note that the same method as the process described in FIG. 3 can be applied as the process of calculating the code amount evaluation value for each partial region 710. For example, by calculating the absolute value sum of the pixel value differences for each partial region, the evaluation value of each code amount of the partial region 710 can be calculated. Further, a reference value serving as a criterion for determining whether or not to apply the smoothing process to the partial area 710 may be determined in advance for each partial area 710. As the reference value for the partial region 710, a value obtained by multiplying the code amount L, which is the determination reference for the entire frame, by a coefficient less than 1 proportional to the area of the partial region 710 may be applied. For example, as the coefficient, a value obtained by dividing the area of the partial region 710 by the area of the entire frame region can be applied. As the area of the partial region, the number of pixels belonging to the partial region can be applied.

  In this example, it is assumed that an evaluation value of a code amount exceeding the reference value is calculated from the partial region 710e. As shown in this example, the smoothing processing unit 220 determines the strength 4 as the strength of the smoothing processing applied to the partial region 710e. And the smoothing process part 220 sets the outer peripheral area | region 720-740 surrounding the partial area 710e. The outer peripheral area 720 is adjacent to the partial area 710 on the inner side. The outer peripheral region 730 is adjacent to the outer peripheral region 720 on the inner side. The outer peripheral region 740 is adjacent to the outer peripheral region 730 on the inner side. The smoothing processing unit 220 applies a smoothing process of strength 3, strength 2 and strength 1 to the outer peripheral regions 720, 730 and 740, respectively. The smoothing processing unit 220 does not apply the smoothing process to the region outside the outer peripheral region 740. In this way, the smoothing processing unit 220 sets the strength of the smoothing process applied to each adjacent partial region so that the difference in the smoothing processing strength between the adjacent partial regions is lower than a predetermined value. To do. Since the difference in the strength of the smoothing process between adjacent partial areas can be reduced, it is possible to prevent the boundary between the partial areas from being noticeable due to the difference in the intensity of the smoothing process.

  In this way, the code amount prediction unit 230 predicts the code amount generated from each of the plurality of partial areas in the frame. Then, the smoothing processing unit 220 applies a smoothing process to a partial region whose code amount exceeds a predetermined reference value among the plurality of partial regions. Then, the compression unit 250 compresses the frame in which the smoothing process is applied to the partial region. In this example, it is determined whether or not to apply the smoothing process to each partial region for the frame 700 in which the evaluation value of the code amount exceeding the reference value L is calculated. However, the code amount prediction unit 230 determines whether or not the code amount evaluation value is calculated from the entire frame, and the code amount evaluation value is a predetermined reference value among the plurality of partial regions in the frame. A smoothing process may be applied to a partial region exceeding.

  The shape and size of the partial area are not limited to those illustrated in this figure. Further, the outer peripheral region adjacent to the partial region is not limited to the one illustrated in this figure. As the partial area and the outer peripheral area, areas having various shapes and sizes can be adopted. As described above, the compression unit 250 compresses the frame for each block, but a region having a larger area than the block may be applied as a partial region to be subjected to the smoothing process. For example, a region having a larger area than the macro block may be applied as a partial region to be smoothed.

  In the above description, the smoothing processing unit 220 picks up the frames constituting the moving image and applies the smoothing processing to the frames when the evaluation value of the code amount exceeds a predetermined value. However, similar processing can be applied to still images. For example, the smoothing processing unit 220 may apply a smoothing process to a still image for which an evaluation value for a code amount exceeding a predetermined value is calculated. In particular, for a still image, the smoothing processing unit 220 may apply a smoothing process to a partial region in which an evaluation value for a code amount exceeding a predetermined value is calculated. That is, the code amount prediction unit 230 predicts the code amount generated from each of the plurality of partial regions in the still image, and the smoothing processing unit 220 determines the code amount in advance among the plurality of partial regions in the still image. A smoothing process may be applied to a partial region exceeding the determined reference value. Then, the compression unit 250 compresses the image in which the smoothing process is applied to the partial area. For this reason, block noise can be suppressed even in a still image.

  FIG. 8 shows a processing flow from the start to the end of the imaging apparatus 100. This flow is started, for example, when a power button as a part of the operation input unit 141 is pressed. This flow is executed by controlling each part of the imaging apparatus 100 mainly by the camera MPU 133.

  In step S800, the camera MPU 133 starts initial setting. For example, the camera MPU 133 develops various parameters for controlling the imaging apparatus 100 from the system memory 139 to the SDRAM 136. Further, the camera MPU 133 sets operating conditions of each unit of the imaging apparatus 100 based on, for example, the state of the imaging mode button as a part of the operation input unit 141 and the developed various parameters. For example, the camera MPU 133 sets the imaging mode according to the position of the imaging mode button. Examples of the imaging mode include a person shooting mode suitable for person shooting, an animal shooting mode suitable for animal shooting, a landscape shooting mode suitable for landscape shooting, and the like. Examples of the operating conditions include imaging modes, imaging conditions such as shutter speed and imaging sensitivity, compression conditions such as a bit rate, and recording conditions such as the number of recording pixels.

  Subsequently, in step S802, the operating condition set in the initial setting is displayed on the display unit 138. For example, the camera MPU 133 causes the display unit 138 to display the set operating conditions in various formats such as icon display.

  Subsequently, in step S804, the camera MPU 133 specifies a user instruction to the operation input unit 141. If the user instruction is an instruction to execute various settings, the camera MPU 133 performs the instructed setting process (step S806). Examples of the setting process include a process for setting an imaging mode, an imaging condition, a compression condition, and a recording condition. The setting process may include a user specifying whether to apply the above-described smoothing process to the frame.

  If it is determined in step S804 that the user instruction is an instruction relating to imaging execution, processing relating to imaging execution is performed (step S812). Examples of instructions related to shooting execution include operations on a live view button, a moving image shooting button, and a release button. If the user instruction in step S804 is an instruction to execute image reproduction, reproduction processing is executed (step S822). Examples of the reproduction process include a process of displaying thumbnail images of images such as still images and moving images recorded in the external memory 160, a process of displaying an image instructed by the user, and the like.

  If there is no user instruction in step S804, a photometric process for measuring ambient light is performed (step S832). For example, the appropriate exposure value is determined based on the photometric result of ambient light. This photometric process is executed at regular time intervals to determine the current appropriate exposure value. Information indicating the current exposure value determined in step S832 is displayed on the display unit 138 in real time and presented to the user.

  When the processing of step S806, step S812, step S822, and step S832 is completed, the process proceeds to step S808, and it is determined whether or not to turn off the power. For example, it is determined that the power is turned off when the power button is pressed again, or when there is no user instruction for a predetermined period after the imaging apparatus 100 starts operating. If it is determined that the power is to be turned off, this flow is terminated. If it is determined that the power is not to be turned off, the process proceeds to step S804.

  FIG. 9 shows an example of an operation flow in moving image shooting. In this example, as a part of the processing in step S812, an operation flow when moving image shooting is instructed is shown. For example, this flow is started when a moving image shooting button is pressed.

  In step S <b> 902, the camera MPU 133 instructs each unit of the imaging apparatus 100 to perform a moving image shooting operation. For example, the main mirror 150 is raised to be in a retracted state from the subject light beam, and rolling reading from the image sensor 132 is started. The read frames are sequentially taken into the SDRAM 136.

  In step S <b> 904, the ASIC 135 reads the compression target frame from the SDRAM 136. In step S906, the reference value determination unit 240 determines a reference value L used to determine whether or not to apply a smoothing process. As described above, the reference value determination unit 240 determines the reference value L based on the imaging sensitivity, the number of recording pixels, and the bit rate. When the imaging sensitivity is automatically adjusted for each frame, the reference value L is appropriately determined according to the imaging sensitivity when each frame is imaged. In step S908, the code amount evaluation value is calculated from the compression target frame.

  In step S910, it is determined whether the compression target frame is an I frame. If the compression target frame is an I frame, it is determined whether the evaluation value of the code amount calculated from the compression target frame exceeds the reference value L (step S912). When the code amount evaluation value exceeds the reference value L, the smoothing processing unit 220 determines the strength of the smoothing process (step S914), and applies the smoothing process to the compression target frame (step S916). Subsequently, the camera MPU 133 temporarily stores smoothing information including specific information for specifying a frame to which the smoothing process is applied in the SDRAM 136 (step S918). As described above, the camera MPU 133 may include information indicating the strength of the applied smoothing process in the smoothing information. Subsequently, in step S920, the compression unit 250 compresses the frame to which the smoothing process is applied.

  If it is determined in step S912 that the code amount evaluation value is less than or equal to the reference value L, it is determined whether or not the strength of the smoothing process applied to the previous frame is greater than a predetermined value (step S912). S932). If the strength of the smoothing process applied to the previous frame is greater than a predetermined value, the process proceeds to step S914. In this case, in step S914, the smoothing processing unit 220 determines the strength of the smoothing process that is one step lower than the strength of the smoothing process applied to the previous frame, for example. As a result, the strength of the smoothing process can be prevented from changing greatly. If the strength of the smoothing process applied to the previous frame is equal to or less than a predetermined value, the smoothing process is not applied and the process proceeds to step S920. For example, when the strength of the smoothing process applied to the previous frame is strength 1, the smoothing process is not applied and the process proceeds to step S920.

  If it is determined in step S910 that the compression target frame is not an I frame, it is determined whether smoothing processing has been applied to a frame that may be referred to in the interframe coding in the compression unit 250 (step S942). When the smoothing process is not applied to a frame that may be referred to, the process proceeds to step S912. When the smoothing process is applied to a frame that may be referred to, the process proceeds to step S914. In this case, in step S914, the smoothing processing unit 220 may determine the same strength as the strength of the smoothing processing applied to a frame that may be referred to. When the smoothing processing strength is gradually increased or gradually decreased, the smoothing processing strength may be determined according to the frame order. For example, in the frame sequence arranged in the order of I, B, B, and P, when the smoothing process strength 1 is applied to the I frame, the smoothing process applied to the P frame is used when the intensity should be gradually increased step by step. You may determine 4 as an intensity | strength.

  Following the processing in step S920, it is determined whether all the compression target frames have been processed (step S922). For example, it is determined whether or not the moving image shooting button has been pressed again to end moving image shooting and all the frames to be compressed have been compressed. If the moving image shooting has not ended, or if all the frames to be compressed have not been processed, the process proceeds to step S904. When the moving image shooting is completed and all the frames to be compressed are processed, the camera MPU 133 records the smoothing information stored in the SDRAM 136 in the external memory 160 as the tag information of the compressed moving image data (step S924). ).

  In this flow, for the purpose of explaining the processing flow in an easy-to-understand manner, it has been described that the smoothing process is applied to the frame in which the evaluation value of the code amount exceeding the reference value L of the entire frame is calculated. However, as described with reference to FIG. 7, the smoothing process may be applied to the partial region in which the evaluation value of the code amount exceeding the predetermined value is calculated. In this case, in step S918, the camera MPU 133 may include information for specifying the partial region to which the smoothing process is applied in the smoothing information.

  FIG. 10 shows an example of an operation flow in still image shooting. In this example, as a part of the processing in step S812, an operation flow when a still image shooting instruction is given is shown. This flow is started, for example, when the release button is pressed.

  In step S1002, the ASIC 135 reads the captured still image. Specifically, when the release button is pressed, the main mirror is raised and still image shooting is performed, and a still image obtained by still image shooting is read from the SDRAM 136 to the ASIC 135.

  In step S1004, the code amount prediction unit 230 calculates an evaluation value of the code amount from a plurality of partial regions. Subsequently, in step S1006, it is determined whether there is a partial region exceeding the reference value. Note that, as described above with reference to FIG. 7 and the like, the reference value of the partial region may be determined according to the reference value L determined based on the imaging sensitivity or the like and the area of the partial region. When there is a partial area exceeding the reference value, the smoothing processing unit 220 determines the strength of the smoothing process for the partial area for each partial area. For example, the smoothing processing unit 220 may determine a large value as the strength of the smoothing process in a partial region having a larger difference from the reference value.

  Subsequently, in step S1010, the smoothing processing unit 220 sets one or more outer peripheral areas around each partial area, and determines the strength of the smoothing process for the one or more outer peripheral areas. For example, as described with reference to FIG. 7 and the like, the smoothing processing unit 220 may determine the strength of the smoothing process for each set outer peripheral region. In step S1012, smoothing processing is applied to each partial region.

  Subsequently, the camera MPU 133 temporarily stores smoothing information including information for specifying the partial region to which the smoothing process is applied in the SDRAM 136 (step S1014). The camera MPU 133 may include the strength of the smoothing process applied to each partial area in the smoothing information. Subsequently, in step S1016, the compression unit 250 compresses the still image to which the smoothing process is applied. For example, encoding conforming to the JPEG standard is applied to a still image to which smoothing processing is applied. Subsequently, in step S1018, the camera MPU 133 records the smoothing information stored in the SDRAM 136 in the external memory 160 as the tag information of the compressed still image data, and ends the process. When a still image is recorded in the Exif format, smoothing information may be included in the Exif information.

  FIG. 11 shows an example of an operation flow in the reproduction process. In this example, as part of the processing in step S822, an operation flow in the case where an instruction to reproduce a moving image is given is shown. This flow is started when a moving image file is selected as an image file to be played back, for example, via the operation input unit 141.

  When this flow is started, the camera MPU 133 reads the moving image file recorded in the external memory 160 to the SDRAM 136 (step S1102), and reads smoothing information attached to the moving image file as tag information to the SDRAM 136 (step S1104). ).

  Subsequently, the decompressing unit 260 selects a decompression target frame (step S1106) and decompresses the frame (step S1108). In step S1110, based on the smoothing information read from the SDRAM 136 by the smoothing information acquisition unit 262, it is determined whether or not the decompression target frame is a frame to which smoothing processing is applied. When the frame to be decompressed is a frame to which smoothing processing has been applied, the decompressed frame is provided to the enhancement processing unit 290 by the switching unit 280, and processing for emphasizing high-frequency components in the enhancement processing unit 290 is applied ( Step S1112). Subsequently, in step S1114, the output image generation unit 272 generates a display frame from the frame to which the enhancement processing is applied. If it is determined in step S1110 that the decompression target frame is not a frame to which smoothing processing is applied, the decompressed frame is provided to the output image generation unit 272 by the switching unit 280, and the process proceeds to step S1114. Transition. The display frame generated in step S1114 is displayed on the display unit 138 in a timely manner, for example, in a predetermined display order.

  In step S1116, it is determined whether the decompression process has been completed for all frames. If the processing has not been completed for all the frames, the process proceeds to step S1106, and another frame to be decompressed is selected. When the decompression process is completed for all frames, this flow ends.

  In this figure, the operation flow in the case of reproducing a moving image has been described. However, similar processing can be applied to still image reproduction processing. For example, the processing in steps S1102 to S1114 may be applied to a still image. For this reason, description of the operation flow when reproducing a still image is omitted.

  In the above description, the processing described as the operation of the camera MPU 133 is realized by the camera MPU 133 controlling each hardware included in the imaging apparatus 100 according to a program. In the above description, the processing realized by the ASIC 135 can be realized by a processor. For example, the processing described as the operation of the ASIC 135 is realized by the processor controlling each hardware included in the imaging apparatus 100 according to the program. That is, the processing described in relation to the imaging apparatus 100 of the present embodiment is performed by the processor operating in accordance with the program to control each hardware, so that each hardware including the processor, the memory, and the like cooperates with the program. It can be realized by operating. That is, the process can be realized by a so-called computer device. The computer device may load a program for controlling the execution of the above-described process, operate according to the read program, and execute the process. The computer device can load the program from a computer-readable recording medium storing the program.

  In the present embodiment, the function and operation of the imaging apparatus 100 as a single-lens reflex camera have been described. As an imaging device, in addition to a single-lens reflex camera, a mirrorless single-lens camera, a compact digital camera, a video camera, an imaging device for broadcasting, a mobile phone with an imaging function, a portable information terminal with an imaging function, a game with an imaging function Various devices having an imaging function, such as entertainment devices such as devices, can be applied. As for the output form, not only image data such as moving images and still images can be recorded on a recording medium, but also devices that output image data to various transmission paths such as a network can be applied. . In addition, not only the imaging apparatus 100 but also an electronic apparatus that performs processing for taking in and compressing image data to be compressed, such as a personal computer, a broadcasting video apparatus, a television apparatus, and a video apparatus. Can do.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. It is not a thing.

DESCRIPTION OF SYMBOLS 100 Imaging device, 120 Interchangeable lens, 121 Lens mount contact, 122 Lens group, 123 Lens MPU, 130 Camera body, 131 Camera mount contact, 132 Image sensor, 133 Camera MPU, 134 A / D converter, 135 ASIC, 136 SDRAM 137 Display control unit, 138 display unit, 139 system memory, 140 drive unit, 141 operation input unit, 142 focus sensor, 145 connection interface, 150 main mirror, 151 sub mirror, 156 optical viewfinder unit, 160 external memory, 170 power supply 200 input image processing unit 202 interpolation processing unit 204 color processing unit 210 switching unit 220 smoothing processing unit 208 contour enhancement processing unit 230 code amount prediction unit 240 reference value determination unit 250 compression unit 25 Smoothing information generation unit, 254 compression control unit, 264 expansion control unit, 260 expansion unit, 262 smoothing information acquisition unit, 270 output image processing unit, 272 output image generation unit, 280 switching unit, 290 enhancement processing unit, 300, 500, 600, 601, 700 frames, 310 pixels, 710 partial area, 720, 730, 740 outer peripheral area

Claims (18)

A code amount predicting unit that predicts a code amount generated from each of a plurality of images constituting the moving image;
A smoothing processing unit that applies a smoothing process to an image in which the code amount exceeds a predetermined reference value among the plurality of images;
A compression unit that compresses a moving image including an image to which the smoothing process is applied;
An image processing apparatus comprising: The image processing apparatus according to claim 1, further comprising: an output unit that outputs specific information for specifying an image to which the smoothing process is applied in association with the moving image compressed by the compression unit. An image acquisition unit for acquiring the moving image output in association with the specific information;
An emphasis processing unit for emphasizing high-frequency components of an image specified by the specifying information among a plurality of images constituting the moving image;
The image processing apparatus according to claim 2, further comprising: A reference value storage unit that stores a smaller code amount reference value in association with higher imaging sensitivity;
A reference value determination unit that determines the reference value using a code amount reference value stored in the reference value storage unit in association with an imaging sensitivity when the image is captured;
The image processing apparatus according to claim 1, wherein the smoothing processing unit applies a smoothing process to an image in which the code amount exceeds the determined reference value. The reference value storage unit further stores a smaller code amount reference value in association with a larger number of pixels,
The image processing apparatus according to claim 4, wherein the reference value determination unit determines the reference value using a code amount reference value stored in the reference value storage unit in association with the number of pixels of the image. The reference value storage unit further stores a smaller code amount reference value in association with a higher compression strength,
The image processing apparatus according to claim 4, wherein the reference value is determined using a code amount reference value stored in the reference value storage unit in association with a compression strength in the compression unit. The said smoothing process part sets the intensity | strength of the said smoothing process applied to a continuous image so that the difference of the intensity | strength of the said smoothing process applied to a continuous image may be less than a predetermined value. The image processing apparatus according to any one of 1 to 6. When the smoothing process is applied to the first image, the smoothing processing unit applies the smoothing process to the second image that is compressed using the inter-image correlation with the first image. The image processing apparatus according to claim 1. The code amount prediction unit predicts a code amount generated from each of a plurality of partial regions in the image,
The image processing apparatus according to any one of claims 1 to 8, wherein the smoothing processing unit applies a smoothing process with stronger intensity to a partial region having a larger code amount predicted by the code amount prediction unit. . The smoothing processing unit sets the strength of the smoothing process to be applied to each adjacent partial region so that a difference in strength of the smoothing processing between adjacent partial regions is less than a predetermined value. Item 10. The image processing apparatus according to Item 9. The compression unit compresses an image for each of a plurality of blocks,
The image processing apparatus according to claim 1, wherein the smoothing processing unit executes a smoothing process by applying a low-pass filter having a size larger than a block size of the plurality of blocks. The image processing apparatus according to claim 1, wherein the code amount prediction unit predicts the code amount based on a difference amount of pixel values from neighboring pixels. A code amount prediction unit that predicts a code amount generated from each of a plurality of partial regions in the image;
A smoothing processing unit that applies a smoothing process to a partial region in which the code amount exceeds a predetermined reference value among the plurality of partial regions;
A compression unit that compresses the image in which the smoothing process is applied to the partial region;
An image processing apparatus comprising: The smoothing processing unit sets the strength of the smoothing process to be applied to each adjacent partial region so that a difference in strength of the smoothing processing between adjacent partial regions is less than a predetermined value. Item 14. The image processing apparatus according to Item 13. The image processing apparatus according to claim 13, further comprising: an output unit that outputs specific information for specifying the partial region to which the smoothing process is applied in association with the image compressed by the compression unit. An image acquisition unit for acquiring an image output in association with the specific information;
An emphasis processing unit that emphasizes a high-frequency component of a partial region specified by the specific information among a plurality of partial regions of the image acquired by the image acquisition unit;
The image processing apparatus according to claim 15, further comprising: Predicting the amount of code generated from each of a plurality of images constituting a video;
Applying a smoothing process to an image in which the code amount exceeds a predetermined reference value among the plurality of images;
Compressing a moving image including an image to which the smoothing process is applied;
A program that causes a computer to execute. Predicting the amount of code generated from each of a plurality of partial areas in the image;
Applying a smoothing process to a partial region of the plurality of partial regions where the code amount exceeds a predetermined reference value;
Compressing the image in which the smoothing process is applied to the partial area;
A program that causes a computer to execute.

Images