JP3907816B2 - Image processing method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention belongs to the technical field of image processing using multistage focus images.
[0002]
[Prior art]
Currently, printing of images taken on photographic films (hereinafter referred to as films) such as negative films and reversal films onto photosensitive materials (printing paper) is performed by projecting the film images onto photosensitive materials and exposing the photosensitive materials to surface exposure. The so-called direct exposure (analog exposure) is the mainstream.
[0003]
On the other hand, in recent years, a printing apparatus using digital exposure, that is, an image recorded on a film is photoelectrically read, and the read image is converted into a digital signal. A digital photo printer that uses image data, scans and exposes a photosensitive material with recording light modulated in accordance with the image data, records an image (latent image), and produces (finished) print has been put to practical use.
[0004]
In digital photo printers, the exposure conditions during printing can be determined by processing the image data as digital image data, so correction of image skipping and blurring caused by backlighting, flash photography, etc., sharpness (sharpening) ) Processing and the like can be suitably performed to obtain a high-quality print that could not be obtained by conventional direct exposure. In addition, image composition and division, character composition, and the like can be performed by image data processing, and prints that are freely edited / processed according to applications can be output.
Moreover, according to the digital photo printer, not only the image can be output as a print (photograph) but also the image data can be supplied to a computer or stored in a recording medium such as a floppy disk. Can be used for various purposes other than photography.
Furthermore, according to the digital photo printer, in addition to the image photographed on the film, an image (image data) photographed by an imaging device such as a digital camera or a digital video camera can be output as a print.
[0005]
Such a digital photo printer basically has a scanner (image reading device) that photoelectrically reads an image recorded on a film, and performs image processing on the read image to obtain output image data (exposure conditions). An image input device having an image processing device, a printer (image recording device) that scans and exposes a photosensitive material in accordance with image data output from the image input device and records a latent image, and develops the exposed photosensitive material And an image output device having a processor (developing device) for printing.
[0006]
In a scanner, reading light emitted from a light source is incident on a film to obtain projection light carrying an image photographed on the film, and this projection light is imaged on an image sensor such as a CCD sensor by an imaging lens. Then, the image is read by photoelectric conversion, subjected to various image processing as required, and then sent to the image processing apparatus as film image data (image data signal).
The image processing apparatus performs predetermined image processing on image data read by a scanner or image data supplied from a digital camera or the like, and sends it to a printer as output image data (exposure conditions) for image recording.
In a printer, for example, if the apparatus uses light beam scanning exposure, the light beam is modulated in accordance with image data sent from the image processing apparatus, and the light beam is deflected in the main scanning direction. The photosensitive material is conveyed in a sub-scanning direction perpendicular to the direction, and the photosensitive material is exposed (baked) by a light beam carrying an image to form a latent image, and then development processing or the like according to the photosensitive material in the processor And a print (photograph) in which an image photographed on the film is reproduced.
[0007]
[Problems to be solved by the invention]
By the way, when shooting a scene optically, it is impossible to focus on all objects (subjects) present in the scene, and depending on the distance from the photographer (camera), for example, The person is in focus but the background building is not in focus, or the background is in focus but the person in front is not in focus Occurs.
[0008]
An object of the present invention is to provide an image processing method capable of extracting a main part of an image in which erroneous extraction is prevented by using a multistage focus image obtained by photographing the same scene at different focal lengths and digital image processing. It is in.
[0009]
[Means for Solving the Problems]
In order to achieve the above object, an image processing method according to the present invention obtains image data of a plurality of images (multistage focus images) obtained by photographing the scene at different focal lengths, and focuses each image. The subject is extracted, the main part is extracted by paying attention to each of these focused subjects, and the consistency of the extraction result of the main part in each image in the image obtained by photographing the scene (multistage focus image) is extracted. The main part is extracted.
[0010]
In addition, the main part is a person's face, and it is preferable to further extract a person's torso and reflect the result in the main part extraction. It is preferable to generate image data of one image by combining image data of a plurality of images taken at the focal length .
[0011]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, the image processing method of the present invention will be described in detail based on preferred embodiments shown in the accompanying drawings.
[0012]
FIG. 1 is a block diagram showing an example of a digital photo printer that uses the image processing method of the present invention.
A digital photo printer (hereinafter referred to as a photo printer 10) shown in FIG. 1 basically includes a scanner (image reading device) 12 that photoelectrically reads an image taken on a film F, and read image data ( An image processing apparatus 14 for performing the image processing method of the present invention, which performs image processing of the image information), operation and control of the entire photo printer 10, and light modulated in accordance with image data output from the image processing apparatus 14 And a printer 16 which exposes a photosensitive material (photographic paper) with a beam, develops it (finished), and outputs it as a print.
Further, the image processing apparatus 14 includes an operation system 18 having a keyboard 18a and a mouse 18b for inputting various settings (settings), processing selection and instructions, instructions for color / density correction, and the like, and a scanner. 12 is connected to a display 20 that displays the image read in step 12, various operation instructions, a condition setting / registration screen, and the like.
[0013]
The scanner 12 is a device that photoelectrically reads an image shot on the film F one frame at a time. The light source 22, the variable aperture 24, and diffusion that makes the reading light incident on the film F uniform in the surface direction of the film F. A box 28, an imaging lens unit 32, an image sensor 34 having a line CCD sensor corresponding to R (red), G (green) and B (blue) image reading, an amplifier 36, and A And a / D (analog / digital) converter 38.
[0014]
Further, in the photo printer 10, the scanner 12 is selected according to the type and size of a film such as a new photographic system (Advanced Photo System) and a 135 size negative (or reversal) film, and the form of a film such as strips and slides. A dedicated carrier that can be attached to the main body is prepared. By exchanging the carrier, various types of films and processing can be handled. Images (frames) photographed on a film and used for print creation are conveyed to a predetermined reading position by this carrier.
In such a scanner 12, when reading an image taken on the film F, the reading light emitted from the light source 22 and adjusted in light amount by the variable aperture 24 is positioned at a predetermined reading position by the carrier. The projection light that carries the image photographed on the film F is obtained by being incident and transmitted.
[0015]
The scanner 12 in the illustrated example reads an image photographed on a film by slit scanning. The film F is positioned at a reading position by a carrier 30 and is conveyed in the longitudinal direction (sub-scanning direction), while reading light is received. Incident. The projection light transmitted through the film F is regulated by a slit disposed in the reading position and extending in the main scanning direction orthogonal to the sub-scanning direction. As a result, the film F is two-dimensionally slit-scanned by the slit extending in the main scanning direction, and the image of each frame photographed on the film F is read.
[0016]
The reading light passes through the film F held on the carrier 30 and becomes projection light that carries an image. This projection light is imaged on the light receiving surface of the image sensor 34 by the imaging lens unit 32.
The image sensor 34 is a so-called three-line color CCD sensor having a line CCD sensor that reads an R image, a line CCD sensor that reads a G image, and a line CCD sensor that reads a B image. Extends in the main scanning direction. The projection light of the film F is separated into the three primary colors R, G and B by the image sensor 34 and is read photoelectrically.
The output signal of the image sensor 34 is amplified by the amplifier 36, converted into a digital signal by the A / D converter 38, and sent to the image processing device 14.
[0017]
In the photo printer 10 using the present invention, the scanner is not limited to the one using the slit scanning as described above, and may use a surface exposure that reads the entire surface of one frame image at a time. .
In this case, for example, an area CCD sensor is used, and means for inserting R, G, and B color filters are provided between the light source and the film F, and the image is read by the area CCD sensor by inserting the color filters. Are sequentially performed by the R, G, and B color filters, and the image photographed on the film is separated into three primary colors and sequentially performed.
[0018]
Further, in the photo printer 10 using the present invention, in addition to the image of the film F read by the scanner 12, a scanner that reads an image of a reflection original, an imaging device such as a digital camera or a digital video camera, and computer communication such as the Internet. Alternatively, image data may be received from an image data supply source R such as a floppy disk or an MO disk (magneto-optical recording medium), and a print may be created by reproducing the image data.
[0019]
As described above, the digital signal output from the scanner 12 or the digital camera is output to the image processing device 14 (hereinafter referred to as the processing device 14).
The processing device 14 includes a data processing unit 40, an image processing unit 42, and an image composition unit 42 .
In addition to this, the processing device 14 includes a CPU for controlling and managing the entire photo printer 10 including the processing device 14, a memory for storing information necessary for the operation of the photo printer 10, and the like. The operation system 18 and the display 20 are connected to each part via the CPU or the like (CPU bus).
[0020]
The R, G, and B digital signals output from the scanner 12 are subjected to predetermined data processing such as dark correction, defective pixel correction, shading correction, and the like in the data processing unit 40, and are further subjected to Log conversion. It is digital image data (density data). When the image data is supplied from an image data supply source R such as a digital camera, the image data is converted into image data corresponding to the photo printer 10 in the data processing unit 40 and subjected to necessary processing. The
[0021]
The image processing unit 42 performs predetermined image processing on the digital image data processed by the data processing unit 40, and further converts the image processed image data using a 3D (three-dimensional) -LUT or the like, The image data corresponds to image recording by the printer 16 and display on the display 22.
The image processing performed by the image processing unit 42 is not particularly limited, and various types of known image processing are exemplified. For example, gray balance adjustment using an LUT (lookup table), gradation correction, and density Adjustment, correction of shooting light source type by matrix (MTX), saturation adjustment of image, etc. In addition, use of low pass filter, adder, LUT, MTX, etc., and appropriate combination of these, averaging processing, interpolation calculation, etc. Electronic scaling processing, dodging processing (compression / expansion of density dynamic range), sharpness (sharpening) processing, etc. are exemplified.
The various processing conditions in the image processing unit 42 include, for example, pre-scan image data for roughly reading an image, which is performed prior to main reading (main scanning) for obtaining output image data, or the printer 16. Setting may be performed using image data obtained by thinning out image data corresponding to image data for output.
[0022]
The image data processed by the image processing unit 42 is output to the image combining unit 44 when image combining is performed, and to the printer 16 and the display 20 in other cases.
[0023]
In the photo printer 10, when the image processing method of the present invention is carried out, the processing device 14 has the same scene captured by an imaging device such as a camera or digital camera that captures an image on the film F at different focal lengths. A plurality of images photographed in Step 1, that is, image data of so-called multistage focus images are sent.
In the present invention, a multi-stage focus image photographed with a digital camera is suitable in that photoelectric reading by the scanner 12 is not required and alignment at the time of image composition is easy.
[0024]
In addition, a camera that captures images may be provided with a multi-stage focus mode that automatically captures a plurality of images with different focal lengths. In particular, in the case of a digital camera, one multi-focus image is used as a reference. As the image, image data of other images may be output as difference data from the reference image, or the difference data may be encoded and output.
[0025]
When the image processing method of the present invention is performed, the processing device 14 includes image data of a multistage focus image, for example, image data N1 having a short focal distance, as shown schematically in FIG. When image data N2 of medium distance and image data N3 of focal distance far away are supplied with three types of image data having different focal distances in the same scene, and the image synthesis is performed, the data processing unit 40 and the image data After the above-described predetermined processing is performed by the processing unit 42, it is supplied to the image composition unit 44.
[0026]
For each of the supplied images, the image composition unit 44 detects the sharpness of the image in units of pixels by using spatial frequency, quantification of the sharpness or edge strength by differential filter processing, and for each image, A subject having the highest sharpness (so-called edged) compared to other images, that is, a subject (or region) in focus in each image having a different focal length is extracted.
In the illustrated example, the near-focus image data N1 is focused on a flower, the mid-range focus image data N2 is focused on a person, and the far-focus image data N3 is focused on a background mountain. These are extracted in the image.
[0027]
Next, the image compositing unit 44 is supplied using image data of an image in which at least the target subject is in focus, according to the supplied information on the target subject (hereinafter referred to as the target subject). One image of this scene is synthesized from the multistage focus image and output to the display 20 or the printer 16.
FIG. 2 shows an example of composition. In the image a of FIG. 2, a person and a flower are indicated as the subject of interest, and in the image b, a person and a mountain are indicated as the subject of interest. Subjects other than are blurred.
[0028]
The subject of interest may be input by a known method using the keyboard 18a, mouse 18b, or the like according to the request of the customer (print creation requester) or the like. As a preferred method, the subject of interest is in focus. A method of instructing by the focal length of the existing image is exemplified. For example, when the subject of interest is a person and a flower as in the image a, the operator has a medium-range focus corresponding to the image data N2 focused on the person and the image data N1 focused on the flower. It is sufficient to instruct and input the short-distance focal point corresponding to as information on the subject of interest.
In addition, the correspondence between each image data of the multistage focus image and the focal length may be such that an operator may input a focal length (short distance, middle distance, long distance, etc.) for each image, and the image data may be a digital camera or the like. In the case of using an imaging device, information on the focal length may be recorded in the header of the image file. In the case of using the film F of the new photographic system, a magnetic recording medium formed on the film. The focal length information may be recorded and read.
[0029]
Another preferred method for inputting the information on the subject of interest is a method of cutting out a region of the image using the mouse 18b or the like and instructing this as information on the subject of interest.
These means may be used in combination.
[0030]
There are no particular limitations on the method of image synthesis, and various known methods for synthesizing a plurality of images (image data) can be used.
For example, a method of incorporating (replacing) an image (image data) of a focused subject focused on another image as a reference image with one image focused on the focused subject as an example. Is done. Alternatively, as described above, when the subject of interest is designated by the focal length, if all the subjects that are in focus at the designated focal length (the sharpest) are incorporated in the same reference image, Good.
[0031]
In addition, when there is an unnecessary subject such as an irrelevant person or dust in the scene, and this is in focus in the synthesized image, for example, an unnecessary subject (including it) is used by using the mouse 18b or the like. The area may be designated, and the image may be synthesized by using an unfocused image in the multistage focused image.
Further, when combining the images, for example, in order to make the created image a natural image, weighting of the image data is performed at the edge portion (end portion) of each subject, and the images are combined. The seam may be finished in a smooth and natural state.
[0032]
Here, since the focal lengths of the multistage focus images are different even in the same scene, the size of each subject in the image is actually slightly different. Therefore, in order to prevent image misalignment when images are merged, image omission due to the image shift, and the like, the reference image is based on the image with the farthest focal length that has the subject of interest focused in the scene. Is preferable.
For example, in the example shown in FIG. 2, when a person and a flower are instructed as the subject of interest as shown in the image a, a medium-focus image (image data N2) in which the person is focused is displayed. As a reference image, a short-focus image (image data N1) or a focused subject is incorporated. Further, as shown in the image b, when a person and a mountain are instructed as the subject of interest, a medium-distance focus image is used with reference to a long-distance focus image (image data N3) in which the mountain is in focus. A person of (image data N2) or a focused subject is incorporated.
[0033]
In addition, the shooting distance of each subject in the scene and the distance between the subjects are obtained in pixel units from the sharpest subjects in each image of the multistage focus image, and the virtual binocular image is obtained using this distance information. It is also possible to create a stereoscopic image.
[0034]
By the way, the image processing unit 42 described above extracts the main part (main subject) of the image as necessary, and performs image processing according to the extracted main part. In the present invention, a multistage focus image is supplied. When this is done, the information on the focused subject (region) extracted for each image is used to extract the main part of the image .
[0035]
For example, in the image shown in FIG. 3A, when extracting the face of a person as the main part, if the color of the ground (background) in the area where the flower exists is close to the skin color, In part extraction, an area including a flower may be erroneously extracted as a face together with the face.
[0036]
Here, all the in-focus subjects in each image of the multi-stage focus image have an approximate distance from the photographer. In a multi-focus image, the photographer usually takes one image focused on the most important subject.
In this method, in each image of the multistage focus image, the main part is extracted by paying attention to the focused subject, and the consistency of the extraction result is checked.
In a photograph, even if there are a plurality of main parts, the distance from the photographer of each main part is usually approximate. For example, when a face as a main part is extracted from a focused subject of each image of a multistage focus image as shown in FIG. 3B, the main part exists only in one of the three images. If the main part is extracted from a plurality of images having different focal lengths, it can be determined that any one is erroneous extraction.
Therefore, as shown in FIG. 3B, a face is extracted from both the medium-distance focus image (image data N2) and the short-distance focus image (image data N1). parts within the case is of medium-range focus image, the main part extracted from a short focal point of the image can be determined to be unsuitable positive, it can be canceled. That is, it is possible to prevent erroneous extraction of the main part.
[0037]
As another face extraction determination method, after applying the face extraction process to the entire screen, it is possible to reduce erroneous extraction by examining the distribution of the focal distance by pixel in each extraction area and checking the consistency. For example, the area extracted by overlapping flowers and mountains in FIG. 3A is divided into areas with greatly different focus distances, so it is determined that the area is not a face, and the area from which the face is extracted is focused. Since the distance is almost equal and the same area, the face is correctly determined.
Alternatively, erroneous extraction can be reduced by dividing the screen by a set of pixels for each predetermined focusing distance range and applying face extraction processing for each divided area. For example, when the face extraction process is applied to each of the focus areas of the multistage focus images N1 to N3 in FIG. 3B, the face area is correctly extracted only in the focus area of the image N2.
[0038]
In portraits and the like, it is preferable to properly finish not only the face but also the human torso, but since the human torso is physically close to the face from the photographer, the face is in focus. By extracting the trunk from the image, the accuracy and efficiency of extraction can be improved. Normally, only the torso part is close to the face from the photographer around the person in the shot scene, so if the torso that is the in-focus subject is known, the region extracted as the face candidate The ratio of the background portion occupying the periphery can also be used for the consistency evaluation of face extraction.
Alternatively, the photographing magnification may be obtained from the focal length information, and the extracted magnification may be compared with the extracted size of the main part or the body that is focused, and used for the evaluation of the consistency of the face extraction.
Furthermore, it analyzes the shooting scenes such as close-up photography, portraits, landscape photographs, etc. from the focal length and shape of the focused subject, exposure control, sharpness enhancement and blurring for each subject according to the type of scene The image processing content in the image processing unit 42 may be controlled.
[0039]
In the above method, the method for extracting the main part is not particularly limited, and various known methods can be used.
For example, face extraction for extracting a region that is estimated to be human skin from the color / density and continuity of an image (pixel), or binarization of an image using a threshold value disclosed in Japanese Patent Laid-Open No. 8-184925. For example, the region extraction is performed by separating into a high concentration region and a low concentration region, and then the contour extraction of each region is performed, and this is used to extract the hair part, torso, face internal structure extraction (eye part extraction), A subject extraction method for performing background part extraction and the like is exemplified.
Other than these, for example, JP-A-4-346332, 4-346333, 4-346334, 5-100328, 5-158164, 5-165119, 5-165120 No. 6-67320, No. 6-160993, No. 6-160993, No. 6-160994, No. 6-160995, No. 8-122944, No. 9-80652, No. 9-101579, Various subject extraction methods disclosed in the publications of JP-A-9-138470 and JP-A-9-138471 are exemplified.
[0040]
As described above, the image (image data) processed by the image processing unit 42 or the image combining unit 44 is output to the display 20 or the printer 16.
The printer 16 exposes the photosensitive material (printing paper) according to the supplied image data and records a latent image, and performs a predetermined process on the exposed photosensitive material and outputs it as a print. And a processor (developing device).
In a printer, for example, after a photosensitive material is cut into a predetermined length corresponding to a print, a backprint is recorded, and then three types of light beams of R exposure, G exposure, and B exposure according to the spectral sensitivity characteristics of the photosensitive material. Is modulated in accordance with the image data output from the processing device 14 and deflected in the main scanning direction, and the photosensitive material is transported in the sub-scanning direction orthogonal to the main scanning direction, so that the photosensitive material 2 A latent image is recorded by scanning exposure in dimension and supplied to the processor. The processor that has received the light-sensitive material is subjected to predetermined wet development processes such as color development, bleach-fixing, and water washing, and is then dried to be printed and sorted into predetermined units such as one film and collected.
[0041]
The image processing method of the present invention has been described in detail above. However, the present invention is not limited to the above-described embodiments, and various improvements and modifications may be made without departing from the gist of the present invention. It is.
[0042]
【The invention's effect】
As described above in detail, according to the present invention, the main part can be correctly extracted while preventing erroneous extraction by using a multistage focus image obtained by photographing the same scene at different focal lengths.
[Brief description of the drawings]
FIG. 1 is a block diagram of a digital photo printer using an image processing method of the present invention.
FIG. 2 is a conceptual diagram for explaining image composition using a multistage focus image ;
FIGS. 3A and 3B are conceptual diagrams for explaining a method of extracting a main subject using a multistage focus image. FIGS.
[Explanation of symbols]
10 (Digital) Photo Printer 12 Scanner 14 (Image) Processing Device 16 Printer 18 Operation System 20 Display 22 Light Source 24 Variable Aperture 28 Diffusion Box 30 Carrier 32 Imaging Lens Unit 34 Image Sensor 36 Amplifier 38 A / D Converter 40 Data Processing Unit 42 Image processing unit 44 Image composition unit

Claims (3)

Obtain image data of multiple images (multistage focus images) of the scene taken at different focal lengths.
A subject that is in focus for each image is extracted, the main part is extracted by paying attention to each focused subject, and the main part in each image in the image (multistage focus image) obtained by photographing the scene is extracted . An image processing method, wherein a main part of the scene is extracted from consistency of extraction results . The image processing method according to claim 1, wherein the main part is a person's face, and the body of the person is further extracted, and the result is reflected in the main part extraction. The image processing method according to claim 1 or 2, wherein image data of one image is generated by synthesizing image data of a plurality of images taken at different focal lengths in accordance with a region designated in the image.

Images