JP2005142765A - Apparatus and method for imaging - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To realize more real remote interaction with presence without making the whole system complicated. <P>SOLUTION: An object to be imaged is imaged by at least three cameras at mutually different angles, and correction processing is carried out by setting normal directions of the images picked up by the respective cameras to a normal direction of a virtually set plane; and pixel positions of the images having been corrected are made to correspond to one another while related to the imaged object to generate relative position information showing relative position relations of the imaged object to the cameras, and an arrangement of pixel positions constituting a virtual viewpoint image to newly be generated and their luminance components are found from the arrangement of pixel positions made to correspond to one another and their luminance components according to the generated relative position information. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention is applied to, for example, a video conference system or a video phone system, and is a communication system that transmits and receives images bidirectionally via a network. The image to be transmitted and received is captured and reconfigured according to the user's line-of-sight direction. The present invention relates to an imaging apparatus and method.

  As represented by a videophone system, a video conference system, and the like, a system has been proposed in which a plurality of users can remotely interact with each other while viewing a display image of the other party from a location apart from each other. In such a system, a display image of the other party is displayed on the display, and a user who visually recognizes the display is imaged as a subject to be photographed, and the obtained image signal is transmitted through a network such as a public line or a dedicated line. By transmitting to the terminal device, it is possible to give both users a sense of reality.

  In a conventional video conference system, a user who visually recognizes the display image of the other party displayed near the center of the display is captured by the camera at the top of the display, so that the image of the user facing down is displayed on the other party's display. Will be displayed. For this reason, there is a problem in that the users who actually view the display are interacted with each other in a state where their lines of sight are inconsistent, giving a sense of discomfort to each other.

  Ideally, if a camera is installed in the vicinity of the center of the display where the display image of the other party is projected, it is possible to realize a conversation in a state in which the lines of sight of both users are matched. However, it is physically difficult to install a camera near the center of such a display.

  In order to solve such a problem related to the line-of-sight mismatch, for example, a videophone device (for example, refer to Patent Document 1) that uses a half mirror to match the camera direction and the display screen, the light transmission state and the light scattering state can be controlled. An image display / control device that performs display and imaging in time series using a screen and a projector (see, for example, Patent Document 2), with an imaging function that can realize both display and imaging simultaneously by using a hologram screen and a projector. A display device (see, for example, Patent Document 3) has been proposed.

  In addition, a bidirectional communication system, a terminal device, and a control method have been proposed that match the line of sight with the display screen by controlling the optical axis of the camera on the other side according to the line of sight and the position of the face (for example, Patent Documents). 4).

  Also, three-dimensional information of the subject is extracted based on input images taken by a plurality of cameras arranged on both sides of the display, and an output image of the subject is determined according to the extracted three-dimensional information and information on the viewpoint position of the receiver. Has been proposed (see Patent Document 5, for example). In this image processing apparatus, by synthesizing a virtual viewpoint camera image centered on the screen using an epipolar plane image generated from a plurality of camera images arranged on a straight line, the user's line of sight is made coincident and a sense of presence is realized. High communication can be realized.

  In addition, an image processing method and apparatus for displaying an image according to the viewpoint position of the observer by switching and displaying an image according to the viewpoint position of the observer from the input image group (see, for example, Patent Document 6). Has also been proposed. In this image processing method or the like, an epipolar plane image can be similarly used to facilitate the search for corresponding points.

  In addition, in order to match each other's line of sight in a video conference, an image communication apparatus that generates three-dimensional position information based on images taken by two cameras installed on the left and right of the screen (see, for example, Patent Document 7). Has also been proposed.

Japanese Patent Laid-Open No. 61-65683 JP-A-4-11485 JP-A-9-168141 JP 2000-83228 A JP 2001-52177 A JP 7-296139 A JP-A-7-99644

  However, in the above-described conventional system, it is possible to realize a conversation with the line of sight matched between users who actually view the display. However, a special device such as a half mirror, a hologram screen, or a projector is used. There is a problem that a simple and inexpensive system cannot be configured.

  In recent years, with the advancement of broadband network technology, the needs for this videophone system and videoconferencing system are increasing. By constantly extracting the direction of the line of sight of the user viewing the display, it is possible to obtain detailed movements and expressions. It is necessary to realize a more realistic and realistic remote dialogue that is accurately captured.

  Accordingly, the present invention has been devised to solve the above-described problems, and the object of the present invention is to realize a more realistic and realistic remote conversation without complicating the entire system. An object of the present invention is to provide a communication system, an imaging apparatus and a method that can be used.

  In order to solve the above-described problems, in the present invention, an imaging target is imaged by at least three cameras from different angles, and the normal direction of each image captured by each camera is virtually set. A correction process is performed by matching with the normal direction of the plane, and each image position subjected to the correction process is associated with each imaging position while being associated with each imaging object, and the relative position of the imaging object with respect to each camera Relative position information indicating the relationship is generated, and from the pixel position and its luminance component associated with each other, the pixel position and its luminance component constituting the virtual viewpoint image to be newly generated according to the generated relative position information Ask for.

  That is, an imaging apparatus to which the present invention is applied includes an imaging unit that includes at least three cameras that capture an image of an imaging target from different angles, and an image correction unit that aligns the normal direction of each image captured by at least each camera. In addition, between each image corrected by the image correction unit, a matching unit that associates each pixel position with each other while associating with the shooting target, and generates relative position information indicating a relative positional relationship of the shooting target with respect to each camera. Based on the generated relative position information, the pixel position constituting the virtual viewpoint image to be newly generated and the luminance component thereof are obtained from the pixel position and the luminance component associated with each other by the information generating means and the matching means. Image generating means.

  In addition, the imaging method to which the present invention is applied captures a subject to be photographed from at least three cameras at different angles, and performs a correction process for aligning the normal direction of each image captured by each camera in one direction. The image processing unit associates each image position with each other while associating each image with the image capturing target, generates relative position information indicating the relative positional relationship of the image capturing target with respect to each camera, and associates the pixel positions with each other. In addition, from the luminance component, the pixel position constituting the virtual viewpoint image to be newly generated and the luminance component thereof are obtained according to the generated relative position information.

  In the imaging apparatus and method to which the present invention is applied, a method of a virtual plane in which a subject to be imaged is imaged by at least three cameras from different angles and the normal direction of each image captured by each camera is virtually set. Relative processing indicating the relative positional relationship of the imaging target with respect to each camera by performing correction processing according to the line direction, associating each image position subjected to this correction processing with each imaging position in association with the imaging target Position information is generated, and a pixel position and its luminance component constituting a virtual viewpoint image to be newly generated are obtained from the pixel position and its luminance component associated with each other according to the generated relative position information.

  As a result, in the imaging apparatus and method to which the present invention is applied, visual communication in which the line of sight is always matched between users who interact with each other without complicating the entire system can be realized, and a more realistic and realistic remote can be realized. Dialogue can be realized. In addition, by performing geometric normalization by aligning the normal vector in one direction with respect to the captured image and parallelizing the epipolar lines, the virtual viewpoint image using the correspondence between the images and the image interpolation is used. Generation can be performed effectively.

  Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.

  In the communication system 1 to which the present invention is applied, for example, as shown in FIG. 1, a user a at a point A and a user b at a point B are remotely interacting with each other while viewing a display image of the other party from a distant place. System.

  At point A, a camera 11a, a camera 12a, and a camera 13a that capture images of the user a as a subject to be photographed from different angles, and a display for displaying the image of the user b captured at the point B side to the user a 5a and a terminal device 2a that generates a virtual viewpoint image Ima based on the images Pa1, Pa2, and Pa3 captured by the cameras 11a, 12a, and 13a, and transmits the virtual viewpoint image Ima to the point B via the network 7. ing.

  At the point B, the camera 11b, the camera 12b, the camera 13b, and the image of the user a captured at the point A are displayed for the user b. A terminal device 2b that generates a virtual viewpoint image Imb based on the display 5b and the images Pb1, Pb2, and Pb3 captured by the cameras 11b, 12b, and 13b and transmits the virtual viewpoint image Imb to the point A via the network 7 is provided. Has been.

  The virtual viewpoint images Ima and Imb generated by the terminal devices 2a and 2b are images captured by a virtual camera virtually installed near the center of the displays 5a and 5b on which the other party's display image is projected. Equivalent to.

  The cameras 11a and 11b are respectively installed on the left side of the displays 5a and 5b when viewed from the users a and b, and the cameras 12a and 12b are respectively installed on the right side of the display when viewed from the users a and b. It becomes. Furthermore, the cameras 13a and 13b are installed on the upper side of the display as viewed from the users a and b, respectively.

  The cameras 11, 12, and 13 are installed with the shooting direction and the shooting angle of view fixed. However, the cameras 11, 12, and 13 can be freely changed based on information input from the users a and b. Good. Incidentally, in the description of the communication system 1 below, an example is given in which a triangular region composed of the focal points of the cameras 11, 12, and 13 is installed according to the height of the user's line of sight. Explain.

  The displays 5a and 5b display images based on the virtual viewpoint images Imb and Ima supplied from the counterpart point via the network 7 via a liquid crystal display surface, for example. The liquid crystal display surfaces of the displays 5a and 5b are composed of a large number of liquid crystal display elements and the like, and the liquid crystal display elements are optically modulated in accordance with output signals based on the virtual viewpoint images Imb and Ima to create an image to be displayed to the user.

  The terminal devices 2a and 2b are usually configured by electronic devices such as a personal computer (PC). These terminal devices 2a and 2b have a function for communicating with each other via the network 7, and transmit images and sounds in response to requests from the other party. The configuration of the terminal devices 2a and 2b will be described later.

  The network 7 includes, for example, an Internet network connected to the terminal device 2 via a telephone line, ISDN (Integrated Services Digital Network) / B (broadband) -ISDN connected to a TA / modem, and the like. It is a public communication network that enables bidirectional transmission and reception. Incidentally, when the communication system 1 is operated in a certain narrow area, the network 7 may be configured by a LAN (Local Area Network). Further, when transmitting moving images, the network 7 is continuously transmitted from one channel having moving images including, for example, MPEG (Moving Picture Experts Group) data, based on the Internet protocol (IP). . In addition, when transmitting a still image, the image is transmitted at regular intervals from a channel different from the channel for transmitting a moving image. Note that a network server (not shown) may be connected to the network 7. This network server (not shown) manages, for example, Internet information, receives predetermined requests from the terminal device 2, and transmits predetermined information stored in itself.

  Next, the configuration of the terminal device 2 will be described with reference to FIG. 2 taking the terminal device 2a as an example.

  The terminal device 2a is connected to the correcting unit 20 to which the images Pa1, Pa2, and Pa3 are supplied from the connected cameras 11a, 12a, and 13a, the matching unit 29 connected to the correcting unit 20, and the matching unit 29. The virtual viewpoint image generation unit 30, the output control unit 31 for transmitting the virtual viewpoint image Ima generated by the virtual viewpoint image generation unit 30 to the other terminal device 2 b, and the respective cameras 11 a, 12 a, 13 a And an information generation unit 33 that generates relative position information indicating the relative positional relationship of the user a.

  The correction unit 20 includes geometric image correction units 21, 22, and 23 for performing geometric image correction on the images Pa1, Pa2, and Pa3 transmitted from the cameras 11a, 12a, and 13a. And a normalization processing unit 24 for normalizing the images that have been subjected to image correction by the image correction units 21, 22, and 23.

  Based on the control information including the geometric positional relationship of the cameras 11a, 12a, and 13a transmitted from the camera calibration unit 26, the geometric image correction units 21, 22, and 23 are used for the images Pa1 and Pa2. , Pa3 is corrected. The geometric positional relationship between the cameras 11a, 12a, and 13a may be parameterized in the control information transmitted from the camera calibration unit 26 described above. Further, when imaging is performed while changing the shooting direction and / or the shooting angle of view of each camera 11a, 12a, 13a, these are parameterized by the camera calibration unit 26, and these parameters are used when correcting the image. May be included in the control information. Thereby, the geometrical image correction units 21, 22, and 23 can perform image correction in real time according to the shooting direction and / or the shooting angle of view of each camera 11a, 12a, and 13a.

  Similarly, the camera calibration unit 26 may parameterize the chromatic aberration and distortion of each lens of the cameras 11 a, 12 a, and 13 a, and shift the optical axis, and transmit these parameters to the correction units 20.

  The normalization processing unit 24 is supplied with images corrected by the geometric image correction units 21, 22, and 23, and performs geometric normalization processing on these images. The normalization processing unit 24 matches the normal directions of the images Pa1, Pa2, and Pa3 captured by the cameras. That is, the normalization processing unit 24 normalizes the normal direction of each image Pa1, Pa2, Pa3 by matching it with the normal direction of the virtually set virtual plane π, and respectively normalizes the normalized images Pm1, Pm2 and Pm3 are generated. In such a case, the normalization processing unit 24 obtains a projective transformation matrix for projecting the images Pa1, Pa2, Pa3 captured by the cameras 11a, 12a, and 13a onto the virtual plane π, and obtains the obtained projective transformation matrix. Based on the above, the normal direction of each image is matched with the normal direction of the virtual plane π.

  Incidentally, when applying a so-called fixed viewpoint camera as the cameras 11a, 12a, and 13a, the camera calibration unit 26 obtains the normal directions of the images Pa1, Pa2, and Pa3 in advance by the camera calibration unit 26. Also good. Further, when imaging is performed while changing the shooting direction and / or shooting angle of view of each camera 11a, 12a, 13a, these are parameterized by the camera calibration unit 26, and these are used when normalizing the image. The parameter may be included in the control information. Accordingly, it is possible to flexibly cope with the case where imaging is performed while sequentially changing the shooting direction according to the positions of the users a and b.

  In addition, by storing these parameters in a ROM or RAM (not shown) in the camera calibration unit 26, the correction unit 20 can refer to them at any time according to the situation, and can perform high-speed correction processing. Can be realized. The camera calibration unit 26 obtains these parameters each time the images Pa1, Pa2, and Pa3 are supplied from the cameras 11a, 12a, and 13a. High correction processing can be realized.

  The matching unit 29 is supplied with the normalized images Pm1, Pm2, and Pm3 generated by the normalization processing unit 24, respectively. The matching unit 29 associates the normalized images Pm1, Pm2, and Pm3 with each other.

  This association is performed by extracting pixel positions and luminance components at the same location constituting the face of the user a between the normalized images Pm1, Pm2, and Pm3. For example, as shown in FIG. 3, with respect to the pixel position P11 on the normalized image Pm1, the pixel position P11 ′ existing at the same location on the normalized image Pm2 is specified as the corresponding point, and further the normalized image A pixel position P11 ″ existing at the same location on Pm3 is specified as the corresponding point. In such a case, when associating the pixel position P11 on the normalized image Pm1 with the pixel position P11 ′ on the normalized image Pm2, the pixel information on the normalized image Pm3 is referred to. You may do it. Based on the same method, for the pixel position P12 on the normalized image Pm1, the pixel position P12 ′ existing at the same location on the normalized image Pm2 is specified as the corresponding point, and further on the normalized image Pm3. The pixel position P12 '' existing at the same location is specified as the corresponding point.

  That is, the matching unit 29 associates the normalized images Pm1, Pm2, and Pm3 normalized by the normalization processing unit 24 for each pixel position while associating them with the imaging target. Incidentally, the matching unit 29 may be executed only for the part where the feature is extracted for this association, or may be executed for all the pixels constituting the normalized images Pm1, Pm2, and Pm3.

  The information generation unit 33 may generate the relative position information to be generated based on the line-of-sight direction of the user a with respect to the display 5a. In such a case, the information generation unit 30 acquires the line-of-sight direction of the user a from the images Pa1, Pa2, and Pa3 supplied from the cameras 11a, 12a, and 13a, and generates relative position information based on the acquired direction. Accordingly, it is possible to realize the same processing as that of adjusting the shooting direction of the virtual camera to the line of sight of the user a. In addition, the information generation unit 33 may acquire depth information from image information generated by each camera 11a, 12a, 13a, and generate relative position information based on the acquired depth information. The information generation unit 33 may generate relative position information based on information input via an operation unit such as a keyboard or a mouse (not shown).

  The virtual viewpoint image generation unit 30 is input with the pixel positions and the luminance components associated with each other by the matching unit 29. Further, the virtual viewpoint image generation unit 30 configures a virtual viewpoint image Ima to be newly generated according to the relative position information generated by the information generation unit 33 from the pixel positions associated with each other and their luminance components. The pixel position and its luminance component are obtained. The virtual viewpoint image generation unit 30 supplies a virtual viewpoint image Ima configured by the obtained pixel position and its luminance component to the output control unit 31.

  The output control unit 31 controls to transmit the virtual viewpoint image Ima generated by the virtual viewpoint image generation unit 30 to the terminal device 2b via the network 7. In such a case, the output control unit 31 may perform control so that the images Pa1, Pa2, and Pa3 generated by the cameras 11a, 12a, and 13a are independently transmitted to the terminal device 2b.

  Next, a specific operation in the terminal device 2 will be described.

  A user a as a subject to be photographed is photographed from different angles by the cameras 11a, 12a, and 13a. As a result, the line of sight of the user a on the images Pa1, Pa2, and Pa3 generated by the cameras 11a, 12a, and 13a, the face orientation, and the like are different from each other. When such images Pa1, Pa2, and Pa3 are respectively supplied to the normalization processing unit 24 in the correction unit 20, they are normalized based on the following method.

  For example, as shown in FIG. 4, when the optical centers C <b> 1, C <b> 2, and C <b> 3 of the cameras 11 a, 12 a, and 13 a are aligned at different optical angles from the mutually different angles to the M points to be imaged, images generated thereby Pa1, Pa2, and Pa3 are parallel to the imaging surfaces of the cameras 11a, 12a, and 13a. Here, the direction of the straight line connecting each camera 11a, 12a, 13a and point M coincides with the normal direction of each image Pa1, Pa2, Pa3 imaged by each camera, but these indicate different directions. Yes. Normalization images Pm1, Pm2, and Pm3 whose image planes are parallel to each other are generated by performing normalization so that the normal directions of these images Pa1, Pa2, and Pa3 are the same.

  Specifically, as shown in FIG. 4, a virtual plane π surrounded by the optical centers C1, C2, and C3 is set, and these images Pa1, Pa2, and Pa3 are set with respect to the normal direction of the virtual plane π. Normalization is performed using the projective transformation matrix P so that the normal direction is the same direction. For this reason, the setting of the virtual plane π determines in which direction the image planes of the images Pa1, Pa2, and Pa3 are parallelized.

Incidentally, as a premise for explaining the relationship between the projective transformation matrix and the image plane, in the following, a point on the image is m = [m ₀ , m ₁ ] ^T, and a point in space is w = [w ₀ , w ₁ , w ₁ ] ^T. Here, as shown in FIG. 5, a representation of the M point in the world coordinate system is indicated by w, and the optical centers C (C1, C2, C3) and M points of the cameras 11a, 12a, 13a The coordinates of the intersections of the straight line connecting the images Pa1, Pa2, and Pa3 are displayed by m. That is, the position in the image plane in each image Pa1, Pa2, Pa3 is defined by m = [m ₀ , m ₁ ] ^T. In such a case, the description in the Saisei coordinate system at each point is as follows.

  The relationship between the point wi in the space and the point mi on the image is given by the following equation.

  Here, s is an arbitrary variable, and P is the projective transformation matrix described above. This projective transformation matrix is represented by P = A [R | t]. R represents an image rotation matrix and is given by the following equation.

  Further, t represents a parallel progression of images and is given by the following equation.

  The matrix A is called a camera internal parameter and is given by the following equation.

Here, (u ₀ , v ₀ ) represents the center of the image, and α (= −f · ku) and β (= −f · kv / sin θ) represent the scale factors of the m ₀ axis and the m ₁ axis, respectively. Γ (= f · ku · cot θ) represents the twist of two axes.

  That is, by obtaining the projective transformation matrices Po1, Po2, Po3 for each of the images Pa1, Pa2, Pa3 based on the above equations, as shown in FIG. 6 (a), for each image Pa1, Pa2, Pa3, 3 Projective transformation from a dimensional space to a two-dimensional image plane can be performed. In other words, projective transformation can be performed from m defined in the world coordinate system to w defined in the camera coordinate system.

  FIG. 6B shows the relationship between the obtained projective transformation matrix Pox (Po1, Po2, Po3) and the image plane. The normalized image Pm (Pm1, Pm2, Pm3) can be created by normalizing each image Pa (Pa1, Pa2, Pa3) that has been projectively transformed into the camera coordinate system.

  In the communication system 1 to which the present invention is applied, a projective transformation matrix Pnx (Pn1, Pn2, Pn3) for performing projective transformation from the three-dimensional space represented as the world coordinate system to the normalized images Pm1, Pm2, and Pm3 is obtained. . This projective transformation matrix Pnx (Pn1, Pn2, Pn3) is realized by redefining the image rotation matrix R described above.

  FIG. 7 shows a procedure for redefining the rotation matrix R. In step S11, r2t is adjusted so that the rotation matrix R = [r0t, r1t, r2t] is perpendicular to a plane passing through the optical center C (C1, C2, C3). Next, the process proceeds to step S12, and the r0t is adjusted to an arbitrary value. In step S13, r1t is obtained from the outer product of r0t and r2t. Finally, the process proceeds to step S14, and the projective transformation matrices Pn1, Pn2, and Pn3 are obtained based on R = [r0t, r1t, r2t] in which each parameter is adjusted in steps S11 to S13.

  Incidentally, the normalized images Pm1, Pm2, and Pm3 subjected to the projective transformation in the normalization processing unit 24 are associated with each pixel position while being associated with the imaging target in the matching unit 29 as described above. In the present invention, the normalization processing unit 24 in the previous stage of the matching unit 29 is normalized in advance and the epipolar lines are parallelized. In combination with this, in the present invention, any two normalized images among the normalized images Pm1 to Pm3 are associated. The matching method is shown, for example, in the following document “A Maximum Likelihood Stereo Algorithm”, Ingemar J. Cox, Sunita L. Hingorani, Satish B. Rao, and Bruce M. Maggs, CVIM vol. 63, (1996). A method may be used.

  The pixel position associated as described above is output to the virtual viewpoint image generation unit 30 together with the luminance component. The relative position information generated by the information generation unit 33 is also transmitted to the virtual viewpoint image generation unit 30.

The virtual viewpoint image generation unit 30 generates a virtual viewpoint image Ima based on the result of association in the matching unit 29 described above. Here, consider a case where the virtual viewpoint image generation unit 30 generates a virtual viewpoint image Ima captured by a virtual camera having the optical center Cq. In this case, when the position of the optical center Cq is defined in relation to the optical centers C1, C2, C3 as shown in FIG. 8, the optical center Cq and the optical centers C1, C2 of the cameras 11a, 12a, 13a are defined. , C3 can be expressed by the following equation (1).
Cq = (1-α) (1-β) · C1 + α (1-β) · C2 + β · C3 (1)
In this equation (1), α and β are relative position information generated by the information generation unit 33. The relative position information α, β is determined in relation to the position of the optical center Cq of the virtual camera with respect to the optical centers C1, C2, C3 of the cameras 11a, 12a, 13a. Here, α decreases as the virtual viewpoint in the virtual camera approaches the optical center C1 of the camera 11a, and increases as the virtual viewpoint approaches the optical center C2 of the camera 12a. Also, β increases as the virtual viewpoint in the virtual camera becomes closer to the optical center C3 of the camera 13a. That is, the information generation unit 33 described above generates the relative position information α and β based on the direction of the line of sight of the user a with respect to the display 5a, thereby setting the optical center Cq of the virtual camera that matches the line of sight of the user a. Is also possible.

Moreover, each pixel position which comprises the virtual viewpoint image Ima imaged with a virtual camera can be determined based on the following formula | equation (2). In Equation (2), the pixel position in the virtual viewpoint image Ima is wq, and the projective transformation matrix of the virtual viewpoint image Ima is Pnq. In addition, the image positions of the normalized images Pm1, Pm2, and Pm3 corresponding to the pixel position wq are w1, w2, and w3, respectively. These pixel positions are all expressed in world coordinates, and are each expressed as a position in the image plane by multiplying by a projective transformation matrix.
Pnq · wq = (1-α) (1-β) · Pn1 · w1 + α (1-β) · Pn2 · w2 + β · Pn3 · w3 (2)
The pixel position wq determined based on this equation (2) approaches w1 as the optical center Cq of the virtual camera approaches the optical center C1 of the camera 11a, approaches w2 as it approaches the optical center C2 of the camera 12a, and further As it approaches the optical center C3 of the camera 13a, it approaches w3. That is, since the pixel position wq can be freely determined according to the position of the virtual camera, the position of the user a displayed on the virtual viewpoint image Ima can be freely changed.

Also, assuming that the luminance components at w1, w2, and w3 are J11, J11 ′, and J11 ″, respectively, the luminance component Pt at the pixel position wq on the virtual viewpoint image Ima is based on the following equation (3). Can be determined.
Pt = (1-α) (1-β) · J11 + α (1-β) · J11 ′ + β · J11 ″ (3)
The luminance component Pt determined based on the equation (3) approaches J11 as the optical center Cq of the virtual camera approaches the optical center C1 of the camera 11a, and approaches J11 ′ as it approaches the optical center C2 of the camera 12a. Furthermore, it approaches J11 ″ as it approaches the optical center C3 of the camera 13a. That is, since the luminance component Pt can be freely determined according to the position of the virtual camera, the luminance of the pixels constituting the user a displayed on the virtual viewpoint image Ima can be freely changed.

  As described above, the virtual viewpoint image generation unit 30 can determine the coordinates of each pixel constituting the virtual viewpoint image Ima and its luminance component based on α and β as relative position information.

  In the present invention, since the geometric normalization as described above is performed and the epipolar lines are parallelized, it is possible to execute such image interpolation more effectively.

  Particularly, since the cameras 11a, 12a, and 13a have different shooting directions, the luminance components at the image positions w1, w2, and w3 indicating the same image are different from each other. The position of the virtual camera is determined by linearly increasing / decreasing the luminance component Pt according to α and β as relative position information so that either one of the different luminance components is the minimum value and the other is the maximum value. Accordingly, it is possible to determine the luminance component of the pixels constituting the user a displayed on the virtual viewpoint image Ima.

  For example, when the image positions w1, w2, and w3 capture the edge portions of the glasses worn by the user a, the luminance components are different because they are captured in different shooting directions. . These luminance components are assigned as minimum values or maximum values, respectively, and the luminance components Pt are determined by linearly increasing / decreasing the luminance components according to α and β as relative position information. This determined luminance component Pt corresponds to a luminance component indicating the edge of the glasses in the virtual viewpoint image Ima to be newly generated.

FIG. 9 shows the virtual viewpoint image Ima and the normal vectors of the images Pa1, Pa2, Pa3 generated by the cameras 11a, 12a, 13a. Normal vector _{n x} of the virtual viewpoint image Ima obtained in this virtual viewpoint image generator 33, image Pa1, Pa2, the equation (4) below by using the normal vectors _{n 1, n 2, n 3} of Pa3 It can also be expressed based on.
_nx = (1-α) (1-β) · n ₁ + α (1-β) · n ₂ + β · n ₃ (4)
By sequentially determining the coordinates at the pixel position wq and the luminance component Pt as described above, the images Pa1, Pa2, Pa3 in which the line of sight of the user a to be displayed, the orientation of the face, etc. are different from each other, It is possible to create a virtual viewpoint image Ima in which the face and line-of-sight direction of the user a are always facing the front.

  The generated virtual viewpoint image Ima is sent to the network 7 via the output control unit 31. Then, the virtual viewpoint image Ima transmitted to the counterpart terminal device 2b is displayed on the display 5b under the control of the terminal device 2b. The user b interacts while visually recognizing the user a on the virtual viewpoint image Ima displayed on the display 5b. However, since the user a's face and line-of-sight direction are always facing the front, It is possible to enjoy the feeling of visually recognizing an image taken by a virtual camera installed near the center of the screen. Similarly, the user a interacts while visually recognizing the user b on the virtual viewpoint image Ima displayed on the display 5a. However, the user b always facing the front can be visually recognized. That is, in this communication system 1, visual communication in which the line of sight is always matched between users who are interacting can be realized, and a more realistic and realistic remote conversation can be realized.

  In particular, in this communication system 1, it is sufficient to arrange at least three cameras 11, 12, and 13 on both sides of the display 5, and it is not necessary to extract the three-dimensional information of the subject each time, so that the entire system becomes complicated. There is also an advantage of not having to do.

  Further, in the communication system 1, it is not necessary to use a special device such as a half mirror, a hologram screen, or a projector, and a simple and inexpensive system can be configured.

  In the above-described embodiment, the case where the imaging target is imaged by three cameras has been described as an example. However, the present invention is not limited to this case, and four or more cameras are provided on the side of the display. It may be installed and imaged. FIG. 10 is a configuration diagram of a terminal device 2a in which n (n is 4 or more) cameras 11a to 100 are arranged. Images captured by the cameras 11 a to 100 are transmitted to the camera selection unit 101. The camera selection unit 101 selects three cameras having the optical center closest to the optical center Cq of the virtual camera based on the relative position information transmitted from the information generation unit 33, and generates the virtual viewpoint image Ima as described above. . By capturing the user a from four or more different directions, even if the height of the user's line of sight slightly changes, an image captured from the upper three directions that captures the movement of the fine line of sight is acquired. Thus, it is possible to realize a conversation while keeping the line of sight consistent.

  Needless to say, the present invention may be applied as an imaging device in which the terminal device 2 to which the present invention is applied and the cameras 11 and 12 are integrated. Of course, the terminal device 2 and the communication system 1 may be applied to various video content productions that require generation of virtual viewpoint images other than the video conference system.

It is a figure which shows the outline of the communication system to which this invention is applied. It is a figure for demonstrating per structure of a terminal device. It is a figure for demonstrating about the matching in a matching part. It is a figure for demonstrating about the normalization which matches the normal line direction of each image Pa1, Pa2, Pa3. It is a figure for demonstrating about the positional relationship of the point of a three-dimensional space, and the point of an image plane. It is a figure for demonstrating about the relationship between the calculated | required projective transformation matrix and an image plane. It is a flowchart which shows the procedure which redefines the rotation matrix R. It is a figure for demonstrating about the method of producing a virtual viewpoint image. It is a figure shown about the relationship of the normal vector of the virtual viewpoint image and the image produced | generated from each camera. It is a figure for demonstrating about the case where four or more cameras are installed in the side surface of a display, and it images.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Communication system, 2 Terminal device, 5 Display, 7 Network, 11, 12, 13 Camera, 21-24 Geometric image correction part, 24 Normalization process part, 26 Camera calibration part, 29 Matching part, 30 Virtual viewpoint Image generator, 31 output generator, 33 information generator

Claims (7)

Imaging means including at least three cameras for imaging a subject to be photographed from different angles;
Image correcting means for aligning at least one normal direction of each image captured by each camera with one direction;
Matching means for associating each image position with each other between the images corrected by the image correction means, in association with the photographing object;
Information generating means for generating relative position information indicating a relative positional relationship of the photographing object with respect to each camera;
Image generating means for obtaining a pixel position and its luminance component constituting a virtual viewpoint image to be newly generated according to the generated relative position information from the pixel position and its luminance component associated with each other by the matching means. An imaging apparatus comprising: The imaging apparatus according to claim 1, wherein the image correction unit matches a normal direction of each image captured by each camera with a normal direction of a virtually set virtual plane. The imaging apparatus according to claim 2, further comprising a virtual plane setting unit that sets the virtual plane according to the relative position information generated by the information generation unit. The image correction means obtains a projective transformation matrix for projecting each image captured by each camera onto the virtual plane, and based on the obtained projective transformation matrix, determines the normal direction of each image on the virtual plane. The imaging apparatus according to claim 2, wherein the imaging apparatus is aligned with a normal direction. The image correction means selects three cameras based on the relative position information generated by the information generation means when four or more cameras in the imaging means are included, and selects the selected three cameras The imaging apparatus according to claim 1, wherein a normal direction of each image captured by the camera is matched with a normal direction of the virtual plane. The imaging means controls the shooting direction and / or shooting angle of view of each camera,
The image correction means sets the normal direction of each image captured by each camera to the normal direction of the virtual plane based on the shooting direction and / or the shooting angle of view of each camera controlled by the shooting means. The imaging device according to claim 1, wherein Capture the subject to be photographed from different angles with at least three cameras,
Apply correction processing to align the normal direction of each image captured by each camera in one direction,
Between each image that has undergone the above correction processing, it is associated with each of the pixel positions while being associated with the subject to be imaged,
Generating relative position information indicating a relative positional relationship of the photographing object with respect to each of the cameras;
An imaging method comprising: obtaining a pixel position and a luminance component of a virtual viewpoint image to be newly generated from the pixel position and the luminance component associated with each other according to the generated relative position information.

Images