JP2021032678A - Device and method for estimating attitude - Google Patents

Abstract

To provide a technique that can reduce the difference of times taken to grasp the attitudes of cameras equipped in a mobile body.SOLUTION: An attitude estimation device 1 includes: an acquisition unit 11 for acquiring images by cameras 21 to 24 equipped in a mobile body; an estimation unit 121 for performing estimating processing of estimating the attitudes of the cameras on the basis of the images; and a setting unit 123 for setting the priority of the estimation processing for each camera on the basis of the number of acquisition of the estimation results acquired by the estimation processing for each camera.SELECTED DRAWING: Figure 1

Description

The present invention relates to a posture estimation device and a posture estimation method.

Conventionally, there is known an in-vehicle camera calibration device that automatically calibrates external parameters such as an in-vehicle camera mounting position and mounting angle (see, for example, Patent Document 1).

The calibration device disclosed in Patent Document 1 includes a vehicle state determination unit, a camera-by-camera processing unit, and an operation control unit. The vehicle condition determination unit determines the vehicle condition from the vehicle information input from the outside. The processing unit for each camera is arranged for each of the cameras, and processes the input information from each of the cameras. The motion control unit determines the operation mode of each camera from the determination result from the vehicle state determination unit and the output from the processing unit for each camera, and determines which camera to prioritize calibration. The processing unit for each camera determines how the feature points appear from the image taken by the camera and informs the operation control unit.

JP-A-2018-177004

For example, the number of feature points that can be acquired among a plurality of cameras may be biased depending on the condition of the road surface on which the vehicle travels. In such a case, there may be a large difference in the time required for grasping the mounting state (posture) of the cameras among the plurality of cameras.

By the way, for example, at the time of automatic parking, a plurality of cameras are usually used at the same time. For this reason, it is desirable that the posture of each of the plurality of cameras is grasped at the start of automatic parking or the like. As a result, automatic driving such as automatic parking can be safely performed using all the cameras to be used. That is, when there are a plurality of cameras used at the same time, it is desired to reduce the difference in time required for grasping the posture of the cameras among all the cameras used at the same time.

In view of the above problems, it is an object of the present invention to provide a technique capable of reducing the difference in time required for grasping the posture of a plurality of cameras mounted on a moving body.

In order to achieve the above object, the posture estimation device of the present invention performs the captured image with an acquisition unit that acquires captured images from each of a plurality of cameras mounted on the moving body and an estimation process that estimates the posture of the cameras. A configuration including an estimation unit that is performed based on the above, and a setting unit that sets the priority of the estimation process for each camera based on the number of estimation results acquired by the estimation process performed for each camera. (First configuration).

Further, in the posture estimation device having the first configuration, the setting unit has a configuration (second configuration) in which the priority is set based on the comparison result of the cumulative values of the acquired numbers for each camera. Is preferable.

Further, the posture estimation device having the first or second configuration determines the estimation of the posture based on the plurality of estimation results for the camera in which the cumulative value of the acquired number reaches a predetermined plurality of times or more. It is preferable that the configuration further includes a fixing portion (third configuration).

Further, in the posture estimation device having the third configuration, it is preferable that the setting unit has a configuration (fourth configuration) in which the priority is changed when the camera whose posture estimation is confirmed is generated. ..

Further, in the posture estimation device having any of the first to fourth configurations, in the setting unit, the difference between the maximum number and the minimum number of the cumulative values of the acquired numbers obtained for each camera is equal to or larger than a predetermined threshold value. When it becomes, it is preferable that the configuration is such that the priority is changed (fifth configuration).

Further, in the posture estimation device having the fifth configuration, the setting unit sets the priority when the difference between the maximum number and the minimum number becomes smaller than the predetermined threshold value after becoming equal to or larger than the predetermined threshold value. It is preferable that the configuration is changed (sixth configuration).

Further, the posture estimation device having any of the first to fourth configurations includes a determination unit for determining whether or not a camera having a priority condition other than the acquired number exists among the plurality of cameras. Further, the setting unit may have a configuration (seventh configuration) in which the priority is set based on the priority condition when a camera having the priority condition exists.

Further, in the posture estimation device having any of the first to seventh configurations, the setting unit confirms whether or not to change the priority at a predetermined timing, and the predetermined timing is the estimation. When the order of processing is given to all of the plurality of cameras once, and there is a camera among the plurality of cameras whose order is skipped by the setting of the priority, the camera is described as described above. It may be a configuration (eighth configuration) in which the predetermined timing is determined on the assumption that the order is given.

Further, in order to achieve the above object, the posture estimation method of the present invention includes an acquisition step of acquiring captured images from each of a plurality of cameras mounted on a moving body and an estimation process of estimating the posture of the cameras. An estimation step performed based on a captured image, and a setting step of setting the priority of the estimation process for each camera based on the number of acquisitions of estimation results acquired by the estimation process performed for each camera. It has a configuration (nineth configuration) to be provided.

According to the present invention, it is possible to reduce the difference in time required for grasping the posture of a plurality of cameras mounted on a moving body.

The figure for demonstrating the structure of the posture estimation apparatus which concerns on 1st Embodiment. A flowchart showing an example of processing executed when estimating the postures of four in-vehicle cameras in the posture estimation device of the first embodiment. A flowchart showing an example of posture estimation processing by the posture estimation device of the first embodiment. Diagram for explaining the extraction of feature points Diagram for explaining the derivation of optical flow Diagram showing a quadrangle virtually formed from an optical flow The figure for demonstrating the method of specifying two sets of planes whose intersection lines with a predetermined plane are parallel to each other. A diagram for explaining a method of finding the direction of intersection between faces based on one of the specified sets of faces. A diagram for explaining a method of finding the direction of intersection between faces based on the other side of the specified set of faces. A flowchart showing an example of a priority change determination process in the posture estimation device of the first embodiment. The figure for demonstrating the setting of the priority in the posture estimation apparatus of 1st Embodiment. The figure for demonstrating the structure of the posture estimation apparatus which concerns on 2nd Embodiment A flowchart showing an example of processing executed when estimating the postures of four in-vehicle cameras in the posture estimation device of the second embodiment. A flowchart showing an example of a priority change determination process in the posture estimation device of the second embodiment. The figure for demonstrating the setting of the priority in the posture estimation apparatus of 2nd Embodiment.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following description, a vehicle will be described as an example of a moving body, but the moving body is not limited to the vehicle. Vehicles include a wide range of vehicles with wheels, such as automobiles, trains, and automatic guided vehicles. Examples of moving objects other than vehicles include ships and aircraft.

Further, in the following description, the direction in which the vehicle travels straight from the driver's seat to the steering wheel is referred to as the "forward direction". Further, the direction in which the vehicle travels straight from the steering wheel to the driver's seat is defined as the "rear direction". Further, the direction from the right side to the left side of the driver who is facing the front direction, which is the direction perpendicular to the straight traveling direction and the vertical straight line of the vehicle, is defined as the "left direction". Further, the direction from the left side to the right side of the driver who is facing the front direction, which is the direction perpendicular to the straight line and the vertical direction of the vehicle, is defined as the "right direction".

<1. First Embodiment>
FIG. 1 is a diagram for explaining the configuration of the posture estimation device 1 according to the first embodiment. FIG. 1 also shows a photographing unit 2 and a steering angle sensor 3 for inputting information to the attitude estimation device 1. The posture estimation device 1 is provided for each vehicle on which the photographing unit 2 is mounted. Hereinafter, a vehicle provided with the posture estimation device 1 may be referred to as a own vehicle.

The photographing unit 2 is provided on the vehicle for the purpose of monitoring the situation around the vehicle. The photographing unit 2 includes a plurality of cameras 21 to 24. The plurality of cameras 21 to 24 are in-vehicle cameras. Each in-vehicle camera 21 to 24 is fixedly arranged on the vehicle. Each of the four vehicle-mounted cameras 21 to 24 is connected to the posture estimation device 1 by wire or wirelessly, and outputs a captured image to the posture estimation device 1. The mounting positions and angles of the in-vehicle cameras 21 to 24 may deviate due to, for example, deterioration over time or an impact from the outside. The posture estimation device 1 can monitor such a change in the mounting state.

Specifically, the photographing unit 2 includes a front camera 21, a back camera 22, a left side camera 23, and a right side camera 24. The front camera 21 is a camera that photographs the front of the vehicle. The back camera 22 is a camera that photographs the rear of the vehicle. The left side camera 23 is a camera that photographs the left side of the vehicle. The right side camera 24 is a camera that photographs the right side of the vehicle. Each of the vehicle-mounted cameras 21 to 24 is configured by using, for example, a fisheye lens, and has an angle of view θ in the horizontal direction of 180 degrees or more. Therefore, the four in-vehicle cameras 21 to 24 can capture the entire circumference of the vehicle in the horizontal direction.

The number of cameras included in the photographing unit 2 may be a plurality of cameras other than four. For example, when an in-vehicle camera is mounted for the purpose of supporting parking in the back, the photographing unit 2 includes three in-vehicle cameras, a back camera 22, a left side camera 23, and a right side camera 24. It may be.

The steering angle sensor 3 is provided in a vehicle on which the attitude estimation device 1 and the photographing unit 2 are mounted, and detects the rotation angle of the steering wheel (steering wheel) of the vehicle. The output of the steering angle sensor 3 is input to the attitude estimation device 1 via a communication bus B1 such as a CAN (Controller Area Network) bus.

As shown in FIG. 1, the posture estimation device 1 includes an acquisition unit 11, a control unit 12, and a storage unit 13.

The acquisition unit 11 acquires captured images from each of the plurality of vehicle-mounted cameras 21 to 24. The acquisition unit 11 continuously acquires analog or digital captured images from the vehicle-mounted cameras 21 to 24 at a predetermined cycle (for example, a 1/30 second cycle). That is, the aggregate of the captured images acquired by the acquisition unit 11 is a moving image captured by the in-vehicle cameras 21 to 24. Then, when the acquired captured image is analog, the acquisition unit 11 converts the analog captured image into a digital captured image (A / D conversion). The acquisition unit 11 outputs the acquired photographed image or the acquired and converted photographed image to the control unit 12. One captured image output from the acquisition unit 11 becomes one frame image.

The control unit 12 is, for example, a microcomputer, and controls the entire posture estimation device 1 in an integrated manner. The control unit 12 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) (not shown). The storage unit 13 is, for example, a non-volatile memory such as a flash memory, and stores various types of information. The storage unit 13 stores a program as firmware and various data.

The estimation unit 121, the determination unit 122, and the setting unit 123 shown in FIG. 1 are functions of the control unit 12 realized by the CPU of the control unit 12 executing arithmetic processing according to a program stored in the storage unit 13. is there. In other words, the posture estimation device 1 includes an estimation unit 121, a determination unit 122, and a setting unit 123.

At least one of the estimation unit 121, the confirmation unit 122, and the setting unit 123 of the control unit 12 is composed of hardware such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array). May be good. Further, the estimation unit 121, the determination unit 122, and the setting unit 123 are conceptual components. The functions executed by one component may be distributed to a plurality of components, or the functions of the plurality of components may be integrated into one component. Further, the acquisition unit 11 may be configured to be realized by the CPU of the control unit 12 performing arithmetic processing according to a program.

The estimation unit 121 performs an estimation process for estimating the postures of the in-vehicle cameras 21 to 24 based on the captured image. Specifically, the estimation unit 121 performs posture estimation processing for each of the vehicle-mounted cameras 21 to 24. The posture estimation processing of the plurality of vehicle-mounted cameras 21 to 24 is performed in order. The order in which the posture estimation processing is performed will be described later. When estimating the posture of any one of the four in-vehicle cameras 21 to 24, the estimation unit 121 performs the estimation process based on the captured image acquired from the camera that estimates the posture. For example, when estimating the posture of the front camera 21, the estimation unit 121 performs the estimation process based on the captured image obtained from the front camera 21.

The posture may be estimated by obtaining an estimated value related to the mounting state, such as the mounting angle of the on-board cameras 21 to 24. However, the posture estimation may be configured to estimate only whether or not the posture has changed from a predetermined state, instead of obtaining a specific estimated value regarding the mounting state. The details of the posture estimation process of this embodiment will be described later.

The confirmation unit 122 confirms the posture estimation based on the plurality of estimation results for the camera in which the cumulative value of the acquired number of the estimation results acquired by the posture estimation process reaches a predetermined plurality of times. That is, in the present embodiment, the posture estimation is not finalized only by obtaining one posture estimation result for each of the vehicle-mounted cameras 21 to 24. In the present embodiment, when the posture estimation results are obtained a predetermined number of times or more for each of the vehicle-mounted cameras 21 to 24, the posture estimation is determined based on the acquired plurality of estimation results. According to this, it is possible to reduce the possibility that an erroneous estimated posture is output from the posture estimation device 1.

The posture estimation result includes estimation information regarding the posture state, such as an estimated value of the posture. In other words, if the estimation information regarding the posture state is not obtained, it is not considered that the estimation result of the posture has been acquired.

Further, the number of the plurality of estimation results used to determine the posture estimation may be the same as the predetermined plurality of times, but may be different. In addition, the predetermined plurality of times may be appropriately set based on, for example, experimental results. It is preferable that the predetermined plurality of times are set so that the final result of the posture estimation is as accurate as possible and the time until the final result of the posture estimation is obtained is not too long. Further, in the present embodiment, the predetermined plurality of times are the same for all the in-vehicle cameras 21 to 24. However, the predetermined plurality of times may be set separately for each of the in-vehicle cameras 21 to 24. That is, the predetermined plurality of times may be different numbers among the plurality of in-vehicle cameras 21 to 24.

The setting unit 123 sets the priority of the posture estimation process for each of the vehicle-mounted cameras 21 to 24 based on the number of acquired estimation results obtained by the posture estimation process performed for each of the vehicle-mounted cameras 21 to 24. Specifically, the setting unit 123 sets the priority of the estimation process for each of the vehicle-mounted cameras 21 to 24 based on the comparison result of the cumulative value of the number of acquired estimation results for each of the vehicle-mounted cameras 21 to 24. The settings may include initial settings and subsequent change settings.

According to the present embodiment, for example, when there is a camera in which the number of acquired posture estimation results is extremely large or small compared to the others among a plurality of in-vehicle cameras 21 to 24, the priority is adjusted. Therefore, it is possible to prevent the difference in the number of acquisitions of the estimation results from becoming too large. According to the present embodiment, the timing at which the final result of the posture estimation can be obtained can be made as close as possible between the plurality of in-vehicle cameras 21 to 24. For example, when the own vehicle is set to automatically drive using a plurality of in-vehicle cameras 21 to 24, the attitudes of all the in-vehicle cameras 21 to 24 can be grasped by using the posture estimation device 1 of the present embodiment. It is possible to increase the possibility that automatic operation can be started in a safe state.

In the present embodiment, when the priority is raised, the frequency of executing the posture estimation process is increased. Further, when the priority is lowered, the frequency of executing the posture estimation process is reduced. The details of the process related to the priority setting will be described later.

The setting unit 123 confirms whether or not to change the priority at a predetermined timing. In the present embodiment, the predetermined timing occurs every time the posture estimation process is given once to all of the plurality of (specifically, four) vehicle-mounted cameras 21 to 24. However, if there is a camera whose order is skipped by setting the priority among the plurality of in-vehicle cameras 21 to 24, it is considered that the order is given to the camera, and the predetermined timing is determined. With this configuration, the priority can be changed at an appropriate timing.

FIG. 2 is a flowchart showing an example of processing executed when the posture estimation device 1 of the first embodiment estimates the postures of the four in-vehicle cameras 21 to 24. At the start of the process of FIG. 2, the number of acquired posture estimation results is zero for all of the four vehicle-mounted cameras 21 to 24, and there is no difference between the four vehicle-mounted cameras 21 to 24. For this reason, the priority of the posture estimation process for each of the four in-vehicle cameras 21 to 24 is the same. That is, at the start of the processing of FIG. 2, there is no difference in the frequency of the posture estimation processing of the four in-vehicle cameras 21 to 24. At the start of the process of FIG. 2, the four in-vehicle cameras 21 to 24 are given equal order. Posture estimation processing is performed on the four in-vehicle cameras 21 to 24 in a predetermined order.

In step S1, the estimation unit 121 performs an estimation process of the posture of the front camera 21. When the posture estimation process of the front camera 21 is completed, the process proceeds to step S2. In step S2, the estimation unit 121 performs an estimation process of the posture of the back camera 22. When the posture estimation process of the back camera 22 is completed, the process proceeds to step S3. In step S3, the estimation unit 121 estimates the posture of the left side camera 23. When the posture estimation process of the left side camera 23 is completed, the process proceeds to step S4. In step S4, the estimation unit 121 performs an estimation process of the posture of the right side camera 24. When the posture estimation process of the right side camera 24 is completed, the process proceeds to step S4.

In this example, the posture estimation process is performed in the order of the front camera 21, the back camera 22, the left side camera 23, and the right side camera 24, but this is an example. Posture estimation processing may be performed in another order.

Here, the posture estimation process of the in-vehicle cameras 21 to 24 will be described in detail. FIG. 3 is a flowchart showing an example of the posture estimation process by the posture estimation device 1 of the first embodiment. FIG. 3 is a diagram showing details of the posture estimation process in steps S1 to S4 of FIG. Since the posture estimation process is the same for each of the in-vehicle cameras 21 to 24, the posture estimation process will be described below by taking the case of the front camera 21 as an example.

As shown in FIG. 3, first, the estimation unit 121 monitors whether or not the own vehicle equipped with the front camera 21 is traveling straight (step S11). Whether or not the own vehicle is traveling straight can be determined based on, for example, the rotation angle information of the steering wheel obtained from the steering angle sensor 3. For example, if the rotation angle of the steering wheel is zero and the vehicle moves completely straight, not only when the rotation angle is zero, but also when the rotation angle is within a certain range of the plus and minus directions. It may be determined that the own vehicle is traveling straight, including in some cases. It should be noted that the straight-ahead includes both a straight-ahead in the forward direction and a straight-ahead in the backward direction.

The estimation unit 121 repeats the monitoring in step S11 until it detects that the own vehicle is traveling straight. In other words, the estimation unit 121 does not proceed with the posture estimation process unless the own vehicle goes straight. According to this, the posture is estimated by using the position change of the feature point (details will be described later) during the straight-ahead movement, and the posture is estimated by using the information when the traveling direction of the own vehicle is bent. Since the estimation is not performed, it is possible to avoid complicating the posture estimation process. However, the posture estimation process may be advanced by using the position change of the feature point when the traveling direction of the own vehicle is bent.

In the present embodiment, when it is determined that the own vehicle is traveling straight (Yes in step S11), the estimation unit 121 acquires a predetermined number of frames of captured images from the front camera 21 via the acquisition unit 11 (step). S12). The estimation unit 121 acquires, for example, captured images of several frames to several tens of frames.

Next, the estimation unit 121 extracts feature points for each frame image (step S13). The feature point is a point that can be conspicuously detected in the captured image, such as an intersection of edges in the captured image. The feature points are, for example, the edge of a white line drawn on the road surface, cracks on the road surface, stains on the road surface, gravel on the road surface, and the like. In the present embodiment, as a preferred embodiment, feature points having a high degree of cornering, which are characteristic of corners, are extracted. A corner is the intersection of two edges. The degree of cornering can be determined by using a known detection method such as a Harris operator or a KLT (Kanade-Lucas-Tomasi) tracker. In the present embodiment, the extraction conditions of the feature points are the same among the four in-vehicle cameras 21 to 24. However, the extraction conditions of the feature points may be different among the four in-vehicle cameras 21 to 24.

FIG. 4 is a diagram for explaining the extraction of feature points by the estimation unit 121. FIG. 4 schematically shows a captured image P captured by the front camera 21. The captured image P includes a region BO in which the body of the own vehicle is reflected. In the example shown in FIG. 4, the first feature point FP1 and the second feature point FP2 having a high degree of cornering are present in the numerical portion indicating the speed limit drawn on the road surface RS. For this purpose, the estimation unit 121 extracts the first feature point FP1 and the second feature point FP2 existing on the road surface RS.

In FIG. 4, only two feature points FP1 and FP2 are shown for convenience, but if there is a feature point whose corner degree exceeds a predetermined feature point threshold value, other feature points are extracted. To. On the contrary, if there is no feature point whose corner degree exceeds a predetermined feature point threshold value, the feature point is not extracted. It is possible that no feature points are extracted in each frame image.

Returning to FIG. 3, the estimation unit 121 derives the optical flow when the extraction processing of the feature points for all the frame images is completed (step S14). The optical flow is a motion vector showing the movement of feature points between two captured images taken at different times. In this example, the interval between different times is the same as the frame period of the acquisition unit 11. However, the present invention is not limited to this, and the interval between different times may be, for example, a plurality of times the frame period of the acquisition unit 11.

FIG. 5 is a diagram for explaining the derivation of the optical flow by the estimation unit 121. FIG. 5 is a schematic view shown for convenience as in FIG. FIG. 5 shows a photographed image P'(for convenience, the current frame P') taken by the front camera 21 after one frame cycle has elapsed after the photographed image P (referred to as the front frame P for convenience) shown in FIG. ). After the captured image P shown in FIG. 4 is captured, the own vehicle is traveling straight forward for one frame cycle. The circled FP1P shown in FIG. 5 indicates the position of the first feature point FP1 at the time of photographing the front frame P shown in FIG. The circled FP2P shown in FIG. 5 indicates the position of the second feature point FP2 at the time of photographing the front frame P shown in FIG.

As shown in FIG. 5, when the own vehicle goes straight ahead, the first feature point FP1 and the second feature point FP2 existing in front of the own vehicle approach the own vehicle. That is, the first feature point FP1 and the second feature point FP2 appear at different positions in the current frame P'and the previous frame P. The estimation unit 121 associates the first feature point FP1 of the current frame P'and the first feature point FP1 of the previous frame P based on the pixel values in the vicinity thereof, and associates the first feature points FP1 with each other. Based on the position, the optical flow OF1 of the first feature point FP1 is derived. Similarly, the estimation unit 121 associates and associates the second feature point FP2 of the current frame P'and the second feature point FP2 of the previous frame P based on the pixel values in the vicinity thereof, and associates the second feature point FP2. The optical flow OF2 of the second feature point FP2 is derived based on each position of.

The estimation unit 121 derives the optical flow for each of the acquired plurality of frame images as described above. Also, more than two optical flows may be obtained for each frame image. For this reason, the estimation unit 121 may obtain a plurality of optical flow candidates to be used for posture estimation. On the contrary, the estimation unit 121 may not be able to obtain an optical flow candidate for use in estimating the posture.

Returning to FIG. 3, when the optical flow is derived, the estimation unit 121 selects the optical flow to be used for the subsequent processing according to a predetermined selection condition (step S15). The predetermined selection condition is set so that a set composed of two optical flows suitable for the subsequent estimation of the posture is selected. If there are a plurality of pairs satisfying a predetermined selection condition, all the pairs satisfying the predetermined selection condition are selected. The number of pairs satisfying a predetermined selection condition varies depending on the number of feature points extracted from the captured image and the like. If there is no set that satisfies the predetermined selection condition, the subsequent processing cannot be performed. Therefore, the posture estimation process is completed. If the posture estimation process is completed at this stage, the posture estimation result will not be acquired. In the present embodiment, the selection conditions of the optical flow to be used are the same among the four in-vehicle cameras 21 to 24. However, the selection conditions may differ among the four in-vehicle cameras 21 to 24.

Here, the set of the optical flow used selected in step S15 includes the set of the first optical flow OF1 and the second optical flow OF2 shown in FIG. 5, and the processing after step S16 will be described.

The estimation unit 121 proceeds with processing using the optical flow OF1 of the first feature point FP1 and the optical flow OF2 of the second feature point FP2. The estimation unit 121 performs coordinate conversion on the feature points using the internal parameters of the front camera 21 stored in the storage unit 13. In the coordinate transformation, aberration correction and distortion correction of the front camera 21 are performed. Aberration correction is performed to correct distortion due to aberration in the optical system of the front camera 21. Specifically, it is the correction of distortion aberrations such as barrel distortion and pincushion distortion. The distortion correction is performed to correct the distortion of the optical system itself of the front camera 21. Specifically, it is fisheye correction. By the coordinate conversion, the coordinates of the feature points are converted into the coordinates obtained on the two-dimensional image when the subject is photographed by perspective projection.

When the coordinate conversion of the feature points is performed, as shown in FIG. 6, the estimation unit 121 uses the start point of the optical flow OF1 of the first feature point FP1 as the apex SP1 and the end point of the optical flow OF1 of the first feature point FP1 as the apex. A square QL is virtually formed in which the start point of the optical flow OF2 of the EP1 and the second feature point FP1 is the vertex SP2, and the end point of the optical flow OF2 of the second feature point FP2 is the vertex EP (step S16). Hereinafter, for the sake of explanation, geometric elements such as a quadrangle and the sides of the quadrangle are virtually formed and used. However, in the actual processing, the processing may be based on vector operations such as the coordinates of feature points and the direction of a straight line, and may not be based on geometric elements having the same effect.

When the quadrangle QL is virtually formed, the estimation unit 121 uses the quadrangle QL and the internal parameters of the front camera 21 stored in the storage unit 13 to display the projection surface IMG of the front camera 21 in the three-dimensional space. (See FIG. 7) The quadrangle QL is moved onto the projection plane IMG to virtually generate the quadrangle QL1 (step S17).

For the sake of explanation, the edges are defined as follows. The first side SD1 of the quadrangle QL1 corresponds to the side connecting the vertices SP1 and the vertices SP2 in the quadrangle QL. That is, it corresponds to the side connecting the first feature point FP1 and the second feature point FP2 in the front frame P. Similarly, the second side SD2 of the quadrangle QL1 corresponds to the side connecting the vertices SP2 and the vertices EP2 in the quadrangle QL. That is, it corresponds to the optical flow OF2 of the second feature point FP2. Similarly, the third side SD3 of the quadrangle QL1 corresponds to the side connecting the vertices EP1 and EP2 in the quadrangle QL. That is, it corresponds to the side connecting the first feature point FP1 and the second feature point FP2 in the current frame P'. Similarly, the fourth side SD4 of the quadrangle QL1 corresponds to the side connecting the vertices SP1 and the vertices EP1 in the quadrangle QL. That is, it corresponds to the optical flow OF1 of the first feature point FP1.

In addition, the surface is defined as follows (see FIG. 7). The surface including the first side SD1 of the quadrangle QL1 and the optical center OC of the front camera 21 is defined as the first surface F1. Similarly, the surface including the second side SD2 of the quadrangle QL1 and the optical center OC of the front camera 21 is referred to as the second surface F2. Similarly, the surface including the third side SD3 of the quadrangle QL1 and the optical center OC of the front camera 21 is referred to as the third surface F3. Similarly, the surface including the fourth side SD4 of the quadrangle QL1 and the optical center OC of the front camera 21 is referred to as the fourth surface F4.

Next, the estimation unit 121 identifies two sets of faces whose lines of intersection with a predetermined plane are parallel to each other (step S18). A predetermined plane is a plane whose normal is known in advance. Specifically, it is a plane on which the vehicle is moving, that is, a road surface. The predetermined plane does not have to be a strict plane, and can be regarded as a plane when the estimation unit 121 identifies two sets of planes whose intersection lines with the predetermined plane are parallel to each other. Good.

In the present embodiment, the posture estimation device 1 extracts feature points from two images taken at different times when the own vehicle is traveling straight, and calculates the optical flow of the feature points. Further, the feature points are extracted from a stationary object located on a predetermined plane such as a road surface. Therefore, the calculated optical flow represents the relative position change of the stationary object with respect to the own vehicle in the real world. That is, it is a movement vector of the own vehicle in the opposite direction.

Since the second side SD2 and the fourth side SD4 of the quadrangle QL1 both correspond to the optical flow of the feature points, they both correspond to the movement vector of the own vehicle in the real world. Therefore, it is assumed that they are parallel to each other on the road surface.

Further, since the first side SD1 and the third side SD3 of the quadrangle QL1 are both in the positional relationship between the feature points, they correspond to the positional relationship between the stationary objects accompanying the movement of the own vehicle in the real world. The positional relationship before the movement corresponds to the first side SD1, and the positional relationship after the movement corresponds to the third side SD3. At this time, since the position of the stationary object does not change, the positional relationship does not change before and after the movement. Therefore, it is assumed that they are also parallel to each other on the road surface.

Therefore, the estimation unit 121 has two sets of a pair of the second surface F2 and the fourth surface F4 and a set of the first surface F1 and the third surface F3 as planes whose intersection lines with the road surface are parallel. To identify. That is, the estimation unit 121 specifies a total of two sets, with the faces including the optical flow as one set and the faces including the feature points photographed at the same time as another set.

In FIG. 7, the square QL2 is the position of the first feature point FP1 at the time of shooting the current frame P'in the three-dimensional space (real world), and the second feature point FP2 at the time of shooting the current frame P'. The position on the three-dimensional space, the position on the three-dimensional space of the first feature point FP1 at the time of shooting the front frame P, and the position on the three-dimensional space of the second feature point FP2 at the time of shooting the front frame P. It is a quadrangle as the apex. The first surface F1 includes the first side SD11 of the quadrangle QL2. Similarly, the second surface F2 includes the second side SD12 of the quadrangle QL2, the third surface F3 includes the third side SD13 of the quadrangle QL2, and the fourth surface F4 includes the fourth side SD14 of the quadrangle QL2. At this time, as described above, the quadrangle QL2 is assumed to be a parallelogram formed on the road surface.

Next, the estimation unit 121 calculates the normal of the road surface (step S19). First, the estimation unit 121 obtains the direction of the line of intersection between the surfaces based on the first surface F1 and the third surface F3, which are one of the previously specified sets of surfaces. Specifically, the direction of the line of intersection CL1 between the first surface F1 and the third surface F3 is obtained (see FIG. 8). The direction vector V1 of the line of intersection CL1 is a vector perpendicular to each of the normal vector of the first surface F1 and the normal vector of the third surface F3. Therefore, the estimation unit 121 obtains the direction vector V1 of the line of intersection CL1 by the outer product of the normal vector of the first surface F1 and the normal vector of the third surface F3. Since the line of intersection of the first surface F1 and the third surface F3 is parallel to the road surface, the direction vector V1 is parallel to the road surface.

Similarly, the estimation unit 121 obtains the direction of the line of intersection between the second surface F2 and the fourth surface F4, which is the other side of the previously specified set of surfaces. Specifically, the direction of the line of intersection CL2 between the second surface F2 and the fourth surface F4 is obtained (see FIG. 9). The direction vector V2 of the line of intersection CL2 is a vector perpendicular to each of the normal vector of the second surface F2 and the normal vector of the fourth surface F4. Therefore, the estimation unit 121 obtains the direction vector V2 of the line of intersection CL2 by the outer product of the normal vector of the second surface F2 and the normal vector of the fourth surface F4. Similarly, the second surface F2 and the fourth surface F4 have parallel lines of intersection with the road surface, so that the direction vector V2 is parallel to the road surface.

The estimation unit 121 calculates the normal of the surface of the quadrangle QL2, that is, the normal of the road surface by the outer product of the direction vector V1 and the direction vector V2. Since the road surface normal calculated by the estimation unit 121 is calculated by the camera coordinate system of the front camera 21, the coordinate system of the three-dimensional space is obtained from the difference from the vertical direction which is the actual road surface normal, and the front with respect to the road surface. The posture of the camera 21 can be estimated. From the estimation result, the estimation unit 121 estimates the posture of the front camera 21 with respect to the own vehicle (step S20). The calculation process in step S19 can be executed using, for example, a known ARToolkit.

When the posture estimation in step S20 is completed, the estimation unit 121 confirms whether or not the posture estimation is completed for all the optical flow sets selected in step S15 (step S21). When the posture estimation is completed for all the groups (Yes in step S21), the flow shown in FIG. 3 ends. On the other hand, if the posture estimation is not completed for all the sets (No in step S21), the process returns to step S16 and the processes after step S16 are repeated.

The attitude estimation device 1 autonomously identifies two sets of faces whose intersection lines with a predetermined plane are parallel to each other by using the movement of the own vehicle, so that only the extraction error of the feature points is estimated accuracy. It is possible to estimate the attitude of the camera that affects the image. That is, the posture estimation device 1 can estimate the posture of the camera with few error factors. Therefore, the posture estimation device 1 can accurately estimate the posture of the camera.

In the above description, the optical center OC of the front camera 21 and the plane including one side of the quadrangle QL1 are specified, but this is not the case. The plane may be specified by specifying the normal direction of the plane. For example, the normal direction of the plane may be obtained from the outer product of the direction vectors from the optical center OC to each vertex, and the surface may be specified by the normal direction. In this case, the normal direction of the first surface F1 and the normal direction of the third surface F3 are regarded as one set, and the normal direction of the second surface F2 and the normal direction of the fourth surface F4 are regarded as another set. It is good to identify two pairs.

Moreover, the surface may be translated. For example, instead of the first surface F1, the surface obtained by translating the first surface F1 may be paired with the third surface F3. This is because the direction of the line of intersection with a predetermined plane does not change even if the translation is performed.

Returning to FIG. 2, in step S5, the determination unit 122 performs a process of determining the posture estimation according to a predetermined condition. Specifically, the determination unit 122 obtains the cumulative value of the number of acquired posture estimation results for each of the four in-vehicle cameras 21 to 24. After the first posture estimation process, the obtained number of acquisitions becomes the cumulative value as it is. From the second time onward, the cumulative value of the number of acquisitions is obtained by adding the number of acquisitions obtained this time to the cumulative value obtained last time.

The confirmation unit 122 compares the cumulative value of the acquired number with a predetermined plurality of times for each of the four in-vehicle cameras 21 to 24. For the camera whose cumulative value of the acquired number reaches a predetermined plurality of times or more, the confirmation unit 122 confirms the posture estimation based on the plurality of estimation results obtained at that time. For example, the confirmation unit 122 sets the average value of a plurality of estimation results, the median value and the peak value of a histogram created from the plurality of estimation results, and the like as the confirmation values of the posture estimation. For cameras whose cumulative number of acquisitions does not reach a predetermined number of times or more, it is determined that the posture estimation is not finalized, and the posture estimation is not finalized. When the estimation confirmation process is completed, the confirmation unit 122 proceeds to the next step S6.

In step S6, the confirmation unit 122 confirms whether or not the posture estimation has been confirmed for all the in-vehicle cameras 21 to 24. When the posture estimation is confirmed for all the in-vehicle cameras 21 to 24 (Yes in step S6), the process shown in FIG. 2 ends. In this case, the posture estimation is confirmed for all the in-vehicle cameras 21 to 24, and the posture estimation determination result can be output from the attitude estimation device 1. In addition, every time a camera whose posture estimation is confirmed is generated, the confirmation result of the attitude estimation may be output to the outside. Based on the final result of the posture estimation, for example, it is determined whether or not the mounting of the in-vehicle cameras 21 to 24 is misaligned. When it is determined that the in-vehicle cameras 21 to 24 are misaligned, for example, the user of the own vehicle is notified of the abnormal situation. As a result, the user of the own vehicle can take measures such as requesting the camera mounting adjustment by the dealer.

Further, the final result of the posture estimation of the vehicle-mounted cameras 21 to 24 may be used for correcting the parameters of the vehicle-mounted cameras 21 to 24. As a result, even if the mounting of the in-vehicle cameras 21 to 24 is misaligned, the camera is calibrated according to the misalignment, and the misalignment of the in-vehicle cameras 21 to 24 has an adverse effect on the captured image. It can be suppressed. Further, the posture estimation result may be transmitted to the center together with the photographed image when the photographed image is transmitted to the center provided so as to be communicable with a plurality of vehicles via the network. As a result, the shooting information can be handled appropriately at the center.

If there are in-vehicle cameras 21 to 24 for which the posture estimation has not been determined (No in step S6), the process proceeds to step S7.

In step S7, the setting unit 123 determines whether or not it is necessary to change the priority of each of the in-vehicle cameras 21 to 24. FIG. 10 is a flowchart showing an example of the priority change determination process in the posture estimation device 1 of the first embodiment. FIG. 10 is a flowchart showing details of the process of step S7 in FIG.

As shown in FIG. 10, first, the setting unit 123 confirms whether or not there are in-vehicle cameras 21 to 24 whose posture estimation is confirmed (step S71). Specifically, the setting unit 123 determines that there are in-vehicle cameras 21 to 24 whose posture estimation is confirmed when a camera whose posture estimation is newly confirmed is generated in the estimation confirmation process performed immediately before. Then, when there are in-vehicle cameras 21 to 24 whose posture estimation is confirmed (Yes in step S71), the setting unit 123 determines that the priority needs to be changed (step S72).

That is, in the present embodiment, the setting unit 123 changes the priority when a camera whose posture estimation is confirmed occurs. According to this, it is possible to preferentially perform the posture estimation process for the camera that needs to accumulate data in order to confirm the posture estimation.

On the other hand, the setting unit 123 determines that there is no in-vehicle camera 21 to 24 whose posture estimation is confirmed when a camera whose attitude estimation is newly confirmed does not occur in the estimation confirmation process performed immediately before ( In step S71, No), the following process is performed. Specifically, the setting unit 123 extracts the maximum number and the minimum number from the cumulative values of the acquired number of estimation results obtained for each of the four in-vehicle cameras 21 to 24. Then, the setting unit 123 confirms whether or not the difference between the maximum number and the minimum number is equal to or greater than a predetermined threshold value (step S73).

When the difference between the maximum number and the minimum number becomes equal to or greater than a predetermined threshold value (Yes in step S73), the setting unit 123 proceeds to the process in step S72. That is, in the present embodiment, when the difference between the maximum number and the minimum number of the cumulative values of the acquired numbers obtained for each of the cameras 21 to 24 becomes a predetermined threshold value or more, the priority is changed. According to this, it is possible to suppress the occurrence of a bias in the cumulative value of the number of acquired estimation results among the plurality of cameras 21 to 24.

On the other hand, when the difference between the maximum number and the minimum number does not reach the predetermined threshold value or more (No in step S73), the setting unit 123 sets the maximum number and the minimum number in the previous priority change determination process. It is confirmed whether or not the difference is equal to or greater than a predetermined threshold value (step S74). If the difference between the maximum number and the minimum number is equal to or greater than a predetermined threshold value during the previous priority change determination process (Yes in step S74), the setting unit 123 proceeds to step S72.

That is, in the present embodiment, the setting unit 123 changes the priority when the difference between the maximum number and the minimum number becomes smaller than the predetermined threshold value after becoming equal to or larger than the predetermined threshold value. According to this, it is possible to appropriately review the priority setting and suppress the occurrence of bias in the cumulative value of the number of acquired estimation results.

When the difference between the maximum number and the minimum number is smaller than the predetermined threshold value (No in step S74) during the previous priority change determination process, the setting unit 123 determines that the priority change is unnecessary (step). S75).

Returning to FIG. 2, when it is determined in step S7 that the priority is to be changed (Yes in step S7), the process proceeds to step S8. If it is determined in step S7 that the priority change is unnecessary (No in step S7), the process proceeds to step S10.

In step S8, the setting unit 123 changes the priority setting. When there are in-vehicle cameras 21 to 24 whose postures have been estimated, the setting unit 123 lowers the priority of the cameras. The setting unit 123 sets the cumulative value of the number of acquired estimation results obtained for each of the in-vehicle cameras 21 to 24 when the difference between the maximum number and the minimum number is equal to or greater than a predetermined threshold value. Decrease the priority of cameras with many. When there is a camera whose difference between the maximum number and the minimum number becomes smaller than the predetermined threshold value after the difference becomes equal to or larger than the predetermined threshold value, the setting unit 123 restores the priority lowered in the previous round. In the present embodiment, when the priority is lowered, the order of posture estimation processing is skipped. By skipping the order of the low-priority cameras, the processing of the high-priority cameras can be prioritized in time. When the priority is set by the setting unit 123, the process proceeds to the next step S9.

In step S9, the setting unit 123 performs posture estimation processing in order for the cameras to be estimated processing according to the priority of the setting change. When the estimation processing is completed for all the cameras targeted for the estimation processing, the process returns to the process of step S5.

In step S10, the setting unit 123 maintains the same priority as the previous time, and performs posture estimation processing for the cameras to be estimated in order. When the estimation processing is completed for all the cameras targeted for the estimation processing, the process returns to the process of step S5.

FIG. 11 is a diagram for explaining the setting of the priority in the posture estimation device 1 of the first embodiment. The setting of the priority in the posture estimation device 1 will be further described based on the specific example shown in FIG. In the example shown in FIG. 11, there is no difference in priority between the four in-vehicle cameras 21 to 24 by default. By default, each vehicle-mounted camera 21 to 24 is given an order of posture estimation processing once. In FIG. 11, the implementation means that the posture estimation process is executed, and the non-execution means that the posture estimation process is not executed. The cumulative value shown in FIG. 11 is the cumulative value of the number of acquired posture estimation results. In the example shown in FIG. 11, when the cumulative value reaches 80 (= predetermined multiple times) or more, the posture estimation is confirmed. The predetermined threshold value for determining the difference between the maximum number and the minimum number of cumulative values is 20.

In the first time, there is no difference in the priority of the estimation processing, and the estimation processing is performed in the order of the front camera 21, the back camera 22, the left side camera 23, and the right side camera 24. The cumulative value of each of the in-vehicle cameras 21 to 24 after the estimation process is 10, and the posture estimation is not finalized in any of the cameras. Further, the difference between the maximum number and the minimum number of cumulative values is zero, which is smaller than a predetermined threshold value (= 20). Furthermore, for the first time, there is no data for the previous round, and there is no fact that the difference between the maximum and minimum cumulative values was greater than or equal to the predetermined threshold value last time. Therefore, in the priority change determination process performed after the estimation processes of the four in-vehicle cameras 21 to 24, it is determined that the priority setting change is unnecessary.

In the second time, as in the first time, the estimation process is performed in the order of the front camera 21, the back camera 22, the left side camera 23, and the right side camera 24. The cumulative values of the in-vehicle cameras 21 to 24 after the estimation process have not reached 80 or more, and the posture estimation is not finalized in any of the cameras. On the other hand, the difference between the maximum number (= 50) and the minimum number (= 20) of the cumulative values is 30, which is larger than the predetermined threshold value (= 20). Therefore, in the priority change determination process performed after the estimation processing of the four in-vehicle cameras 21 to 24, it is determined that the priority setting change is necessary. In this example, the priority is lowered for cameras whose cumulative value is 20 or more (this setting may be changed as appropriate) with respect to the minimum number. That is, the priority of the front camera 21 and the back camera 22 is lowered, and the next (third) turn is skipped.

In the third time, the estimation process is performed in the order of the left side camera 23 and the right side camera 24. The cumulative values of the in-vehicle cameras 21 to 24 after the estimation process have not reached 80 or more, and the posture estimation is not finalized in any of the cameras. The difference between the maximum number (= 50) and the minimum number (= 40) of the cumulative values is smaller than the predetermined number (= 20). However, at the time of the previous confirmation, the difference between the maximum number and the minimum number of the cumulative values was larger than the predetermined threshold value, so it corresponds to the case where the difference between the maximum number and the minimum number becomes smaller than the predetermined threshold value after becoming equal to or more than the predetermined threshold value. To do. Therefore, in the priority change determination process performed after the estimation processing of the two in-vehicle cameras 23 and 24, it is determined that the priority setting change is necessary. Specifically, the priority settings of the front camera 21 and the back camera 22, whose priorities were lowered last time, are restored. That is, there is no difference in the priority of the estimation processing of the four vehicle-mounted cameras 21 to 24, and the estimation processing of the four vehicle-mounted cameras 21 to 24 is performed next time (fourth time).

In the fourth time, the estimation process is performed in the order of the front camera 21, the back camera 22, the left side camera 23, and the right side camera 24. After the estimation process, the cumulative value of the front camera 21 has reached 80 or more, and the posture estimation is confirmed. Therefore, in the priority change determination process performed after the estimation processing of the four in-vehicle cameras 21 to 24, it is determined that the priority setting change is necessary. Specifically, the priority setting of the front camera 21 whose posture estimation is confirmed is lowered, and the next (fifth) turn is skipped.

In this example, in the fourth time, the maximum number of cumulative values is 80 and the minimum number is 65. Therefore, the difference between the maximum number and the minimum number is smaller than the predetermined threshold value (= 20), and the priority setting is not changed based on this viewpoint. However, it is assumed that the cumulative value of the right side camera 24 is 50. In this case, since the difference between the maximum number (= 80) and the minimum number (= 50) is equal to or greater than the predetermined threshold value (= 20), the priority is set in consideration of the difference between the maximum number and the minimum number. Is done. For example, the priority of the back camera 22 and the left side camera 23 whose cumulative value is equal to or greater than the minimum number (20 in this example) is also lowered.

In the fifth time, the estimation process is performed in the order of the back camera 22, the left side camera 23, and the right side camera 24. The cumulative value has reached 80 or more for all of the three cameras 22 to 24 that have undergone the estimation process, and the posture estimation is confirmed for these cameras 22 to 24. In this case, since the posture estimation has been confirmed for all of the four vehicle-mounted cameras 21 to 24, the series of posture estimation processes is temporarily terminated.

In the above, when determining the maximum number and the minimum number of cumulative values, all four in-vehicle cameras 21 to 24 are always used, but such a configuration is not necessary. For example, when a camera whose posture estimation is confirmed occurs, the camera may be excluded and the maximum number and the minimum number of cumulative values are compared. That is, the configuration may be such that the maximum number and the minimum number of cumulative values are compared by paying attention only to the camera whose posture estimation process is not fixed.

<2. Second Embodiment>
Next, the posture estimation device of the second embodiment will be described. In the description of the second embodiment, the description of the contents overlapping with the first embodiment will be omitted unless there is a particular need for explanation.

FIG. 12 is a diagram for explaining the configuration of the posture estimation device 1A according to the second embodiment. As shown in FIG. 12, in the second embodiment, the determination unit 124 is included in the function realized by the CPU of the control unit 12A executing the arithmetic processing according to the program stored in the storage unit 13. That is, the posture estimation device 1A includes the determination unit 124. This point is different from the first embodiment.

The determination unit 124 determines whether or not there is a camera having a priority condition based on other than the number of acquired estimation results among the plurality of in-vehicle cameras 21 to 24. The priority condition based on the number of acquisitions of the estimation result may be, for example, a priority condition based on the driver's field of view, a priority condition based on the operation status of the own vehicle, a priority condition based on information from the navigation device, or the like.

The priority condition based on the driver's field of view is stored in the storage unit 13 in advance, for example. For example, the front camera 21 and the right side camera 24 (assuming the driver's seat is on the right side) that shoot a direction in which the driver's field of view is easy to secure are the back camera 22 and the left side that shoot a direction in which the driver's field of view is difficult to secure. Compared to the camera 23, the priority of the estimation process is lowered.

Even if the priority condition based on the operation status of the own vehicle is set by the determination unit 124 based on, for example, information obtained from the steering angle sensor 3 via the communication bus B1, information obtained from a shift sensor (not shown), or the like. It may be input from the outside. For example, when the shift sensor is retracted, the priority of the estimation process of the back camera 22 is raised.

The priority condition based on the information from the navigation device may be set by the determination unit 124 based on the information obtained from the navigation device (not shown) connected to the posture estimation device 1A, or may be input from the outside. For example, when a right turn or a left turn is expected from the route set in the navigation device, the priority of the camera estimation process corresponding to the turning direction is raised.

When there are in-vehicle cameras 21 to 24 having priority conditions, the setting unit 123 sets the priority of the estimation process based on the priority conditions. According to this, it is possible to change the method of setting the priority of the estimation process according to the driving scene of the vehicle and the like, and it is possible to estimate the posture of the camera suitable for the driving scene.

FIG. 13 is a flowchart showing an example of processing executed when the postures of the four in-vehicle cameras 21 to 24 are estimated in the posture estimation device 1A of the second embodiment. At the start of the process of FIG. 13, the priority of the posture estimation process for each of the four vehicle-mounted cameras 21 to 24 is the same. That is, at the start of the processing of FIG. 13, there is no difference in the frequency of the posture estimation processing of the four in-vehicle cameras 21 to 24. In this example, when a new priority condition occurs, the old priority condition is updated. Further, when the estimation of the posture of the camera to be prioritized is determined according to the priority condition, it is treated as if there is no priority condition unless a new priority condition is generated. However, the treatment of this priority condition is merely an example and may be changed as appropriate.

In step S31, the determination unit 124 confirms whether or not there is a camera having a priority condition (hereinafter, may be simply referred to as a priority condition) based on the number of acquisitions of the estimation process. The determination unit 124 confirms the information stored in the storage unit 13, the CAN information obtained by using the communication bus B1, the route information obtained from the navigation device, and the like, and whether or not there is a camera having a priority condition. To judge. When the determination unit 124 determines that there is a camera having a priority condition (Yes in step S31), the process proceeds to step S32. On the other hand, when the determination unit 124 determines that there is no camera having the priority condition (No in step S31), the process proceeds to step S34.

In step S32, the setting unit 123 sets the priority of each in-vehicle camera 21 to 24 according to the priority condition. As a result, the priority setting is changed from the setting at the start of processing. When the priority is set by the setting unit 123, the process proceeds to step S33.

In step S33, the estimation processing is sequentially performed for the cameras to be the target of the posture estimation processing according to the priority set to be changed. It should be noted that the number of cameras for which the posture is estimated may be plural or singular. When the estimation process is completed for all the cameras targeted for estimation, the process proceeds to step S35.

In step S34, since there is no camera having a priority condition, the priority is not changed, and the priority set at the start of the process (same for all cameras 21 to 24) is used to estimate the camera to be estimated. The processing is performed in order. In this example, the posture estimation process is performed in the order of the front camera 21, the back camera 22, the left side camera 23, and the right side camera 24. When the posture estimation processing for all four in-vehicle cameras 21 to 24 is completed, the processing proceeds to step S35.

In step S35, the determination unit 122 performs a process of determining the posture estimation according to a predetermined condition. Since the process is the same as step S5 of FIG. 2 described above, the description thereof will be omitted. When the estimation confirmation process is completed, the confirmation unit 122 proceeds to the next step S36.

In step S36, the confirmation unit 122 confirms whether or not the posture estimation has been confirmed for all the in-vehicle cameras 21 to 24. When the posture estimation is confirmed for all the in-vehicle cameras 21 to 24 (Yes in step S36), the process shown in FIG. 13 ends. If there are in-vehicle cameras 21 to 24 for which the posture estimation has not been determined (No in step S36), the process proceeds to step S37.

In step S37, the setting unit 123 determines whether or not it is necessary to change the priority of each of the in-vehicle cameras 21 to 24. FIG. 14 is a flowchart showing an example of the priority change determination process in the posture estimation device 1A of the second embodiment. FIG. 14 is a diagram showing details of the process of step S37 in FIG.

First, the setting unit 123 confirms whether or not a new priority condition has occurred (step S371). The new priority condition is a priority condition that is not taken into consideration in the priority of each camera 21 to 24 set at that time. When a new priority condition occurs (Yes in step S371), the setting unit 123 determines that the priority needs to be changed (step S372).

When a new priority condition has not occurred (No in step S371), the setting unit 123 confirms whether or not there are in-vehicle cameras 21 to 24 whose posture estimation is confirmed (step S373). The confirmation process is the same as in step S71 of FIG. When there are in-vehicle cameras 21 to 24 whose posture estimation is confirmed (Yes in step S373), the setting unit 123 determines that the priority needs to be changed (step S372).

When there is no in-vehicle camera 21 to 24 whose posture estimation is confirmed (No in step S373), the setting unit 123 confirms whether or not the current priority is set according to the priority condition (step S374). When the current priority is set according to the priority condition (Yes in step S374), it is determined that the priority setting change is unnecessary (step S377).

If the current priority is not set according to the priority condition (No in step S374), the setting unit 123 confirms whether or not the difference between the maximum number and the minimum number of the cumulative values is equal to or greater than the predetermined threshold value (step). S375). The process is the same as step S73 shown in FIG. The step S376 performed after this is the same as the step S74 shown in FIG. Therefore, these explanations will be omitted.

Returning to FIG. 13, when it is determined in step S37 that the priority is to be changed (Yes in step S37), the process proceeds to step S38. If it is determined in step S37 that the priority change is unnecessary (No in step S37), the process proceeds to step S40.

The processing of steps S38 to S40 is the same as the processing of steps 8 to S10 shown in FIG. Therefore, the description thereof will be omitted. In addition, when changing the priority setting, if a new priority condition occurs, the point of discarding the previous priority condition and setting the priority with the new priority condition is the first implementation. Different from the form.

FIG. 15 is a diagram for explaining the setting of the priority in the posture estimation device 1A of the second embodiment. The setting of the priority in the posture estimation device 1A will be further described based on the specific example shown in FIG. In the example shown in FIG. 15, priority conditions are stored in the storage unit 13 so as to prioritize the estimation processing of the back camera 22 and the left side camera 23. In FIG. 15, the implementation means that the posture estimation process is executed, and the non-execution means that the posture estimation process is not executed. The cumulative value shown in FIG. 15 is the cumulative value of the number of acquired posture estimation results. In the example shown in FIG. 15, when the cumulative value reaches 80 (= predetermined multiple times) or more, the posture estimation is confirmed.

In the first time, the priority of the posture estimation processing of the back camera 22 and the left side camera 23 is raised by the priority condition stored in the storage unit 13. As a result, in the first time, the order of the front camera 21 and the right side camera 24 is skipped, and the estimation process is performed in the order of the back camera 22 and the left side camera 23. The cumulative value (= 30) of the back camera 22 and the cumulative value (= 10) of the left side camera 23 after the estimation process have not reached 80 or more, and the posture estimation is not confirmed in any of the cameras. .. In addition, the current priority setting is based on the priority condition. Therefore, in the priority change determination process performed after the estimation process of the back camera 22 and the left side camera 23, if the occurrence of a new priority condition is not recognized, it is determined that the priority setting change is unnecessary. In this example, no new priority condition has occurred at this stage, and the priority setting is not changed.

In this example, at the end of the first time, the difference between the cumulative values is large between the back camera 22 and the left side camera 23. In order to eliminate this difference, in the second time, the estimation process of the left side camera 23 may be prioritized (the order of the back camera 22 is skipped). That is, the priority may be changed when the difference between the maximum number and the minimum number of the cumulative values of the number of acquired estimation results obtained for each camera targeted for priority processing becomes equal to or greater than a predetermined threshold value.

In the second time, as in the first time, the estimation process is performed in the order of the back camera 22 and the left side camera 23. Again, the cumulative values of the back camera 22 and the left side camera 23 after the estimation process have not reached 80 or more, and the posture estimation is not finalized in any of the cameras. In addition, the current priority setting is based on the priority condition. Therefore, in the priority change determination process performed after the estimation process of the back camera 22 and the left side camera 23, if the occurrence of a new priority condition is not recognized, it is determined that the priority setting change is unnecessary. In this example, a new priority condition arises at this stage. Therefore, it is determined that the priority setting needs to be changed. The new priority condition is a condition generated by obtaining information from the navigation device that there is a plan to turn left in the future, and specifically, a condition in which the estimation process of the left side camera 23 is prioritized. The priority setting is changed due to the occurrence of a new priority condition. As a result, in the next time (third time), only the estimation processing of the left side camera 23 is performed.

In the third time, the estimation process is performed only on the left side camera 23. The cumulative value of the left side camera 23 after the estimation process has not reached 80 or more, and the posture estimation is not finalized. In addition, the current priority setting is based on the priority condition. Therefore, in the priority change determination process performed after the estimation process of the left side camera 23, if the occurrence of a new priority condition is not recognized, it is determined that the priority setting change is unnecessary. In this example, no new priority condition has occurred at this stage, and the priority setting is not changed.

In the fourth time, the estimation process is performed only on the left side camera 23. The cumulative value of the left side camera 23 after the estimation process has reached 80 or more, and the posture estimation is confirmed. Therefore, in the priority change determination process performed after the estimation process of the left side camera 23, it is determined that the priority setting change is necessary. For the left side camera 23 whose posture estimation is confirmed, the priority setting is lowered and the next (fifth) turn is skipped. Further, by confirming the attitude estimation of the left side camera 23, the priority condition does not exist. As a result, the result of whether or not the difference between the maximum number (= 85) and the minimum number (= 0) of the cumulative values obtained for each of the in-vehicle cameras 21 to 24 is equal to or greater than a predetermined threshold value is taken into consideration, and based on this. Priority is decided. As a result, the priority of the back camera 22 having a large cumulative value is lowered, and the next turn is skipped.

In the fifth time, the estimation process is performed in the order of the front camera 21 and the right side camera 24. The cumulative values of the front camera 21 and the right side camera 24 after the estimation process have not reached 80 or more, and the posture estimation has not been confirmed except for the left side camera 23 which has been determined earlier. Therefore, even after this, the estimation process for determining the posture estimation is continued according to the flowchart shown in FIG. 13 and the like described above.

<3. Notes>
The configurations of the embodiments and modifications in the present specification are merely examples of the present invention. The configurations of the embodiments and modifications may be appropriately changed without exceeding the technical idea of the present invention. Moreover, a plurality of embodiments and modifications may be carried out in combination to the extent possible.

The method of estimating the posture of the camera in the above is only an example. The method of estimating the posture of the camera may be changed as appropriate. The posture estimation process of the camera may be, for example, a configuration obtained by comparing the movement of the feature points extracted from the captured image with the movement information (movement amount and speed) of the own vehicle.

The present invention can be used to estimate the posture of a camera that assists driving a moving object such as automatic parking. Further, the present invention can be used to estimate the posture of a camera that records driving information of a drive recorder or the like. Further, the present invention can be used as a correction device or the like for correcting a captured image by using the estimation information of the posture of the camera. Further, the present invention can be used for a device or the like that operates in cooperation with a center provided so as to be able to communicate with a plurality of mobile bodies via a network. The device may have a configuration in which, for example, when transmitting a captured image to the center, the estimated information on the posture of the camera is transmitted as a set with the captured image. Then, at the center, various image processing (processing of changing the viewpoint / viewing direction of the image in consideration of the camera posture, for example, conversion of the viewpoint / viewing direction to an image in the front direction of the vehicle body) using the estimation information of the camera posture. Generate useful data to users by performing correction processing for camera posture in measurement processing using images, statistical processing for secular change of camera posture (data of many vehicles), etc. And so on.

1, 1A Posture estimation device 21 to 24 On-board camera 11 Acquisition unit 121 Estimate unit 122 Confirmation unit 123 Setting unit 124 Judgment unit

Claims (9)

An acquisition unit that acquires captured images from each of multiple cameras mounted on the moving body,
An estimation unit that performs estimation processing for estimating the posture of the camera based on the captured image, and
A setting unit that sets the priority of the estimation process for each camera based on the number of estimation results acquired by the estimation process performed for each camera.
A posture estimation device. The posture estimation device according to claim 1, wherein the setting unit sets the priority based on a comparison result of the cumulative values of the acquired numbers for each camera. The posture estimation according to claim 1 or 2, further comprising a confirmation unit for determining the posture estimation based on the plurality of estimation results for the camera in which the cumulative value of the acquired number reaches a predetermined plurality of times or more. apparatus. The posture estimation device according to claim 3, wherein the setting unit changes the priority when the camera whose posture estimation is confirmed is generated. The setting unit changes the priority when the difference between the maximum number and the minimum number of the cumulative values of the acquired numbers obtained for each camera becomes a predetermined threshold value or more, any one of claims 1 to 4. The posture estimation device described in 1. The posture estimation device according to claim 5, wherein the setting unit changes the priority when the difference between the maximum number and the minimum number becomes smaller than the predetermined threshold value after becoming equal to or larger than the predetermined threshold value. A determination unit for determining whether or not a camera having a priority condition other than the acquired number exists among the plurality of cameras is further provided.
The posture estimation device according to any one of claims 1 to 4, wherein the setting unit sets the priority based on the priority condition when a camera having the priority condition exists. The setting unit confirms whether or not to change the priority at a predetermined timing, and confirms whether or not the priority is changed.
The predetermined timing occurs every time the order of the estimation processing is given to all of the plurality of cameras once.
Claims 1 to 7 in which, when there is a camera in which the order is skipped due to the priority setting among the plurality of cameras, the predetermined timing is determined by assuming that the order has been given to the camera. The posture estimation device according to any one of the above items. The acquisition process of acquiring captured images from each of multiple cameras mounted on the moving body,
An estimation process in which the estimation process for estimating the posture of the camera is performed based on the captured image, and
A setting step of setting the priority of the estimation process for each camera based on the number of acquisitions of the estimation result acquired by the estimation process performed for each camera.
A posture estimation method.

Images