JP2009285752A - Mobile robot - Google Patents

Abstract

An object of the present invention is to provide a mobile robot capable of increasing the number of samplings for processing an image while reducing the influence of image blur from the result of image processing during movement.
A robot R includes an image control processing unit 10 that processes an image captured by a camera C, a main storage unit 30 that stores action plan information including information on a scheduled landing time of a leg R3, and action plan information. A movement control unit 42 that outputs a command for controlling the movement of the robot to the autonomous movement control unit 50, and a movement plan information notification unit 43 that notifies the image control processing unit 10 of information on the planned landing time. The control processing unit 10 includes an image capture plan processing unit 11 that determines the timing before and after landing as the image capture timing from the camera C based on the scheduled landing timing, and capture when the image capture timing is reached. A capture trigger generation unit 13 that generates a trigger and outputs the trigger to the camera C, and an image recognition processing unit 16 that processes an image captured at the time of image capture are provided.
[Selection] Figure 3

Description

  The present invention relates to a mobile robot that can move autonomously, and more particularly to a mobile robot having a visual sensor that captures an image of the outside world and an image control processing unit that processes the image captured by the visual sensor. .

Conventionally, walking type (leg type) and wheel type mobile robots are known as mobile robots that can move autonomously (see, for example, Patent Document 1). The legged mobile robot described in Patent Document 1 is a dog-shaped pet robot that can walk on four legs, and includes a visual sensor (video camera) that captures an image of the outside world at the position of the nose of the head. By using the image recognition result of the ball photographed by the camera and keeping track of the distance from the ball, it moves like a pet. The pet robot has no problem in determining the distance to the ball when it is stationary, but it requires a large center of gravity movement to the left and right while walking as a repetitive movement, so the head and torso are in the roll direction. Shake occurs. Therefore, when the pet robot tilts to the left with a walking motion, it captures the ball that should be in the center of the screen at the upper right side of the screen, and when it tilts to the right with a walking motion, the ball that should originally be in the center of the screen Will end up shooting. If the ball position differs depending on the timing of shooting, the distance cannot be accurately determined. Therefore, this pet robot controls the video camera to reduce the influence of the swing in the roll direction by capturing the image of the ball only when the center of gravity is at the center of the left and right during walking, for example. By using the result of the above processing for calculating the distance, the distance to the ball is determined without stopping. The pet robot usually captures an image based on a walking cycle of 0.5 seconds or more (that is, a cycle until the center of gravity is restored by tilting to the right or left side from the center of the left and right, for example) The distance from the ball is maintained by the magnitude of the value and the predetermined value.
Japanese Patent Laying-Open No. 2005-161475 (paragraphs 0034-0043, FIG. 4)

  However, since the mobile robot (pet robot) described in Patent Document 1 captures one image with a walking cycle of 0.5 seconds or more, the position of the object (ball) to be recognized changes drastically with time. Cannot fully capture images. Therefore, even if it is attempted to increase the number of images to be captured without changing the walking cycle, the mobile robot described in Patent Document 1 requires a large center of gravity movement to the left and right during walking. The increased image has no periodicity and cannot be used for distance determination. For this reason, the movement performed by the mobile robot has to be restricted.

  In addition, in a conventional mobile robot that walks without moving the center of gravity to the left and right, if the number of captured images is simply increased, there are the following problems. That is, for example, when the tip of a leg of a legged mobile robot lands or when a wheeled robot passes through a step, the visual sensor vibrates due to the impact, so an image acquired by the vibrated visual sensor The blur will be larger. When an image having a large blur is used, the angular resolution and the recognition accuracy of the target object as image processing results are lowered.

  Accordingly, the present invention aims to solve the above-described problems and provide a mobile robot that can increase the number of times of sampling for processing an image while reducing the influence of image blurring from the result of image processing during movement. And

  The present invention was created to achieve the above object, and the mobile robot according to claim 1 of the present invention includes a visual sensor that captures an image of the outside world, and an image captured by the visual sensor. Including an image control processing unit to be processed, autonomous moving means having a function of moving autonomously, and timing information estimated that a value indicating blurring of an image captured by the visual sensor is larger than a predetermined value In the mobile robot, comprising: a storage unit that stores movement plan information; and a movement control unit that outputs a command for controlling movement of the mobile robot based on the movement plan information to the autonomous movement unit. A movement plan information notifying unit for notifying the image control processing unit of the plan information, and the image control processing unit uses the visual sensor based on the movement plan information. Value indicating the image blur captured Te is at a timing that is estimated to be less than a predetermined value, characterized by processing the image captured by the visual sensor.

  According to such a configuration, the mobile robot receives the movement plan information including timing information estimated by the movement plan information notification unit as a value indicating blurring of the image captured by the visual sensor is greater than a predetermined value. The control processing unit is notified, and the image control processing unit processes an image estimated to have a value indicating blurring of the captured image equal to or less than a predetermined value. Therefore, the mobile robot can reduce the influence of image blur from the image processing result during movement. Here, the autonomous moving means having the function of moving autonomously includes, for example, those using legs, wheels, endless tracks, and the like. The timing at which the value indicating the blur of the captured image is estimated to be larger than a predetermined value is determined in advance according to, for example, the operation purpose, operation function, performance, type of autonomous moving means, etc. of the mobile robot. Required by the rules established.

  The mobile robot according to claim 2 is the mobile robot according to claim 1, wherein the autonomous movement means includes a leg portion and an autonomous movement control unit that drives the leg portion. The timing information included in the movement plan information is information on the planned landing time of the leg, and the movement plan information notification unit sends the information on the planned landing time of the leg to the image control processing unit. A scheduled landing time notification unit for notifying, and the image control processing unit blurs an image captured by the visual sensor before and after the tip of the leg is landed based on information on the planned landing time of the leg. The image captured by the visual sensor is processed at a timing when the indicated value is estimated to be equal to or less than a predetermined value.

  According to such a configuration, the mobile robot notifies the image control processing unit of the information on the planned landing time of the leg by the planned landing time notification unit, and before and after the tip of the leg is landed by the image control processing unit. Then, an image at a timing when it is estimated that a value indicating blurring of the image captured by the visual sensor is equal to or less than a predetermined value is processed. Therefore, the mobile robot can reduce the influence of image blur from the image processing result during movement. Here, the movement by the leg means that the tip of the leg performs each operation of landing, sinking, and kicking up, for example, walking, running, and stepping on the spot. Further, the landing time can be classified as, for example, the moment of landing, the sinking period, and the kicking period. The timing before and after the tip of the leg lands is preferably between the landing timing and the next landing timing, but may be a timing at which the landing timing switches from the sinking period to the kicking period. Further, when a plurality of timings are used for processing the captured image, it is preferable that the intervals be equal.

  The mobile robot according to claim 3 is the mobile robot according to claim 2, wherein the image control processing unit sets a time when the image is captured by the visual sensor based on the scheduled landing time. An image capture plan processing unit for determining an image capture time to be displayed, an image capture time determination unit for determining whether or not the image capture time has come, and a capture trigger when the image capture time has come And a capture trigger generation unit that outputs the generated image to the visual sensor and an image processing unit that processes an image captured at the image capture time.

  According to such a configuration, the mobile robot determines the image capture time based on the planned landing time by the image capture plan processing unit, and when the image capture time comes, the capture trigger generation unit Output capture trigger to visual sensor. Therefore, it is possible to reduce blurring in the captured image by determining the image capture time so as to capture the images before and after the tip of the leg portion lands. Here, when the mobile robot performs, for example, biped walking, when one leg of the leg portion supports the skeleton and the other leg is not in contact with the ground, 1 before landing the other leg. The above timing can be set as the image capture time. Subsequently, one or more timings before landing one leg can be set as the image capture timing. Similarly, one or more timings before alternately landing on the foot can be set as the image capture timing.

  Further, the mobile robot according to claim 4 is the mobile robot according to claim 3, wherein the movement plan information is a value indicating blurring of an image previously associated with the information on the scheduled landing time. Including a correlation between a time at which the value is equal to or less than a predetermined value and the moving speed of the robot, and the movement plan information notification unit further includes a speed correlation notification unit that notifies the image control processing unit of the correlation. The image capture plan processing unit acquires the information on the scheduled landing time and the correlation, acquires the movement speed during the movement of the robot, and determines the correlation according to the acquired movement speed. And a time at which a value indicating blurring of the image is equal to or less than a predetermined value is determined as the image capturing time from the acquired information on the scheduled landing time.

  According to such a configuration, the mobile robot determines the image capture time based on the moving speed in addition to the planned landing time by the image capture plan processing unit, and captures when the image capture time comes. The trigger generation unit outputs a capture trigger to the visual sensor. Therefore, the mobile robot can reduce the blurring in the captured image by determining the image capture timing so that an image suitable for the moving speed is captured before and after the tip of the leg portion lands. Here, for example, when the moving speed is increased when the timing immediately before the tip of the leg lands is set as the image capture timing, the timing can be made earlier than the image capture timing before the speed change.

  The mobile robot according to claim 5 is the mobile robot according to claim 3 or 4, wherein the movement plan information indicates the posture of the visual sensor during the movement of the mobile robot. The movement plan information notification unit further includes a posture measurement value notification unit that notifies the image control processing unit of the measurement value, including a measurement value measured in advance as the posture information to be displayed, and the image capture plan processing unit Obtaining the estimated landing time information and the measured value, obtaining an estimated value of the posture of the visual sensor during movement estimated from the acquired measured value and the scheduled landing time information, and A time when the value falls within a predetermined range assuming that the value indicating the vibration of the visual sensor is equal to or less than a predetermined value is determined as the image capture time.

  According to this configuration, the mobile robot determines the image capture time based on the posture information of the visual sensor in addition to the planned landing time by the image capture plan processing unit, and when the image capture time comes Then, the capture trigger generator outputs the capture trigger to the visual sensor. The image capture plan processing unit captures the time when the estimated value of the posture information is a value that indicates the vibration of the visual sensor is equal to or less than a predetermined value at the timing before and after the tip of the leg of the planned landing time is landed. It is time to include. Therefore, the mobile robot can reduce blurring in the captured image by determining the image capture timing so that an image suitable for the posture of the visual sensor is captured before and after the tip of the leg is landed. it can. Here, the posture information can include, for example, information on the angular velocity (rotational angular velocity) of the pitch angle as the posture of the visual sensor. In the case of this angular velocity, the image capture timing can be determined so that the absolute value thereof is not more than a predetermined value.

  The mobile robot according to claim 6 is the mobile robot according to claim 2, wherein the image control processing unit starts image blurring from a moment when a tip of the leg portion lands. A buffer timing discriminating unit for discriminating whether or not a buffer timing indicating a predetermined time range estimated to be greater than a predetermined value is reached; and the image periodically captured by the visual sensor And an image processing unit for processing an image captured at a time other than the buffering time.

  According to this configuration, when the mobile robot periodically captures an image with the visual sensor and the buffer time determination unit determines that the buffer time is not estimated to be greater than a predetermined value, the image blur is greater than a predetermined value. In addition, the image processing unit processes an image periodically captured from the visual sensor. Therefore, the mobile robot can reduce the influence of image blur from the result of image processing during movement.

  The mobile robot according to claim 7 is the mobile robot according to claim 6, wherein the buffer timing determination unit acquires the movement speed during the movement of the robot and moves the robot. A buffering time corresponding to the acquired moving speed is selected from a plurality of buffering time candidates determined to be shorter as the speed is higher, and it is determined whether or not the selected buffering time is reached. The result is output to the image processing unit.

  According to such a configuration, the mobile robot selects a buffering time according to the moving speed of the robot from among a plurality of buffering time candidates determined by the buffering time determination unit so as to decrease as the moving speed increases. . Therefore, the mobile robot can reduce the influence of image blurring from the image processing result corresponding to the moving speed.

  According to the present invention, the mobile robot uses an image captured by a visual sensor as an image of the outside world using the movement plan information possessed in order to generate a command that the movement control unit outputs to the autonomous moving means. Since the image is processed at the timing when it is estimated that the value indicating the image blur is equal to or less than the predetermined value, it is possible to reduce the influence of the image blur from the image processing result during movement. In addition, the mobile robot can increase the number of samplings for processing an image while reducing image blurring during movement. As a result, the mobile robot can accurately determine the object in the image and the distance information to the object in the image by image processing even during high-speed movement.

  Hereinafter, the best mode (hereinafter referred to as “embodiment”) for carrying out the mobile robot (hereinafter referred to as “robot”) of the present invention with reference to the drawings will be described in detail in the first embodiment and the second embodiment. Explained. First, the overall configuration of the robot according to the first and second embodiments of the present invention will be described, and then the main part of the robot according to the first embodiment and the main part of the robot according to the second embodiment will be sequentially described.

(Whole structure of robot)
FIG. 1 is a block diagram schematically illustrating the configuration of the robot according to the first and second embodiments of the present invention, and FIG. 2 is a diagram schematically illustrating the appearance thereof. Here, an autonomous mobile legged biped robot will be described as an example. The robot R executes a task in accordance with an execution command input from the robot control device 1. The tasks include, for example, a task (guidance task) for guiding a visitor (person) H to a predetermined place such as a conference room, a task for delivering a package to a person (package delivery task), and the like. Exists. FIG. 2 shows a state where the robot R is executing a guidance task in a task execution area such as a hallway.

  As shown in FIG. 2, the robot R has a head R1, an arm R2, a leg R3, a torso R4, and a rear housing R5, and the head R1 and arms connected to the torso R4, respectively. The part R2 and the leg part R3 are each driven by an actuator (drive means), and the bipedal walking is controlled by the autonomous movement control part 50 (see FIG. 1).

  In addition to the head R1, the arm R2, the leg R3, the trunk R4, and the rear housing part R5, the robot R has cameras C, C, images as shown in FIG. It includes a control processing unit 10, a voice processing unit 20, a main storage unit 30, a main control unit 40, an autonomous movement control unit 50, a wireless communication unit 60, a target detection unit 70, a surrounding state detection unit 80, and a posture detection unit 90. In the robot R, the autonomous movement control unit 50 and the leg R3 constitute an autonomous moving means.

  Cameras (visual sensors) C and C can capture, as digital data, images of the robot R moving in the forward direction as external images. For example, a color CCD (Charge-Coupled Device) camera is used. The The cameras C and C are arranged side by side in parallel on the left and right, and the captured image is output to the image control processing unit 10. The cameras C and C are disposed inside the head R1.

  The image control processing unit 10 processes images (captured images) captured by the cameras C and C, and recognizes surrounding obstacles and persons in order to grasp the situation around the robot R from the captured images. is there. Details of the image control processing unit 10 will be described later.

  The voice processing unit 20 outputs voice to a speaker (not shown) based on an utterance command from the main control unit 40, or generates character information (text data) from voice data input from a microphone (not shown). Or output to 40.

  The main storage unit 30 is composed of, for example, a general hard disk or the like, and stores necessary information (local map data, conversation data, etc.) transmitted from the robot control device 1. In addition, as will be described later, the main storage unit 30 stores information necessary for performing various operations of the main control unit 40. The main storage unit 30 stores, for example, movement plan information including timing information estimated that a value indicating blurring of images captured by the cameras C and C is larger than a predetermined value. Here, the timing at which the value indicating the blur of the captured image is estimated to be larger than a predetermined value is determined in advance according to, for example, the operation purpose, operation function, performance, type of autonomous moving means of the robot R, and the like. Required by the rules established. The timing estimation itself may be performed in advance before the movement, or may be performed during the movement based on a detection result of some information according to a predetermined rule. In the present embodiment, information on the scheduled landing time of the leg portion R3 is included in the movement plan information as information on the timing at which the value indicating the blur of the captured image is estimated to be larger than a predetermined value.

  The main control unit 40 controls the image control processing unit 10, the sound processing unit 20, the main storage unit 30, the autonomous movement control unit 50, the wireless communication unit 60, the target detection unit 70, the surrounding state detection unit 80, and the posture detection unit 90. It is something to control. The main control unit 40 includes, for example, control for communicating with the robot control device 1, control for executing a predetermined task based on a task execution command acquired from the robot control device 1, and using the robot R as a destination. In order to perform control and the like for movement, various determinations are made and commands for the operation of each unit are generated.

  The autonomous movement control unit 50 drives the head R1, the arm R2, the leg R3, and the trunk R4 in accordance with instructions from the main control unit 40. Although not shown, the autonomous movement control unit 50 includes a neck control unit that drives the neck joint of the head R1, a hand control unit that drives the finger joint at the tip of the arm R2, and a shoulder joint of the arm unit R2. , Arm control unit for driving the elbow joint and wrist joint, waist control unit for rotating the torso R4 horizontally with respect to the leg R3, foot control unit for driving the hip joint, knee joint and ankle joint of the leg R3 have. The neck control unit, hand control unit, arm control unit, waist control unit, and foot control unit output drive signals to actuators that drive the head R1, arm R2, leg R3, and torso R4.

  The wireless communication unit 60 is a communication device that transmits and receives data to and from the robot control device 1. The wireless communication unit 60 selects and uses near field communication such as a public line such as a mobile phone line or a PHS (Personal Handyphone System) line or a wireless LAN compliant with the IEEE802.11b standard.

  The target detection unit 70 detects whether or not there is a person with the tag T around the robot R. For example, the object detection unit 70 emits infrared light from the LED and receives a reception report signal generated by the tag T that has received the infrared light. Specify whether there is a person to prepare for. In addition, the target detection unit 70 specifies the distance to the person including the tag T based on the radio wave intensity of the reception report signal acquired from the tag T.

  The peripheral state detection unit 80 detects the peripheral state of the robot R, analyzes the image obtained by irradiating a slit-like laser beam or infrared ray toward the search area of the floor surface, and images the floor surface to analyze the road surface state. Is detected. In addition, the peripheral state detection unit 80 detects the mark M (see FIG. 2) by analyzing the image when the search area on the floor surface is irradiated with infrared rays, and detects the mark M from the position (coordinates) of the detected mark M. And the relative positional relationship between the robot R and the robot R are calculated.

  The posture detection unit 90 includes, for example, a gyro sensor that detects the direction in which the robot R is facing, and a GPS (Global Positioning System) receiver that acquires the position coordinates of the robot R on a preset map. Have. Data detected by the posture detection unit 90 is output to the main control unit 40 and used to determine the behavior of the robot R.

(First embodiment)
Next, the main part of the robot according to the first embodiment will be described with reference to FIG. 3 (see FIGS. 1 and 2 as appropriate). FIG. 3 is a block diagram schematically showing the main part of the robot according to the first embodiment of the present invention.

<Configuration example of main control unit>
As shown in FIG. 3, the main control unit 40 includes an action generation unit 41, a movement control unit 42, and a movement plan information notification unit 43.

  The behavior generation unit 41 stores various programs (modules) for executing behavior patterns such as movement of the robot R and utterances, and refers to the main storage unit 30 when executing the behavior patterns. Information (local map data, conversation data, etc.) is reflected in the action pattern. The action generation unit 41 also manages an action plan for executing various modules according to a predetermined schedule. Examples of modules include a destination movement module, a local avoidance module, a delivery task execution module, a guidance task execution module, and a human correspondence module. The action generation unit 41 uses the image recognition result (recognized face position, etc.) and distance information acquired from the image control processing unit 10 as information when the robot R moves and with the recognized person. Use it to communicate and execute behavior patterns.

  Here, each module will be briefly described. The destination movement module performs route search (for example, search for a route between nodes) and movement from the current position of the robot R to a destination such as a task execution position in a predetermined task execution area. is there. This destination movement module obtains the shortest distance to the destination while referring to the map data and the current position. The local avoidance module searches for a detour path that avoids an obstacle when the surrounding state detection unit 80 detects an obstacle such as a cardboard in the traveling direction. The delivery task execution module operates when executing a package delivery task, and receives (holds) an article from a person (client) requesting transportation of the article, and delivers the received article to the recipient. The operation is performed (to release the article). The guidance task execution module executes, for example, a task for guiding a visitor who has visited a guidance start point in a task execution area to a predetermined position. The person handling module performs, for example, speech, posture change, vertical movement and gripping of the arm portion R2 based on a predetermined scenario when executing a package delivery task or a guidance task.

  The movement control unit 42 outputs a command for controlling the movement of the robot R to the autonomous movement control unit 50 based on the movement plan information stored in the main storage unit 30. In the present embodiment, the movement control unit 42 acquires movement plan information from the behavior generation unit 41 when the behavior generation unit 41 executes a behavior pattern involving the movement of the robot R, and issues a command to the autonomous movement control unit 50. put out. The movement plan information can include, for example, the movement speed (target speed) of the robot R, the position information of the destination, the route information to the destination, etc., in addition to the information on the scheduled landing time of the leg R3.

Here, the information on the scheduled landing time of the leg R3 can include, for example, the following (1) to (3) as time information. (1) The instant t ₁ when the tip (toe or foot or heel) of the leg R3 lands, (2) the time interval Δt from the current time t ₀ to the instant t ₁ when the tip of the leg R3 lands, (3 ) Both the landing instant t ₁ and the difference (landing cycle) up to the next landing instant t ₂ . In addition, the expected landing time of the information of the legs R3, as position information, e.g., corresponding to the instant t ₁ the lands may include a position coordinate of the next position of the tip of the leg R3. The next position of the tip of the leg R3 is determined by a gait plan that is one or two steps ahead.

  In the present embodiment, the movement plan information further includes a time when a value indicating blurring of images taken by the cameras C, C previously associated with the information on the scheduled landing time is equal to or less than a predetermined value, and the robot R The correlation with the moving speed (hereinafter referred to as “landing-speed” correlation) was included. Here, the “landing-speed” correlation is such that, for example, when walking at a speed of 2.7 [km / h] per hour, the time when the image blurring value is equal to or less than a predetermined value is P [ms] from the moment of landing. Later, when traveling at a speed of 6 km / h, the time when the image blurring value is equal to or less than a predetermined value is created in advance according to the moving speed, such as Q [ms] after the moment of landing. It is possible to calculate with a determined table or a predetermined decision function.

  In the present embodiment, the movement plan information further includes measurement values (hereinafter referred to as camera posture advance measurement values) measured in advance as posture information indicating the postures of the cameras C and C while the robot R is moving. Included. In this embodiment, information on the angular velocity of the pitch angle of the cameras C and C is used as the camera posture prior measurement value.

  The movement control unit 42 acquires information related to the operation state of the leg R3 from the autonomous movement control unit 50, and detects the current movement speed of the robot R. In addition, the movement control unit 42 acquires the image recognition result acquired from the image control processing unit 10 via the behavior generation unit 41, so that the movement of the robot R according to the relative distance to an object such as a person is obtained. Control to change the speed.

  Note that the movement control unit 42 controls the movement of the center of gravity by controlling not only the leg portion R3 but also the head portion R1 and the arm portion R2 by a command output to the autonomous movement control unit 50. The movement control unit 42 acquires information related to the posture of the robot R from the posture detection unit 90 and reflects it in a command output to the autonomous movement control unit 50, for example, to smoothly move the center of gravity during cornering traveling. .

  The movement plan information notification unit 43 notifies the image control processing unit 10 of movement plan information. In the present embodiment, the movement plan information notification unit 43 includes a scheduled landing time notification unit 43a, a speed correlation notification unit 43b, and a posture measurement value notification unit 43c, as shown in FIG. The scheduled landing time notification unit 43a notifies the image control processing unit 10 of information on the planned landing time of the leg R3. The speed correlation notification unit 43b notifies the image control processing unit 10 of the above-described “landing-speed” correlation. The posture measurement value notification unit 43c notifies the image control processing unit 10 of the camera posture preliminary measurement value. The movement plan information notification unit 43 acquires information about the current movement speed of the robot R from the movement control unit 42 and notifies the image control processing unit 10 of the information.

<Configuration Example of Image Control Processing Unit>
The image control processing unit 10 lands the tip of the leg R3 based on the information on the scheduled landing time of the leg R3 acquired from the movement plan information notification unit 43, the “landing-speed” correlation, and the camera posture premeasurement value. Images captured by the cameras C and C are processed at a timing other than the moment. The control of the cameras C and C differs between the first embodiment and the second embodiment. In the first embodiment, the cameras C and C are controlled to capture an image when receiving a trigger from the image control processing unit 10. In FIG. 3, the camera codes are denoted as C ₀ and C ₀ in order to distinguish them from the second embodiment. As shown in FIG. 3, the image control processing unit 10 includes an image capture plan processing unit 11, a capture time determination unit 12, a capture trigger generation unit 13, an image correction processing unit 14, and stereo image processing. Unit 15 and an image recognition processing unit 16.

  The image capture plan processing unit 11 determines an image capture time indicating the time when an image is captured by the cameras C and C based on the planned landing time. In the present embodiment, the image capture plan processing unit 11 determines the image capture time based on the moving speed of the robot R and the posture information of the cameras C and C in addition to the planned landing time.

  Specifically, in the present embodiment, the image capture plan processing unit 11 acquires information on the scheduled landing time and the “landing-speed” correlation as the movement plan information, and the robot R is moving during the movement. According to the acquired moving speed, the time at which the value indicating the blur of the image is equal to or less than a predetermined value is obtained from the acquired information on the scheduled landing time using the “landing-speed” correlation. It is determined as the capture time.

  Further, in this embodiment, the image capture plan processing unit 11 estimates the angular velocity of the pitch angle of the cameras C and C during the movement, which is estimated from the camera posture preliminary measurement value and the acquired information on the scheduled landing time. The time when the estimated value falls within a predetermined range assuming that the value indicating the vibration of the cameras C and C is equal to or less than a predetermined value is determined as the image capture time.

  Here, an example of the image capture time will be described. FIG. 4 is a diagram illustrating an example of a temporal change in the angular velocity of the head pitch angle. In the graph shown in FIG. 4, the horizontal axis represents time, and the vertical axis represents the detected value of angular velocity (rotational angular velocity) measured in advance by an angular velocity sensor (attitude detection sensor) when the head R1 of the robot R vibrates in the pitch direction. Show. As an example, an experimental result when the robot R walks at a moving speed of 2.7 [km / h] at a step length of 450 mm is shown. The robot R will land on one leg (for example, the left leg) when the time is 200 [ms], and land on the other leg (for example, the right leg) when the time is 800 [ms]. That is, the landing period is 600 [ms]. The landing cycle includes a period of a scheduled landing time at an early stage. In this embodiment, when the robot R moves forward, the time when the heel (tip) of the foot touches the ground is called the moment of landing. At this time, almost the entire bottom surface of the foot touches the floor after landing, and finally the toes Get out of bed. When the robot R moves backward, the moment when the toes (tips) of the feet touch the ground is the moment of landing. However, the present invention is not limited to this, and in the form in which almost the entire bottom surface of the foot is in contact with the floor simultaneously with the heel and toes, the time when the foot contacts the ground is the moment of landing.

  As shown in FIG. 4, the absolute value of the angular velocity is large because the vibration is large at the moment of landing. However, there is a timing at which the absolute value of the angular velocity becomes very small at the initial stage of the landing period. There are eight times when the angular velocity becomes 0, for example, in the landing cycle shown in FIG. At least one of these timings may be set to a time when the value indicating the vibration of the cameras C and C falls within a predetermined range as being a predetermined value or less. Further, the absolute value of the angular velocity is determined in advance without limiting the time when the value indicating the vibration of the cameras C and C is within a predetermined range as being equal to or less than a predetermined value to the time when the angular velocity is 0. It is possible to determine a larger number of image capture timings so as to be less than or equal to the given value. Further, when determining a plurality of times of image capture, for example, it is preferable to set substantially equal intervals as image capture timings as indicated by reference numerals 301, 302, and 303, for example. In particular, the timing indicated by 303 shows the timing immediately after the landing of one foot and immediately before the landing of the other foot, and is suitable for the case where the moving speed is increased and the landing cycle is shortened. Is most preferable as the image capture time. When the robot R travels with a stride of 525 mm and a moving speed of 6 km / h, there is a jump period of 80 [ms] as a period for getting out of both feet, and the floor reaction force is applied during the jump period. Therefore, it is preferable to set one or more image capture timings during this period.

  The capture time determination unit 12 shown in FIG. 3 determines whether or not the image capture time determined by the image capture plan processing unit 11 has come. The determination result is output to the capture trigger generation unit 13.

  The capture trigger generation unit 13 generates a trigger (capture trigger) for notifying the cameras C and C when the image capture timing is reached and causing the cameras C and C to shoot and outputs the trigger to the cameras C and C. It is. In the present embodiment, when the capture trigger generation unit 13 outputs the capture trigger to the cameras C and C, the capture trigger generation unit 13 notifies the image recognition processing unit 16 of the processing timing with another trigger at the same time.

  The image correction processing unit (image processing unit) 14 processes an image captured at the image capturing time. The image correction processing unit 14 corrects distortion by the lenses of the cameras C and C from the images captured by the cameras C and C and corrects the epipolar lines of the cameras C and C to the scanning lines of the respective camera images (rectify). ), The captured image is corrected and output to the stereo image processing unit 15. Lens distortion correction can be performed, for example, by photographing a plurality of straight lines and approximating the deviation from the center with an approximate expression of a fifth order function so that the lines in the image become straight lines. The image correction processing unit 14 also includes internal parameters (such as the focal length of the lens and the center position of the lens) and external parameters (such as the main point position of the lens with respect to the 3D reference point outside the cameras C and C). Is used to calculate the rectify using that information.

  Here, an example of image correction will be briefly described. FIG. 5 shows an example of an image obtained by photographing a checkered pattern and an image obtained by correcting the distortion of the image. When the checkered pattern drawn on the plane is photographed from the front by the cameras C and C, as shown in FIG. 5A, before the correction, the barrel-shaped distortion occurs in the left camera image 200L and the right camera image 200R. Yes. In FIG. 5A, a straight line 210 indicates the same scanning line of the left camera image 200L and the right camera image 200R. In the right camera image 200R, the lower edge of the mark 220 on the checkered pattern matches the straight line 210, but in the left camera image 200L, the lower edge of the mark 220 does not match the straight line 210. When such image distortion is corrected, as shown in FIG. 5B, after correction, barrel distortion is removed from the left camera image 200L and the right camera image 200R. In FIG. 5B, a straight line 230 that is the same scanning line of the left and right images coincides with the lower edge of the mark 220 in the left and right images.

  A stereo image processing unit (image processing unit) 15 shown in FIG. 3 processes an image captured at the image capturing time. As a stereo process, the stereo image processing unit 15 performs pattern matching on the basis of one of the two images taken by the left and right cameras C and C, calculates the parallax of each corresponding pixel in the left and right images, and produces a parallax. An image is generated, and the generated parallax image and the original image are output to the image recognition processing unit 16. This parallax represents the distance from the robot R to the photographed object. Further, the stereo image processing unit 15 outputs the result of calculating the parallax of each corresponding pixel in the left and right images to the main control unit 40 as distance information.

  The image recognition processing unit (image processing unit) 16 processes an image captured at the image capture time. Based on the data output from the stereo image processing unit 15, the image recognition processing unit 16 extracts a moving object (moving body) in the captured image, and estimates that the moving body is a person. Recognize In order to extract a moving object, the image recognition processing unit 16 stores images of several past frames (several frames), and compares the latest frame (image) with the past frame (image). Then, pattern matching is performed, the movement amount of each pixel is calculated, and a movement amount image is generated. Then, from the parallax image and the movement amount image, when there is a pixel having a large movement amount within a predetermined distance range from the cameras C and C, it is estimated that there is a person, and as a parallax image of only the predetermined distance range Extract the moving body. Further, the image recognition processing unit 16 recognizes, for example, the face area and the position of the face from the extracted size and shape of a part of the moving body, and outputs the result to the main control unit 40 as an image recognition result.

<Robot motion>
The operation of the robot R according to the first embodiment will be described with reference to FIG. 6 (refer to FIG. 3 as appropriate) mainly focusing on the operation of the image control processing unit 10. FIG. 6 is a flowchart showing the operation of the image control processing unit shown in FIG. The image control processing unit 10 acquires the scheduled landing time of the leg R3 from the movement plan information notification unit 43 together with the movement speed of the robot R and the posture information of the cameras C and C (step S1). The processing unit 11 determines the image capture time (step S2). Then, the image control processing unit 10 determines the capture time by the capture time determination unit 12 (step S3), and when the image capture time comes, the capture trigger generation unit 13 sets the capture trigger. Output to the cameras C and C (step S4). As a result, the cameras C and C acquire the video (captured image) of the robot R in the forward movement direction and output it to the image control processing unit 10 (step S5).

  Then, the image control processing unit 10 corrects the captured image by performing distortion correction by the camera lens and rectification from the captured image by the image correction processing unit 14 (step S6). Subsequently, the image control processing unit 10 performs stereo processing on the corrected image by the stereo image processing unit 15 (step S7). Then, the image control processing unit 10 recognizes an object (person) as a moving body using the parallax image and the original corrected image by the image recognition processing unit 16 (step S8), and the recognized object (person). Is output to the action generation unit 41 of the main control unit 40 as an image processing result (step S9). Further, the image control processing unit 10 outputs the distance information calculated by the stereo image processing unit 15 to the action generation unit 41 of the main control unit 40 following or in parallel with step S7 or step S9.

  According to the first embodiment, the robot R notifies the image control processing unit 10 of information on the planned landing time of the leg R3 by the movement plan information notification unit 43, and based on the planned landing time by the image control processing unit 10. The images are captured from the cameras C and C at the image capture time determined in the above. Therefore, by setting the image capture timing to the timing before and after the tip of the leg lands, it is possible to process an image that is estimated that a value indicating blurring of the captured image is equal to or less than a predetermined value. Therefore, the robot R can reduce the influence of image blur from the image processing result during movement. Further, the robot R can easily increase the number of times of sampling for processing the image.

(Second Embodiment)
Next, the main part of the robot according to the second embodiment will be described with reference to FIG. 7 (refer to FIGS. 1, 2 and 3 as appropriate). FIG. 7 is a block diagram schematically showing the main part of the robot according to the second embodiment of the present invention.

<Configuration example of main control unit>
In 2nd Embodiment, since the structure of the main control part 40 is the same as that of 1st Embodiment, the same code | symbol is attached | subjected to the same structure and description is abbreviate | omitted. However, the movement plan information notification unit 43A illustrated in FIG. 7 does not include the posture measurement value notification unit 43c illustrated in FIG. That is, in the present embodiment, the movement plan information notification unit 43A notifies the image control processing unit 10A of the “landing-speed” correlation and the current movement speed information of the robot R in addition to the scheduled landing time of the leg R3. To do.

<Configuration Example of Image Control Processing Unit>
As shown in FIG. 7, the image control processing unit 10A includes an image correction processing unit 14, a stereo image processing unit 15, an image recognition processing unit 16, a planned landing time acquisition unit 17, and a buffer time determination unit 18. I have. Among these, the image correction processing unit 14, the stereo image processing unit 15, and the image recognition processing unit 16 are the same as those in the first embodiment, and thus description thereof is omitted. However, in the second embodiment, the cameras C and C are controlled so as to capture images periodically. Therefore, the stereo image processing unit 15 and the image recognition processing unit 16 process images captured at a time other than the buffering time described later, among the images periodically captured by the cameras C and C. For example, the cameras C and C capture images at intervals of 60 [ms]. In FIG. 7, the camera codes are denoted as C ₁ and C ₁ in order to distinguish them from the first embodiment.

  The planned landing time acquisition unit 17 acquires information on the planned landing time as movement plan information from the movement plan information notification unit 43A of the main control unit 40 together with information on the current movement speed of the robot R, and a buffer time determination unit 18. Is output.

  Whether the buffer time determination unit 18 has reached a buffer time indicating a predetermined time range in which a value indicating image blur is estimated to be greater than a predetermined value starting from the moment when the tip of the leg R3 lands. It is to determine whether or not. In the present embodiment, the buffer time determination unit 18 acquires the movement speed during the movement of the robot R, and acquires the movement speed from the buffer time candidates determined so as to decrease as the movement speed of the robot R increases. A buffering time corresponding to the moving speed is selected, and the result of determining whether or not the selected buffering time is reached is output to the image correction processing unit 14 and the image recognition processing unit 16. Here, the buffer time candidates are associated with the moving speed in advance as a table, for example. For example, when the moving speed is 2.7 [km / h], the buffer time can be set to 3 [ms] after landing, for example. In the present embodiment, the buffer time determination unit 18 uses the “landing-speed” correlation acquired from the speed correlation notification unit 43b from the viewpoint opposite to that of the first embodiment to determine whether or not the buffer time has come. It was decided to.

<Robot motion>
The operation of the robot R according to the second embodiment will be described with reference to FIG. 8 (refer to FIG. 7 as appropriate) mainly with respect to the operation of the image control processing unit 10A. FIG. 8 is a flowchart showing the operation of the image control processing unit shown in FIG. The image control processing unit 10A acquires the planned landing time of the leg R3 together with the moving speed of the robot R from the movement plan information notification unit 43A by the planned landing time acquisition unit 17 (step S11). Further, the cameras C and C periodically acquire a video (captured image) of the robot R in the forward movement direction side and independently output it to the image control processing unit 10A independently of step S11 (step S12). In addition, the buffer time determination unit 18 of the image control processing unit 10A determines whether or not the buffer time has come, independently of step S11, and outputs the determination result to the image correction processing unit 14 and the image recognition processing unit 16. I keep doing it. At this time, if the moving speed acquired by the buffer time determination unit 18 together with the planned landing time changes in step S11, the buffer time determination unit 18 switches to the buffer time corresponding to the moving speed and continues the determination. As a result, when the buffer time comes, the image correction processing unit 14 and the image recognition processing unit 16 stop sampling as the buffer time for a predetermined period after landing (step S13). The following steps S14 to S17 executed by the image control processing unit 10A at times other than the buffer time are the same as the above-described steps S6 to S9, and thus description thereof is omitted.

  According to the second embodiment, the robot R periodically captures images with the cameras C and C, and notifies the image control processing unit 10A of information on the scheduled landing time of the leg R3 by the movement plan information notification unit 43A. Then, image processing of the captured image is performed at a time other than the buffering time determined by the image control processing unit 10A based on the scheduled landing time. That is, the robot R does not perform image processing on the image captured at the buffer time. Therefore, the robot R can reduce the influence of image blur from the image processing result during movement. Further, the robot R can easily increase the number of times of sampling for processing the image. Furthermore, the robot R does not need to output a capture trigger from the image control processing unit 10A to the cameras C and C.

  As mentioned above, although preferred embodiment of this invention was described, this invention is not limited to each above-described embodiment. For example, in each embodiment, the image control processing unit 10 (10A) is in the best mode including the image correction processing unit 14, the stereo image processing unit 15, and the image recognition processing unit 16 as the image processing unit. The correction processing unit 14 is not essential. In each embodiment, images are captured by two cameras C and C. However, three or more cameras can be used. For example, with nine cameras arranged in 3 rows and 3 columns, the position (distance information) of an object is detected from the other 8 cameras using the camera arranged in the center as a reference camera, and the 8 cameras. By taking the average at the positions, it is also possible to detect the position of the object more accurately.

  In each embodiment, information on the expected landing time of the leg R3 is used as the movement plan information as information on the timing at which it is estimated that the value indicating the blurring of the images captured by the cameras C and C is larger than a predetermined value. Although it is included, a plurality of types of the predetermined value may be set. In the first embodiment, the image capture plan processing unit 11 obtains an estimated value of the angular velocity of the pitch angle of the cameras C and C during the movement from the camera posture premeasured value and the information on the scheduled landing time. The estimated value is determined as the image capture time when the value indicating the vibration of the cameras C and C is equal to or less than the predetermined value. A plurality of types may be set.

  In the first embodiment, the image capture plan processing unit 11 is described as determining the image capture time based on the moving speed of the robot R and the posture information of the cameras C and C in addition to the planned landing time. However, the moving speed and the posture information of the cameras C and C are not essential for the present invention.

  In the first embodiment, the posture information of the cameras C and C has been described as information on the angular velocity (rotational angular velocity) of the pitch angle, but may include information on the angular velocity (rotational angular velocity) of the yaw angle or roll angle.

  In the second embodiment, the buffer time candidates are previously associated with the moving speed as a table, but may be obtained by calculation each time using a predetermined function with the moving speed as an argument. .

  In the second embodiment, the buffer time determination unit 18 has been described as determining whether or not the buffer time has come based on the moving speed of the robot R in addition to the scheduled landing time. Speed is not essential. For example, in addition to the scheduled landing time, it is also possible to determine whether or not the buffer time has come based on the road surface condition. In this case, the buffer time candidates are associated in advance as a table according to the road surface condition. Then, the robot R selects the buffer time according to the road surface state detected by analyzing the image obtained by capturing the floor surface by the peripheral state detection unit 80. Specifically, if the floor has more steps than a normal corridor or has a strong elasticity, the buffering time can be extended.

  In each embodiment, the robot R has been described as a legged biped robot. However, the present invention is not limited to this, and the robot R may be a walking robot that moves with one leg or multiple legs. Furthermore, as a moving mechanism, not only a leg part but a moving mechanism corresponding to the leg part, for example, a wheel or an endless track may be used. As a modification of the robot R of the first embodiment, for example, a case of a wheel type robot will be described with reference to FIG. 9 (refer to FIGS. 1 and 3 as appropriate).

  FIG. 9 is a block diagram schematically showing the main part of a modified example of the robot according to the first embodiment of the present invention. In addition, the same code | symbol is attached | subjected to the structure same as the structure shown in FIG. 3, and description is abbreviate | omitted. The moving mechanism R31 is composed of a plurality of wheels, for example, and is driven by the autonomous movement control unit 50. That is, in the wheel type mobile robot, the autonomous movement control unit 50 and the movement mechanism R31 constitute an autonomous movement means. Further, the movement control unit 42 of the main control unit 40 instructs the wheel rotation speed instead of instructing one step of the walking type (leg type). Moreover, the action production | generation part 41 can be provided with an obstruction passage module, for example. This obstacle passing module is configured to pass the step from the current time until it reaches the step when the surrounding state detection unit 80 (see FIG. 1) detects an obstacle such as a step in the traveling direction. Information (hereinafter referred to as step difference passage time) is calculated. The calculated step passage time is notified to the image control processing unit 10 as information on timing at which it is estimated that a value indicating blurring of images captured by the cameras C and C is larger than a predetermined value.

  The movement plan information notification unit 43B does not include the planned landing time notification unit 43a shown in FIG. Instead, the movement plan information notification unit 43B includes a step passage time notification unit 43d. The step passage time notification unit 43d acquires the step passage time calculated by the obstacle passage module included in the action generation unit 41 via the movement control unit 42 and notifies the image control processing unit 10 of the step passage time. Thereby, the image capturing plan processing unit 11 of the image control processing unit 10 determines the image capturing time indicating the time when the images are captured by the cameras C and C based on the step passing time instead of the planned landing time. . Note that the image capture plan processing unit 11 can determine the image capture time based on the moving speed of the robot R and the posture information of the cameras C and C in addition to the step passage time.

  According to the modification of the first embodiment, the wheel-type robot notifies the image control processing unit 10 of information on the step passing time by the movement plan information notifying unit 43B, and the step passing time by the image control processing unit 10. The images are captured from the cameras C and C at the image capture time determined based on the above. Therefore, by setting the timing before and after passing through the step as the image capture timing, it is possible to process an image that is estimated that a value indicating blurring of the captured image is equal to or less than a predetermined value. Therefore, the wheel type robot can reduce the influence of image blurring from the image processing result during movement, and can easily increase the number of times of sampling for processing the image.

It is a block diagram which shows typically the structure of the robot which concerns on 1st and 2nd embodiment of this invention. It is a figure which shows typically the external appearance of the robot which concerns on 1st and 2nd embodiment of this invention. It is a block diagram showing typically the principal part of the robot concerning a 1st embodiment of the present invention. It is a figure which shows an example of the time change of the angular velocity of the pitch angle of a head. It is a figure which shows an example of the picked-up image before and after image correction. 4 is a flowchart illustrating an operation of an image control processing unit illustrated in FIG. 3. It is a block diagram which shows typically the principal part of the robot which concerns on 2nd Embodiment of this invention. It is a flowchart which shows operation | movement of the image control process part shown in FIG. It is a block diagram which shows typically the principal part of the modification of the robot which concerns on 1st Embodiment of this invention.

Explanation of symbols

R robot (mobile robot)
R1 Head R2 Arm R3 Leg R31 Movement mechanism R4 Torso R5 Back storage 1 Robot controller 10 (10A) Image control processor 11 Image capture plan processor 12 Capture timing discriminator 13 Capture trigger generator 14 Image correction processing unit (image processing unit)
15 Stereo image processing unit (image processing unit)
16 Image recognition processing unit (image processing unit)
17 Scheduled landing time acquisition unit 18 Buffer timing determination unit 20 Audio processing unit 30 Storage unit 40 Main control unit 41 Action generation unit 42 Movement control unit 43 (43A, 43B) Movement plan information notification unit 43a Scheduled landing time notification unit 43b Speed correlation Notification unit 43c Posture measurement value notification unit 43d Step passage time notification unit 50 Autonomous movement control unit 60 Wireless communication unit 70 Target detection unit 80 Peripheral state detection unit 90 Posture detection unit C (C ₀ , C ₁ ) Camera (visual sensor)

Claims (7)

A visual sensor that captures an image of the outside world, an image control processing unit that processes the image captured by the visual sensor, an autonomous moving means having a function of autonomously moving, and blurring of the image captured by the visual sensor A storage unit for storing movement plan information including timing information estimated to be greater than a predetermined value; and a command for controlling movement of the mobile robot based on the movement plan information. In a mobile robot having a movement control unit that outputs to
A movement plan information notification unit for notifying the movement plan information to the image control processing unit;
The image control processing unit processes the image captured by the visual sensor at a timing when it is estimated that a value indicating blurring of the image captured by the visual sensor is equal to or less than a predetermined value based on the movement plan information. A mobile robot characterized by The autonomous moving means has a leg part and an autonomous movement control part that drives the leg part,
The timing information included in the movement plan information is information on the planned landing time of the legs,
The movement plan information notification unit includes a planned landing time notification unit that notifies the image control processing unit of information on the planned landing time of the legs,
The image control processing unit estimates that a value indicating blurring of an image captured by the visual sensor before and after the tip of the leg is landed is less than or equal to a predetermined value based on information on a planned landing time of the leg. 2. The mobile robot according to claim 1, wherein an image captured by the visual sensor is processed at a timing. The image control processing unit
Based on the scheduled landing time, an image capturing plan processing unit for determining an image capturing time indicating a time for capturing an image by the visual sensor;
An image capture time determination unit for determining whether or not the image capture time has come;
A capture trigger generation unit that generates a capture trigger and outputs to the visual sensor when the image capture time comes,
The mobile robot according to claim 2, further comprising: an image processing unit that processes an image captured at the image capture time. The movement plan information includes a correlation between a time at which a value indicating blurring of an image associated with the information on the scheduled landing time in advance is equal to or less than a predetermined value, and a movement speed of the robot,
The movement plan information notification unit further includes a speed correlation notification unit that notifies the image control processing unit of the correlation.
The image capture plan processing unit acquires the information on the scheduled landing time and the correlation, acquires the movement speed during movement of the robot, and determines the correlation according to the acquired movement speed. 4. The movement according to claim 3, wherein a time at which a value indicating blurring of the image is equal to or less than a predetermined value is determined as the image capture time from the acquired information on the scheduled landing time. Type robot. The movement plan information includes a measurement value measured in advance as posture information indicating the posture of the visual sensor during movement of the mobile robot,
The movement plan information notification unit further includes a posture measurement value notification unit that notifies the image control processing unit of the measurement value,
The image capture plan processing unit acquires the information on the planned landing time and the measured value, and also determines the posture of the visual sensor during movement estimated from the acquired measured value and information on the planned landing time. An estimated value is obtained, and a time when the estimated value falls within a predetermined range assuming that a value indicating the vibration of the visual sensor is equal to or less than a predetermined value is determined as the image capture time. The mobile robot according to claim 3 or 4. The image control processing unit
A buffering time for determining whether or not a buffering time indicating a predetermined time range in which an image blurring value is estimated to be larger than a predetermined value starting from the moment when the tip of the leg is landed is determined. A discriminator;
The mobile robot according to claim 2, further comprising: an image processing unit that processes an image captured at a time other than the buffering time among images periodically captured by the visual sensor.   The buffer time determination unit acquires the movement speed during the movement of the robot, and the acquired movement from a plurality of buffer time candidates determined to be shorter as the movement speed of the robot is higher. The mobile robot according to claim 6, wherein a buffering time corresponding to a speed is selected, and a result of determining whether or not the selected buffering time is reached is output to the image processing unit.

Images