JP7716089B2 - Agricultural machinery, agricultural machinery system, agricultural machinery connection method and program - Google Patents

Description

The present invention relates to technology for connecting agricultural machinery.

Working machines that are towed by or attached to tractors to perform various tasks are known. There are many types of working machines. Examples include machines that perform tasks such as fertilizing, tilling, spraying pesticides, and sowing seeds.

The appropriate implement is selected depending on the type of agricultural work being performed. This requires connecting the tractor and implement. This connection process is complicated, and there are concerns about accidents occurring during the process. Known technologies for addressing this issue include those described in Patent Document 1 and Non-Patent Document 1.

Patent document 1 describes a connecting member that makes it easier to connect a tractor and a work implement. Non-patent document 1 discloses technology that uses a camera and a target to automatically connect a tractor to a work implement.

KR102062994 publication

Ben Burgess, John Deere AutoConnect [online], November 28, 2014, (Retrieved October 13, 2021), Internet, <URL: http://tokkyo.shinsakijun.com/information/newtech.html>

One possible method would be to install measuring equipment on the tractor, measure the relative positions and postures of the tractor and implement, and then operate the tractor automatically based on these measurement results, connecting the tractor and implement. However, from a practical standpoint, using expensive measuring equipment would not be advisable.

Against this background, the present invention aims to provide technology that enables low-cost alignment when connecting a moving body, such as a tractor, to an object.

The present invention relates to an agricultural machine connected to a self-propelled mobile body, which includes a connection mechanism for connecting to the mobile body, and a first type of location target and a second type of location target positioned facing the mobile body when connected to the mobile body, with the first type of location target being for location from a relatively long distance and the second type of location target being for location from a relatively short distance, with the first type of location targets being arranged in pairs, one on the left and one on the right, and the second type of location target being positioned between the two first type location targets.

In one embodiment of the present invention, the first type of orientation target and the second type of orientation target have a plurality of orientation points, the markings constituting the orientation points of the first type of orientation target are larger than the markings constituting the orientation points of the second type of orientation target, and the number of orientation points of the second type of orientation target is greater than the number of orientation points of the first type of orientation target. In one embodiment of the present invention, the identification surface of one or both of the first type of orientation target and the second type of orientation target faces in the direction of the depression angle.

The present invention is an agricultural machinery system including the above-mentioned agricultural machinery and a self-propelled mobile body that is equipped with a camera and can be connected to the agricultural machinery, wherein the camera photographs the second type of location target from the front, captures the second type of location target in the center of the camera's shooting screen, and is set in a field of view that is outside the shooting range while the second type of location target can be identified.

The present invention also provides a method for connecting a self-propelled mobile body to an agricultural machine, wherein the mobile body is equipped with a camera for photographing the mobile body, and the agricultural machine is equipped with a coupling mechanism for connecting to the mobile body, a first type of location target, and a second type of location target, the first type of location target is for location from a relatively long distance, and the second type of location target is for location from a relatively short distance, the first type of location target is arranged in two, one on the left and one on the right, and the second type of location target is arranged between the two first type location targets, and the camera captures the first type of location target at a distance where the camera cannot recognize the image of the second type of location target. The position and orientation of the moving body relative to the agricultural machine are detected by image recognition of the orientation targets, a route for the moving body to approach the agricultural machine is set based on the position and orientation of the moving body relative to the agricultural machine, the moving body is moved to the agricultural machine according to the set route, and as the moving body approaches the agricultural machine through this movement, the first type of orientation target moves out of the camera's shooting range and the camera becomes able to image recognize the second type of orientation target, the distance between the moving body and the agricultural machine is then measured using the second type of orientation target, and the distance between the moving body and the agricultural machine is adjusted, thereby connecting the moving body to the agricultural machine.

In the above-mentioned agricultural machinery connection method invention, the movement of the moving body is controlled along the set path, and during the movement control process, the camera is positioned using the first type of positioning target and/or the second type of positioning target, the actual movement path of the moving body is calculated based on the positioning, the set path and the actual movement path are compared, and the movement path is re-set based on the results of the comparison.

Furthermore, in the above-mentioned agricultural machine connection method invention, there is a first stage in which the distance between the moving body and the agricultural machine is relatively long, and a second stage in which the distance between the moving body and the agricultural machine is relatively short, and in the first stage, the camera is aligned using the first type of alignment target, and in the second stage, the camera is aligned using the second type of alignment target, and the alignment in the second stage has a larger number of alignment points and is more accurate than the alignment in the first stage.

The present invention also relates to a program for causing a computer to execute control for connecting a self-propelled mobile body to agricultural machinery, wherein the mobile body is equipped with a camera for photographing the agricultural machinery, and the agricultural machinery is equipped with a connection mechanism for connecting to the mobile body, a first type of location target, and a second type of location target, wherein the first type of location target is for location from a relatively long distance, and the second type of location target is for location from a relatively short distance, and the first type of location target is arranged in two, one on the left and one on the right, and the second type of location target is arranged between the two first type location targets, and the computer is able to recognize the image of the second type of location target using the camera at a distance where the camera cannot recognize the image of the second type of location target. The program detects the position and orientation of the moving body relative to the agricultural machine by image recognition of the first type of location target, sets a route for the moving body to approach the agricultural machine based on the position and orientation of the moving body relative to the agricultural machine, and controls the movement of the moving body toward the agricultural machine according to the set route. As a result of the movement control, the first type of location target moves out of the camera's shooting range when the moving body approaches the agricultural machine, enabling the camera to recognize images of the second type of location target, and then measures the distance between the moving body and the agricultural machine using the second type of location target, and connects the moving body to the agricultural machine by adjusting the distance between the moving body and the agricultural machine.

The present invention provides technology that enables low-cost alignment of a moving object with an object.

This is an overall schematic diagram. FIG. 2 is a perspective view of the tractor-side hitch frame. This is a diagram illustrating the principle of a three-point link hitch. FIG. This is a diagram illustrating the principle of orientation. FIG. 1 is a diagram showing an outline of a target. FIG. FIG. 1 is a diagram illustrating the principle of setting a movement path. 10 is a flowchart illustrating an example of a processing procedure. 10A and 10B show other target variations.

1. Overview FIG. 1 shows an overview of an embodiment. FIG. 1 shows a tractor 100 and a work implement 200. In this example, the work implement 200 is, for example, a work implement that applies fertilizer. Types of work implements include work implements that apply fertilizer, tillage, spray pesticides, sow seeds, etc. There are also multiple types of tillage, and a work implement may be prepared for each purpose. Here, there is one work implement, but there may also be multiple work implements, one of which may be connected to the tractor.

(Connection structure)
The tractor 100 and the implement 200 are connected by a hitch frame, which will be described below. Note that the implement 200 may not have wheels and may be attached to the rear of the tractor 100 in a floating manner. Implements of this type are used for fertilizing.

(Tractor side hitch frame)
The tractor 100 is provided at its rear with a tractor-side hitch frame 110. The work implement 200 is provided with a work implement-side hitch frame 210 for connection to the tractor-side hitch frame 110.

Figures 2A and 2B are perspective views of the tractor-side hitch frame 110. Figure 2A is a perspective view looking at the side attached to the tractor 100, and Figure 2B is a perspective view looking at the side facing the implement-side hitch frame 201.

In the following, left and right are defined based on the view of the tractor 100 as seen from behind. The direction in which the tractor 100 moves forward is defined as the front.

The tractor-side hitch frame 110 has an inverted V-shaped frame structure 116, which forms the hypotenuse of a triangle. The lower part of the frame structure 116 has a right lower support section 111 and a left lower support section 112 that extend forward. The right lower support section 111 has a right lower support pin 111a at its tip, and the left lower support section 112 has a right lower support pin 112a at its tip.

The right lower support pin 111a is connected to the right lower link 141 (see Figure 3) located at the rear of the tractor 100. As shown in Figure 3, the right lower link 141 extends rearward from the rear of the tractor 100, and a support hole 141a is provided at the tip of the right lower link 141, which serves as the right lower hitch point. The right lower support pin 111a shown in Figure 2(A) is inserted into and engages with this support hole 141a, connecting the right lower support part 111 to the right lower link 141 on the tractor 100 side.

Similarly, a left lower link 142 extending rearward is provided at the rear of the tractor 100, and a support hole 142a is provided at the tip of the left lower link 142, which serves as the left lower hitch point. The left lower support pin 112a shown in Figure 2(A) is inserted into and engages with this support hole 142a, connecting the left lower support part 112 to the left lower link 142 on the tractor 100 side.

As described above, the right lower support portion 111 in Figure 2 is connected to the right lower link 141 on the tractor 100 side in Figure 3, and the left lower support portion 112 in Figure 2 is connected to the left lower link 142 on the tractor 100 side in Figure 3, thereby attaching the tractor-side hitch frame 110 to the rear of the tractor 100.

Also, as shown in Figure 2(A), an upper support portion 190 is disposed on the tractor-side hitch frame 110. The upper support portion 190 has a long hole and several round holes. The tip of the upper support arm 143 on the tractor 100 side shown in Figure 3 engages with one of these holes in a freely rotatable manner via a horizontal rod (not shown). The upper support portion 190 has multiple holes to accommodate tractors of different sizes and models. The holes to be used in the upper support portion 190 are selected depending on the model and size of the tractor.

In this way, the tractor-side hitch frame 110 is attached to the rear of the tractor 100 by the right lower link 141, left lower link 142, and upper support arm 143 on the tractor 100 side. The upper support arm 143 is power-operated and can be extended and retracted, and by adjusting the degree of extension and retraction, the fore-and-aft tilt of the tractor-side hitch frame 110 can be adjusted.

The tip of the right lower link 141 of the tractor 100 can be lifted by the right lift rod 144, and the tip of the left lower link 142 can be lifted by the left lift rod 145. Here, when the right lift rod drive unit 146 rotates, the right lift rod 144 moves back and forth in its axial direction, causing the tip of the right lower link 141 to move up and down. In addition, when the left lift rod drive unit 147 rotates, the left lift rod 145 moves back and forth in its axial direction, causing the tip of the left lower link 142 to move up and down. This up and down movement causes the tractor-side hitch frame 110 to move up and down relative to the tractor 100.

When the tractor-side hitch frame 110 moves up and down relative to the tractor 100, the tractor-side hitch frame 110 tilts forward and backward. In this case, the upper support arm 143 is extended and retracted to adjust the tractor-side hitch frame 110 so that it is vertical. The position and posture of the tractor-side hitch frame 110 on the tractor 100 are predetermined and known.

A PTO (Power Take Off) shaft 148 protrudes rearward from the rear of the tractor 100. The PTO shaft 148 is connected to the spline shaft 117 of the tractor-side hitch frame 110 via an expandable joint (not shown).

The spline shaft 117 has teeth extending in the axial direction formed on its outer periphery, allowing it to be driven back and forth in the axial direction and move. This mechanism will be explained below. First, a linear bearing retaining plate 170 is fixed to the tractor-side hitch frame 110. A linear bearing 171 is fixed to the linear bearing retaining plate 170.

The linear bearing 171 is a sliding bearing that holds the sliding shaft 172 in a state where it can slide axially. The sliding shaft 172 has a pin 172a that protrudes perpendicular to the axis, and the fork assembly (link) 113 engages with this pin 172a. The fork assembly (link) 113 is driven to rotate by the electric cylinder 114 via the drive arm 115. The rotation of this fork assembly (link) 113 moves the sliding shaft 172 back and forth.

A bearing holder plate 173, on which the spline shaft 117 is rotatably supported, is fixed to the rear end of the sliding shaft 172 (the end on the side of the work machine 200). As the sliding shaft 172 moves back and forth, the spline shaft 111 moves back and forth (axially) together with the bearing holder plate 173.

After connecting the tractor-side hitch frame 110 and the implement-side hitch frame 210, the electric cylinder 114 is operated to slide the sliding shaft 172 rearward (toward the implement). This causes the bearing retaining plate 173 holding the spline shaft 117 to move rearward (toward the implement 200), and the spline shaft 117 connects with the spline sleeve shaft 212 (see Figure 4) of the spline sleeve structure on the implement 200 side. In this way, the PTO shaft 148 of the tractor 100 is connected to the PIC shaft (not shown) of the implement 200. The PIC shaft is called the power input connect shaft and is the shaft on the implement side that receives driving force from the tractor 100.

The tractor-side hitch frame 110 also has a hook 119 driven by an electric cylinder 118. By rotating the hook 119 using the electric cylinder 118, the hook 119 catches in an opening 218 provided in the work implement-side hitch frame 210 shown in Figure 4. This locks the tractor-side hitch frame 110 and work implement-side hitch frame 210 in a connected state.

A connector 160 is located on the top of the tractor-side hitch frame 110. The connector 160 connects to the receiving connector on the work implement-side hitch frame 210. The connector 160 connects the hydraulic system and various electrical wiring between the tractor 100 and work implement 200.

Reference numerals 161 and 162 denote hydraulic transmission pipes connecting the connector 160 and the tractor 100. Reference numeral 163 denotes an electrical wiring cable connecting the connector 160 and the tractor 100.

(Hitch frame on work equipment side)
Next, we will explain the work implement hitch frame 210 that is connected to the tractor hitch frame 110. Figure 4(A) is a perspective view of the work implement hitch frame 210, looking at the side facing the tractor 100. Figure 4(B) is a perspective view of the work implement 200 side.

The work implement hitch frame 210 has a triangular frame structure 211. This triangular frame structure 211 has a recessed U-shape, and the frame structure 116 of the tractor hitch frame 110 fits into the inside of this structure from below.

By aligning the horizontal positions of frame structure 116 and frame structure 211, positioning frame structure 211 upward, and frame structure 116 downward, and then moving the tractor-side hitch frame 110 upward, the frame structure 116 of the tractor-side hitch frame 110 fits from below into the inside of the frame structure 211 of the implement-side hitch frame 210, connecting the tractor-side hitch frame 110 and the implement-side hitch frame 210.

In this state, the tractor hitch frame 110 and the implement hitch frame 210 cannot separate front to back, and the tractor hitch frame 110 supports the implement hitch frame 210 from below. Also, in this state, the connector 160 of the tractor hitch frame 110 connects to the receiving connector housed inside the part marked 219 of the implement hitch frame 210.

A spline sleeve shaft 212, supported by a bearing structure, is located below the work implement hitch frame 210. The spline shaft 117 of the tractor hitch frame 110 is connected to the spline sleeve shaft 212. The work implement 200 side of the spline sleeve shaft 212 is connected to a PIC shaft (not shown) on the work implement 200 side.

The inner edge of the spline sleeve shaft 212 where the spline shaft 117 fits and the end of the teeth are tapered to make it easier for the spline shaft 117 to fit. For the same purpose, the end of the spline shaft 117 is also tapered. The spline shaft 117 can be fitted into the spline sleeve shaft 212 even when the spline shaft is rotated (of course, it can also be fitted without rotating).

As shown in Figure 4(B), an upper fixing member 213 and lower fixing members 215, 215 are arranged on the side of the work machine side hitch frame 210 facing the work machine 200 (see Figure 1) to secure the work machine side hitch frame 210 to the work machine 200. The position and posture of the work machine side hitch frame 210 on the work machine 200 are predetermined and known.

Reference numerals 231 and 232 denote connection ports for transmitting hydraulic pressure from the receiving connector located inside the portion denoted by reference numeral 219. The hydraulic cable (not shown) of the work machine 200 is connected to these connection ports. Reference numeral 233 denotes a connector for electrical wiring connected to the receiving connector. Wiring from the work machine 200 is connected to this connector 233.

(Orientation target)
As shown in Fig. 4, a location target unit 220 is attached to the upper part of the work machine side hitch frame 210. The location target unit 220 is made up of two global targets 221a and 221b spaced apart on the left and right, and a detailed target 222 (see Fig. 6 for details) located between the global targets 221a and 221b.

The orientation of the global targets 221a, 221b and the detailed target 222 is adjusted so that their faces face toward the tractor 100 (the direction of the camera 140) when the tractor 100 and the work implement 200 are connected. In this example, when the tractor 100 and the work implement 200 are connected, the global targets 221a, 221b and the detailed target 222 are set to face the tractor 100 directly.

Global targets 221a and 221b are orientation targets intended for image recognition at relatively long distances (approximately 2m to 5m or more in this example). Detail target 222 is an orientation target intended for image recognition at relatively short distances (approximately 2m to less than 5m in this example).

Global targets 221a and 221b have a display that combines multiple rectangular designs. The corners of each rectangle are image-recognized as feature points. These feature points become orientation points. The position of each rectangle in global targets 221a and 221b, as well as the relative positions of the rectangles, are known, and the positions of the feature points and their relative positions are also known.

The positional relationship between the global targets 221a and 221b is also known. Furthermore, the position and orientation (posture) of the global targets 221a and 221b on the work machine side hitch frame 210 (work machine 200) are also known.

Therefore, by performing orientation using the global targets 221a and 221b, it is possible to determine the exterior orientation parameters (position and orientation) of the camera 140 that photographed the global targets 221a and 221b relative to the work machine side hitch frame 210 (work machine 200). The principles of orientation will be described later.

The designs of the global targets 221a and 221b are set to be relatively large so that they can be recognized from a distance. However, as the designs become larger, the density of the orientation points decreases and the number of orientation points is limited.

The detailed target 222 has a more detailed pattern and a larger number of feature points than the global targets 221a and 221b. This allows for more orientation points to be obtained, enabling orientation with greater accuracy. The position and orientation of the detailed target 222 relative to the work machine side hitch frame 210 and the work machine 200 are also known.

In this example, the detailed target 222 is composed of a large number of rectangular AR markers arranged in a matrix, both vertically and horizontally. Each AR marker is composed of a combination of a large number of rectangular patterns, and the corners of each rectangular pattern are extracted from the image as feature points, which become orientation points. The positions of the corners of the patterns that become the orientation points for each AR marker are known, and the relative positions of the AR markers arranged vertically and horizontally are also known. Therefore, a large number of orientation points can be obtained from the captured image of the detailed target 222.

Note that the AR marker is just one example; coded targets, targets that use appropriate graphics as codes, and other known targets can also be used. The important thing is that they can be used as orientation targets.

The target display surfaces of the global targets 221a, 221b and the detailed target 222 are flat, and a pattern for image recognition of the orientation point is displayed on this flat surface.

The location target unit 220 can be tilted up and down. Figure 4 shows the location target unit 220 tilted slightly upward. This vertical tilt is adjusted taking into account the vertical positional relationship of the camera 140 on the tractor 100.

For example, the effect of sunlight reflection can be reduced by tilting the orientation target unit 220 slightly downward. Reflection of sunlight from the orientation target unit 220 can make it difficult to recognize the orientation pattern. As a countermeasure in this case, the orientation target unit 220 is tilted slightly downward (toward the depression angle). Because sunlight does not strike from the depression angle, tilting the orientation target unit 220 slightly downward can avoid the problem of sunlight being reflected from the orientation target unit 220 and appearing in the image captured by the camera 140.

The tilt angle should be approximately 10° to 45°. If the tilt angle is known, location can be performed without any problems. It is also possible to arrange one of the global target and the detailed target facing downward. Furthermore, the global targets 221a, 221b and the detailed target 222 shown in Figure 4 may be arranged vertically, and the global target and/or the detailed target may be arranged tilted downward.

(Principle of orientation)
Orientation will be explained below. Orientation is the process of determining the exterior orientation parameters (position and orientation) of a camera using orientation points. Here, we will explain single photo orientation, which performs orientation from a single photo. Figure 5 is a diagram illustrating the principle of the intersection method, which is a basic method for determining the position and orientation of a camera in single photo orientation.

The basic principle is that if there are three or more points (orientation points) with known 3D coordinates on a single photograph, the position (3D position) and orientation (the direction in which it is facing in the 3D coordinate system) of the camera that took the photograph can be determined.

This principle will be explained below using Figure 5. Point O is the viewpoint of the photograph (the projection origin of the camera used). P1, P2, and P3 are control points with known three-dimensional positions (three-dimensional coordinates). p1, p2, and p3 are the positions (photo coordinates) of points P1, P2, and P3 on the photographic screen obtained by photographing.

In this case, a line is set geometrically connecting points p1 and P1, a line connecting points p2 and P2, and a line connecting points p3 and P3. The position of the intersection of these three lines becomes the coordinates of point O. Since the positions of points P1, P2, and P3 are known, the coordinates of point O in the coordinate system describing points P1, P2, and P3 can be determined.

Furthermore, the line connecting point O and the center of the shooting screen becomes the camera's optical axis. In this way, the position and orientation of the camera in the coordinate system describing points P1, P2, and P3 are determined.

The above is the basic principle. Below, we will explain the actual calculation method. In this embodiment, the DLT method is used to determine the camera position and orientation. The DLT method approximates the relationship between the photograph coordinates and the three-dimensional coordinates of the subject using a third-order projective transformation formula. The calculation formula is shown in Equation 1 below.

In the DLT method, if there are nine or more points whose positions are known in a single photograph, it is possible to determine the position and orientation of the camera in a coordinate system that describes those positions. In the above case, L ₁ to L ₁₁ are unknown parameters related to the position and orientation of the camera.

In this way, the position and orientation of the camera 140 relative to the orientation target 220 are determined. Since the position and orientation of the camera 140 on the tractor 100 and the position and orientation of the orientation target 220 on the work implement 200 are known, the positional and orientational relationship between the tractor 100 and work implement 200 can be determined as a result of the above orientation. For example, the distance between the tractor 100 and work implement 200 and the orientational relationship between the tractor 100 and work implement 200 can be determined. In addition, the positional and orientational relationship between the tractor-side hitch frame 110 and work implement-side hitch frame 210 can be determined.

Here, the greater the number of orientation points, and the greater the range of orientation points from the camera's perspective (up to a 90° viewing angle range) rather than a narrow range, the higher the accuracy of calculating the position and orientation of the camera 140.

In this example, large-area targets 221a and 221b are placed far apart on the left and right to allow image recognition from a distance, and multiple control points are distributed far apart on the left and right, ensuring accuracy of control over long distances. On the other hand, the detailed target 222, which is used at close range, has more control points to ensure accuracy of control over long distances.

(How the target appears)
Fig. 6 shows the transition of how the orientation target 220 appears in the image captured by the camera 140. Fig. 6(A) shows the case where the orientation target 220 is captured from a distance where it fits within the captured image, and Fig. 6(B) shows the case where the orientation target 220 is captured even closer than the distance in Fig. 6(A), and the orientation target 220 extends beyond the captured image.

When the distance is long, the display of the detailed target 222 is difficult to recognize, and orientation using the detailed target 222 is difficult or results in large errors. The reason for the large errors is that the display is blurred, which reduces the positional accuracy of feature points on the screen and increases the number of incorrectly extracted or unextracted feature points. This is particularly noticeable when an inexpensive camera 140 is used.

In this case, orientation is performed using the large-display global targets 221a and 221b. In this case, because the global targets 221a and 221b are positioned apart on the left and right, orientation accuracy in the left-right direction can be increased even with a small number of orientation points. As a result, information on the position and orientation of the tractor 100 relative to the implement 200 can be obtained with high accuracy.

In the case of Figure 6 (B), as the camera 140 approaches the orientation target section 220, the global targets 221a and 221b move out of the camera 140's shooting range (angle of view), and the detailed target 222 mainly occupies the screen.

In other words, the relationship between the size of the designs of the global targets 221a and 221b and the detailed target 222, and the optical characteristics (magnification and focal length) of the camera 140 are set so that when the camera approaches, the global targets 221a and 221b are out of focus and the detailed target 222 is mainly captured.

At the stage shown in Figure 6 (B), the relationship between the positions and attitudes of the tractor 100 and the implement 200 is determined using a detailed target 222 with many orientation points. This makes it possible to determine with high accuracy the position and attitude (mainly position (distance) at this stage) of the tractor 100 relative to the implement 200 at the final stage when the tractor-side hitch frame 110 comes into contact with the implement-side hitch frame 210.

In particular, the detailed target 220 can obtain a large number of control points distributed vertically and horizontally, allowing for precise measurement of the precise distance required in the final stage (the distance between the tractor-side hitch frame 110 and the implement-side hitch frame 210).

(Control system configuration)
7 is a control block diagram of the tractor 100. The tractor 100 is equipped with a control computer 120, a GPS device 130, a camera 140, and a drive system 150. The control computer 120 is equipped with a CPU, a storage device, and various interface devices.

The control computer 120 comprises an image data acquisition unit 121, a target recognition unit 122, a control point extraction unit 123, a guidance target detection unit 124, a location processing unit 125, a movement path setting unit 126, a vehicle control unit 127, a guidance determination unit 128, and a location-based movement path calculation unit 129. These functional units may be configured as software or as dedicated hardware. It is also possible to configure some or all of the above functional units using FPGAs, etc.

The image data acquisition unit 121 acquires image data of images captured by the camera 140. The target recognition unit 122 detects global targets 221a, 221b and detailed targets 222 from the images captured by the camera 140. The orientation point extraction unit 123 extracts orientation points from the image-recognized global targets 221a, 221b and detailed targets 222.

The guidance target detection unit 124 detects the area (guidance target) that serves as the target for guiding the tractor 100. In this example, the central position between global targets 221a and 221b, which are spaced apart on the left and right, is set as the guidance target. Note that this central position (guidance target) between global targets 221a and 221b is set to coincide with the center position of the detailed target 222.

The orientation processing unit 125 orients the camera 140 using the orientation target unit 220 (global targets 221a, 221b and detailed target 222). This orientation determines the position and orientation of the camera 140 relative to the orientation target. The position and orientation of the camera 140 on the tractor 100 are known, and the position and orientation of the orientation target 220 relative to the work implement 200 are also known. Therefore, by determining the position and orientation of the camera 140 relative to the orientation target 220, the position and orientation of the tractor 100 relative to the work implement 200 can be determined. In addition, the relationship between the positions and orientations of the tractor-side hitch frame 110 and the work implement-side hitch frame 210 can be determined.

The movement path setting unit 126 sets the movement path of the tractor 100 relative to the work implement 200 required to couple the tractor 100 to the work implement 200 based on the position and attitude of the tractor 100 relative to the work implement 200 determined by the above orientation. In other words, it sets the planned movement path of the tractor 100 relative to the work implement 200 required to couple the tractor-side hitch frame 110 to the work implement-side hitch frame 210.

Figure 8 shows an example of the planned movement path. In the case of Figure 8, a path is set that involves two turns so that the back of the tractor 100 faces the work implement 200 directly (the difference in direction is 0°) at position P1, a distance L1 before the final target position.

P1 is set as the position where the global targets 221a and 221b are out of the field of view of the camera 140 (but do not have to be completely out of view), the detailed target 222 is mainly visible on the image capture screen, and the center of the image capture screen and the center of the detailed target 222 (guidance target) coincide (or may coincide approximately). L1 is set to approximately 50 cm to 1 m. The target position is the position where the tractor side hitch frame 110 and the implement side hitch frame 210 are connected (the position where the separation distance between the tractor side hitch frame 110 and the implement side hitch frame 210 is zero).

The planned movement path in Figure 8 is set as follows. First, the position where camera 140 can be positioned using global targets 221a and 221b is set as the starting point PA. Then, the front position P1 is determined based on the positional relationship between starting point PA and the target position obtained as a result of camera 140 positioning, the relationship between the orientations of tractor 100 and work implement 200, and the front direction of work implement 200. The front direction of work implement 200 is determined from the direction of the surface of the positioning target section 220 relative to work implement 200.

The front position P1 is set as a position where the camera 140 can image-recognize and locate the detailed target 222, and where the tractor-side hitch frame 110 faces the implement-side hitch frame 220 (they are oriented in the same horizontal direction).

Next, the midpoint C1 between PA and P1 is found, and the tractor's route to P1 is set by making two left and right turns with the same curvature radius. In this way, the planned travel route shown in Figure 8 is set.

The planned movement path can include one or more turns. For example, a path can include repeated small turns to the left and right, with the tractor-side hitch frame 110 facing the implement-side hitch frame 210 at position P1.

The vehicle control unit 127 performs autonomous control (automatic driving control) related to the movement of the tractor 100. Specifically, it performs the drive tire rotation control and steering control required to move the tractor 100 along the movement path from PA to P1 shown in Figure 8, and the drive tire rotation control and steering control required to move the tractor 100 from P1 to the target position.

The vehicle control unit 127 also performs the drive control required to connect the tractor-side hitch frame 110 to the work implement-side hitch frame 210, drive control of the PTO shaft, and drive control for connecting the PTO shaft to the PTO shaft of the work implement 200.

The guidance determination unit 128 determines whether the tractor 100 is moving along the movement path shown in Figure 8, and whether the tractor is being guided appropriately as it approaches the work implement 200 from P1.

The orientation-based movement path calculation unit 129 calculates the actual movement path of the tractor 100 based on the results of orientation of the camera 140 using the global targets 221a, 221b and/or the detailed targets 222.

For example, suppose the above orientation is performed every second while the tractor 100 is moving. In this case, the position of the camera 140 relative to the work implement 200 is obtained every second. By tracking the plot of this position, the actual movement path of the tractor 100 relative to the work implement 200 can be obtained.

For example, by drawing a line connecting the above plotted points on the diagram in Figure 8, the set movement path can be compared with the actual movement path of the tractor 100. For example, by performing this comparison, it can be determined whether guidance is being provided appropriately.

The GPS device 130 performs positioning using navigation signals from navigation satellites. Although not shown, the tractor 100 is also equipped with a direction sensor and a gyro sensor in addition to the GPS device 130, which measure not only the position of the tractor 100 but also its attitude. The relationship between the position and attitude of the GPS device 130, direction sensor, and gyro sensor on the tractor 100 is measured in advance and is known data.

The camera 140 is positioned behind the tractor 100 and is set to capture images of the area behind the tractor 100. In this example, the camera 140 is positioned on the central axis of the tractor 100, facing directly rearward. The position and orientation of the camera 140 on the tractor 100 are measured in advance and are known data. In addition, the positions of the camera 140 and the position of the orientation target 220 are adjusted so that the center of the orientation target 220 (center of the detailed target 222: guidance target) is on the optical axis of the camera 140 when coupled.

Camera 140 is a digital still camera that continuously captures still and moving images. The interval between captures is 0.25 to 1 second. It is also possible to capture moving images and use the frame images that make up the moving images as still images.

The camera 140 does not have to be a digital still camera; it can also be an inexpensive industrial camera or webcam. Inexpensive cameras have poor focus adjustment (some do not have any focus adjustment) so that they can capture images over a wide distance range, and images from a distance have low resolution. However, because long-distance orientation is performed using a global target and close-distance orientation is performed using a detailed target, even inexpensive cameras can be used.

The camera 140 is calibrated in advance to determine its internal orientation parameters (correction parameters that correct distortions in the optical system, etc.).

The drive system 150 is the power system of the tractor 100 and drives various parts of the tractor 100. Drive sources include internal combustion engines such as diesel engines or gasoline engines, electric motors, and hydraulics.

The drive system 150 provides the drive for moving (driving) the vehicle, the drive for steering the vehicle, the drive for the hitch frame, the drive for the PTO shaft, and the drive for connecting the PTO shaft.

(Example of processing)
Fig. 9 is a flowchart showing an example of processing performed in the control computer 120. A program for executing the processing in Fig. 9 is stored in a semiconductor storage device or a hard disk device provided in the control computer 120, and is read and executed by the CPU of the control computer 120. The program may also be stored in an appropriate storage medium.

When processing begins, the camera 140 begins capturing images (step S101). Next, the tractor 100 is guided to the vicinity of the target work implement using GPS or manual driving by the driver (step S102). This guidance is performed so that the tractor 100 is positioned approximately 5 to 10 meters from the target work implement, with its back facing the work implement 200 as far as possible. The position of the tractor 100 is defined as the position of the center of gravity of the tractor 100 or the center of the vehicle. The same applies to the position of the work implement 200.

For example, the location information of the work machine 200 from the last time it was used is recorded, and the above-mentioned GPS-based guidance is performed based on that information.

During this guidance process, it is determined whether the camera 140 has been able to recognize the images of the global targets 221a and 221b (step S103). In this example, the camera 140 and the designs of the global targets 221a and 221b are set so that the global targets 221a and 221b can be recognized when the camera 140 approaches within a distance of 10 m, or the designs of the global targets 221a and 221b are set to match the performance of the camera 140. This image recognition is performed by the target recognition unit 122.

Once the global targets 221a and 221b have been image-recognized (captured), the tractor 100 is stopped at that position, and orientation using the global targets 221a and 221b begins (step S104). This process is performed by the orientation processing unit 125. By performing this orientation, the relationship between the position and attitude (orientation) of the tractor 100 relative to the implement 200 at that time is determined. This orientation is then repeatedly performed.

The interval at which orientation is repeated is matched to the interval at which camera 140 takes photographs. For example, if camera 140 takes photographs at 0.5 second intervals, orientation will also be performed at 0.5 second intervals.

Once the relationship between the position and attitude (orientation) of the tractor 100 and work implement 200 at the above-mentioned stopped position has been determined, a movement path from the starting point PA to the preceding target P1 is set using the method shown in Figure 8 (step S105). Here, the movement path shown in Figure 8 is set using the above-mentioned stopped position as the starting point PA. This process is performed by the movement path setting unit 126.

Once the travel route has been set, the tractor is moved along that travel route (step S106). In this example, the vehicle control unit 127 performs automatic driving along the set travel route. That is, steering control and drive wheel rotation control are performed so that the tractor 100 moves autonomously along the set travel route.

Instead of automatic driving, it is also possible to provide the driver with information to assist them in moving along the designated route, and then manually drive the tractor along the set route. In this case, the route is displayed on a display, for example, and the driver drives the tractor 100 while looking at it.

Even after guidance has begun, the location determination in step S104 continues to be repeated. At this point, a determination is made as to whether the guidance in step S106 is appropriate (step S107). This process is performed by the guidance determination unit 128.

The determination in step S107 will be described below. During the guidance process, the orientation of the camera 140 is continuously and repeatedly performed, and the position and attitude of the camera 140 are calculated every moment. This information is used to calculate the actual movement path of the tractor 100. This calculation of the actual movement path is performed by the movement path calculation unit 129 based on the orientation.

By comparing the calculated actual movement path with the planned movement path set in step S105, it is determined whether the set path is being accurately traced. Note that GPS information and/or attitude information based on a direction sensor or gyro sensor may also be used in calculating the actual movement path of the tractor 100.

For example, the travel path of the tractor 100 obtained from the results of the above positioning is created as an actual travel path. Then, the actual travel path is compared with the travel path set in step S105 (planned travel path). If there is no deviation between the two, the guidance at that time is determined to be appropriate. If there is an unacceptable deviation between the two, the guidance at that time is determined to be inappropriate.

If it is determined that the guidance is inappropriate, the processing from step S105 onwards is repeated. In this case, a new travel route is set in step S105. The new travel route is set using the position of the tractor 100 at the time the above determination was made as the starting point PA and the position of point P1 in Figure 8 as the end point.

The tractor's driving wheels (drive wheels) are large and not suitable for precise control, and various factors, such as less-than-ideal road conditions, can prevent the tractor 100 from accurately tracing the set travel path. In such cases, a new travel path is set using the method described above. This gradually updates the travel path to the target position (in this case, P1 in Figure 8), improving the accuracy of reaching the target position.

If guidance is being performed appropriately, it is determined whether the detailed target 222 has been recognized (captured) in the image captured by the camera 140 (step S108). In this example, the camera 140 and the design of the detailed target 222 are set so that the detailed target 222 can be recognized as an image when the target approaches within a distance of 3 m. This image recognition is performed by the target recognition unit 122.

If the detailed target 222 cannot be captured, the processing from step S106 onwards is repeated. If the detailed target 222 can be captured, orientation of the camera 140 using the detailed target 222 is started (step S109). This processing is performed by the orientation processing unit 125. The orientation performed in step S109 uses more orientation points than the orientation performed in step S104, and is therefore performed with higher accuracy. This orientation is then continuously and repeatedly performed.

In step S109, the measurement of the position and orientation of the tractor 100 relative to the implement 200 is switched from one based on orientation using the global targets 221a and 221b to one based on orientation using the detailed target 222. It is also possible to continue orientation using the global targets 221a and 221b, and then perform further orientation using the detailed target 222.

Next, it is determined whether guidance is appropriate again (step S110). The determination method is the same as in step S107. However, because the position and orientation information used is more accurate than in step S107, the determination conditions are stricter. For example, suppose the allowable error in step S107 is 3 cm. In this case, the allowable error in step S110 is, for example, 5 mm.

If it is determined in step S110 that the guidance is inappropriate, the process returns to the previous step of step S105, and the travel route is set again. The new travel route is set using the position of the tractor 100 at the time the determination in step S110 is made as the starting point PA and the position of point P1 in Figure 8 as the end point.

The determination in step S110 uses the detailed target 222, which has more orientation points than the global targets 221a and 221b. As a result, the orientation accuracy is higher than in step S104. As a result, the orientation accuracy, i.e., the measurement accuracy of the position and orientation of the tractor 100 relative to the work implement 200, is higher when the tractor 100 is closer to the work implement 200. This also increases the accuracy of reaching point P1 in Figure 8.

Furthermore, when the travel path is set again, the accuracy of the P1 setting is also improved. This also improves the accuracy of the final connection work between the tractor side hitch frame 110 and the implement side hitch frame 210.

If guidance is determined to be appropriate in step S110, it is determined whether the tractor is at the front target position P1, i.e., whether the tractor 100 is at P1 (step S111). The front target position P1 is the position where the tractor-side hitch frame 110 faces the implement-side hitch frame 210, and the tractor 100 can then be driven backward in a straight line.

If the tractor 100 is not at the front target position P1, the process of step S110 is repeated. If the tractor 100 is at the front target position P1, the tractor 100 is stopped there, the height position of the tractor-side hitch frame 110 is lowered to a position lower than the height position of the implement-side hitch frame 210, and then the tractor 100 moves linearly a predetermined distance L1 (step S112).

The preset specified distance L1 is the distance from the front position P1 to the position (target point) where the tractor-side hitch frame 110 and the implement-side hitch frame 210 are connected, as shown in Figure 8. L1 is usually set to approximately 50 cm to 1 m.

In step S112, positioning continues using the detailed target 222 to measure the distance between the tractor-side hitch frame 110 and the implement-side hitch frame 210, and movement of the distance L1 is precisely controlled. It is also possible to fine-tune the orientation of the tractor 100 here.

As a result of step S112, the tractor-side hitch frame 110 and the implement-side hitch frame 210 are horizontally aligned. The tractor-side hitch frame 110 is then raised, and the tractor-side hitch frame 110 and the implement-side hitch frame 210 are connected.

Next, the spline shaft 117 is moved rearward to connect to the spline sleeve 212. The spline shaft 117 is then locked using the hook 119. In this way, the tractor 100 and the implement 200 are connected.

A contact sensor that detects contact between the tractor-side hitch frame 110 and the implement-side hitch frame 210 may be placed on the tractor-side hitch frame 110, and the tractor 100 may be stopped from reversing when contact between the two is detected.

(superiority)
By using two types of orientation targets, it is possible to measure the relationship between the position and orientation of the tractor relative to the implement with high accuracy even when using an inexpensive camera for orientation, thereby ensuring reliable and stable automatic coupling of the tractor to the implement.

(others)
Figure 10 shows an example of a coded target. In this case, the central circle is the orientation point. The position of the curvature center of the curved display is set to be the orientation point. The target can be identified by its display pattern.

If both global and detailed targets can be recognized by image, it is also possible to perform orientation using both global and detailed targets.

Figures 6 and 10 show examples of location targets divided into two levels according to distance, but location targets may also be divided into three or more levels. For example, it is possible to arrange even finer, more detailed targets that can be image-recognized and located at the closest point, thereby increasing the accuracy of location at the closest point.

The present invention can be used not only for agricultural machinery but also for aligning a moving body with an object that the moving body is connected to or comes into contact with. For example, the present invention can be used for aligning a moving body that transports cargo with the cargo. The present invention can also be used in technology for determining the relative positions of a moving body and other moving bodies, and technology for aligning positions.

A stereo camera can also be used as the camera. In this case, the relationship between the position and attitude of the tractor and work equipment can be obtained through stereo photo measurement. In this case, orientation is still required, and the present invention is used. Therefore, the technology described in this specification can be used even when a stereo camera is used. It is also possible to build a stereo camera system with a longer baseline length using two cameras.

For example, the work machine 200 may be located indoors or in a covered area. If there is a roof, GPS cannot be used. In this case, an IMU (inertial measurement unit) is installed on the tractor 100, and the measurements from the IMU are used to autonomously move the tractor 100 to the starting point PA in Figure 8.

In this case, the position where the work machine 200 was detached during the previous operation is stored, and autonomous driving to the starting point PA is performed using inertial guidance with the IMU. Even if an error occurs during this operation, the exact position of the starting point PA relative to the work machine 200 can be determined by performing orientation using the global targets 221a and 221b.

100...Tractor, 110...Tractor side hitch frame, 111...Right lower support portion, 111a...Right lower support pin, 112...Left support portion, 112a...Left lower support pin, 113...Fork ASSY, 114...Electric cylinder, 115...Drive arm, 116...Frame structure, 117...Spline shaft, 118...Electric cylinder, 119...Hook, 141...Right lower link, 141a...Support hole, 142...Left lower link, 142a...Support hole, 143...Upper support arm, 144...Right lift rod, 145...Left lift rod, 146...Right lift rod drive portion, 147...Left lift rod drive portion, 148...PTO shaft, 160...Connector, 161, 162...Pipe for transmitting hydraulic pressure, 163...Cable for electrical wiring, 170...Linear bearing retaining plate, 171...Linear bearing, 172...Sliding shaft, 173...Bearing retaining plate, 190...Upper support part, 200...Work machine, 210...Hitch frame on work machine side, 212...Spline sleeve, 213...Upper fixing member, 214, 215...Lower fixing member, 218...Opening, 219...Part with receiving connector inside, 220...Location target part, 221a...Global target, 221b...Global target, 222...Detailed target, 231, 232...Connection port for connecting hydraulic transmission pipe, 233...Connector for electrical wiring.

Claims (8)

An agricultural machine connected by a self-propelled mobile body,
a coupling mechanism for coupling with the moving body;
a first type of location target and a second type of location target arranged at positions facing the moving body when the target is connected to the moving body;
the first type of location target is for location from a relatively long distance,
the second type of location target is for location from a relatively short distance,
The agricultural machine has two first type location targets arranged on the left and right, and the second type location target arranged between the two first type location targets. the first type of location target and the second type of location target each have a plurality of location points;
the indications constituting the orientation points of the first type orientation targets are larger than the indications constituting the orientation points of the second type orientation targets;
2. The agricultural machine according to claim 1, wherein the number of control points of the second type of location target is greater than the number of control points of the first type of location target. The agricultural machine described in claim 1 or 2, wherein one or both of the first type of location target and the second type of location target have an identification surface facing in the direction of the depression angle. The agricultural machine according to any one of claims 1 to 3,
An agricultural machinery system including a camera, a self-propelled mobile body that can be connected to the agricultural machinery ,
The camera is
The second type of location target is photographed from the front, and the second type of location target is captured at the center of the photographed screen of the camera, and in a state where the second type of location target can be identified,
An agricultural machinery system in which the first type of location target is set in a field of view outside the shooting range. An agricultural machine connecting method for connecting a self-propelled mobile body to an agricultural machine,
The moving body is equipped with a camera that photographs the agricultural machine,
the agricultural machine includes a coupling mechanism for coupling with the moving body, a first type of location target, and a second type of location target;
the first type of location target is for location from a relatively long distance,
the second type of location target is for location from a relatively short distance,
the first type of location target is arranged in two, one on the left and one on the right, and the second type of location target is arranged between the two first type of location targets;
At a distance where the camera cannot recognize an image of the second type of location target, the camera recognizes an image of the first type of location target, thereby detecting the position and orientation of the moving body relative to the agricultural machine;
A route for the moving body to approach the agricultural machine is set based on the position and orientation of the moving body relative to the agricultural machine;
moving the moving body to the agricultural machine along the set route;
When the moving body approaches the agricultural machine due to the movement, the first type of location target moves out of the imaging range of the camera, and the camera becomes able to recognize the image of the second type of location target, and then the distance between the moving body and the agricultural machine is measured using the second type of location target;
An agricultural machine connecting method for connecting the moving body to the agricultural machine by adjusting the distance between the moving body and the agricultural machine. movement control of the moving object along the set route is performed;
During the movement control process, the camera is oriented using the first type of orientation target and/or the second type of orientation target;
Calculating an actual moving path of the moving object based on the orientation;
comparing the set route with the actual travel route;
The agricultural machine connecting method according to claim 5, wherein the movement route is set again based on the result of the comparison. a first stage in which the distance between the moving body and the agricultural machine is relatively long;
a second stage in which the distance between the moving body and the agricultural machine is relatively short,
In the first stage, the camera is oriented using the first type of orientation target;
In the second stage, the camera is oriented using the second type of orientation target;
7. The agricultural machine coupling method according to claim 5, wherein the number of orientation points in the second stage is greater than that in the first stage, and the orientation accuracy is higher. A program for causing a computer to execute control for connecting a self-propelled mobile body to an agricultural machine,
The moving body is equipped with a camera that photographs the agricultural machine,
the agricultural machine includes a connection mechanism with the moving body, a first type of location target, and a second type of location target;
the first type of location target is for location from a relatively long distance,
the second type of location target is for location from a relatively short distance,
the first type of location target is arranged in two, one on the left and one on the right, and the second type of location target is arranged between the two first type of location targets;
A computer detects the position and orientation of the moving body relative to the agricultural machine by image-recognizing the first type of location target with the camera at a distance where the camera cannot image-recognize the second type of location target;
setting a route for the moving body to approach the agricultural machine based on the position and orientation of the moving body relative to the agricultural machine;
Controlling the movement of the moving body to the agricultural machine according to the set route;
By the movement control, when the moving body approaches the agricultural machine, the first type of location target moves out of the imaging range of the camera, and the camera becomes able to recognize the image of the second type of location target, and then the distance between the moving body and the agricultural machine is measured using the second type of location target;
and coupling the mobile body to the agricultural machine by adjusting the distance between the mobile body and the agricultural machine.