JP2025004533A - Crop Detection System - Google Patents

Abstract

To provide a crop detection system that facilitates the identification of the state of crops, including underground parts, planted in a field.SOLUTION: The present invention provides a crop detection system that detects a crop in a field. The crop detection system includes: a first sensor for detecting above-ground information, which is information on the above-ground part of the crop; a second sensor for detecting underground information, which is information on the underground part of the crop; and an information processing device for estimating the entire crop, including both the above-ground and underground parts, based on the above-ground information and the underground information.SELECTED DRAWING: Figure 4

Description

This disclosure relates to a crop detection system for farm fields.

Conventionally, crop detection systems for managing crops in fields are known. Technologies related to such systems are described in Patent Documents 1 and 2.

According to the technology described in Patent Document 1, a vegetation index for agricultural crops is calculated by analyzing aerial images of a field taken from above. According to the technology described in Patent Document 2, a map of the field viewed from above is generated based on captured images and position information acquired by a photographing device and a positioning device, respectively, installed on an unmanned aerial vehicle.

JP 2019-185773 A Patent No. 6929190

According to the technology described in Patent Documents 1 and 2, in a system for detecting crops in a field, the above-ground parts of the crops that grow above ground are focused on, and the growth of the crops is understood from the above-ground parts. However, because crops grow by obtaining nutrients from underground parts, it is necessary to understand the growth of the crops while also taking into account the underground parts.

The present invention has been developed in consideration of the above problems, and aims to provide a crop detection system that can easily grasp the condition of crops, including underground parts, planted in a field.

A crop detection system according to one aspect of the present disclosure is a crop detection system that detects crops in a field, and includes a first sensor that detects aboveground information, which is information about the aboveground parts of the crops, a second sensor that detects underground information, which is information about the underground parts of the crops, and an information processing device that estimates the crop, which is the combination of the aboveground parts of the crops and the underground parts of the crops, based on the aboveground information and the underground information.

The general or specific aspects of the present disclosure may be realized by an apparatus, a system, a method, an integrated circuit, a computer program, or a computer-readable non-transitory storage medium, or any combination thereof. The computer-readable storage medium may include a volatile storage medium or a non-volatile storage medium. The apparatus may be composed of multiple devices. When the apparatus is composed of two or more devices, the two or more devices may be arranged in one device, or may be arranged separately in two or more separate devices.

According to an embodiment of the present disclosure, it is possible to detect the above-ground and underground parts of crops in a field and estimate the condition of the crops based on information obtained from the above-ground and underground parts.

FIG. 1 is a schematic diagram showing the above-ground and underground parts of a grapevine. FIG. 1 is a diagram for explaining an overview of an agricultural management system according to an exemplary embodiment of the present disclosure. 1 is a side view showing a schematic diagram of an example of a work vehicle and a work implement coupled to the work vehicle; FIG. 11 is a side view showing a schematic diagram of another configuration example of the work vehicle. 1 is a block diagram showing an example of the configuration of a work vehicle and a work machine. FIG. 1 is a conceptual diagram showing an example of a work vehicle performing positioning using RTK-GNSS. 4 is a diagram showing an example of an operation terminal and an operation switch group provided inside the cabin. FIG. FIG. 1 is a diagram illustrating a state in which a ground penetrating radar device searches underground. FIG. 1 is a block diagram of a configuration example of an underground radar device. FIG. 2 is a diagram illustrating an example of the arrangement of antenna units of an underground radar device. A side view of a work vehicle showing typically the direction and range in which a laser pulse is emitted from a LiDAR sensor. This is a schematic diagram of a LiDAR sensor viewed vertically from above. FIG. 1 is a schematic diagram showing the distribution of underground parts of crops planted in a field. FIG. 2 is a diagram illustrating the left and right lateral sides of the above-ground part of a crop. FIG. 1 shows a schematic diagram of a vineyard having five rows of trees. FIG. 13 is a diagram showing the result of associating point cloud data of the left and right sides of the above-ground part that constitute the same tree row from a plurality of point cloud data divided into lateral units. FIG. 2 is a diagram illustrating the left and right sides of the underground part of a crop. FIG. 1 shows a schematic diagram of a vineyard having five rows of trees. This figure shows the results of associating data groups from the left and right sides of the underground part that make up the same tree row from multiple underground sensing data groups divided into lateral units. FIG. 2 is an enlarged plan view showing a cross section of a trunk, which is part of the underground portion of a tree. This figure shows a table containing identification information for the entire crop, obtained by combining the table containing identification information for the above-ground parts shown in Figure 12 and the table containing identification information for the underground parts shown in Figure 15. FIG. 2 is a diagram showing a schematic shape of a portion of a row of trees. FIG. 2 is a diagram showing a schematic appearance of one tree included in a row of trees. FIG. 2 is a diagram showing an example of a farm field map. FIG. 13 is a diagram showing a schematic diagram of a distribution of tree trunks in a row, generated based on a farm field map with location information. 11 is a flowchart showing an outline of a process flow of an estimation method for estimating crop information. 4 is a flowchart showing an example of a steering control operation during automatic driving executed by the control device. FIG. 2 is a diagram showing an example of a work vehicle traveling along a target route. FIG. 13 is a diagram showing an example of a work vehicle in a position shifted to the right from the target route. FIG. 13 is a diagram showing an example of a work vehicle in a position shifted to the left from a target route. FIG. 13 is a diagram showing an example of a work vehicle facing in an inclined direction with respect to a target route.

(Definition of terms)
In this disclosure, "agricultural machinery" refers to machinery used for agricultural purposes. Examples of agricultural machinery include tractors, harvesters, rice transplanters, riding management machines, vegetable transplanters, mowers, sowing machines, fertilizer applicators, and agricultural mobile robots. Not only a working vehicle such as a tractor functions as an "agricultural machinery" by itself, but also a working implement attached to or towed by the working vehicle and the working vehicle as a whole may function as one "agricultural machinery". Agricultural machinery performs agricultural work such as plowing, sowing, pest control, fertilization, planting crops, or harvesting on the ground in a field. These agricultural works are sometimes referred to as "ground work" or simply "work". Traveling of a vehicle-type agricultural machine while performing agricultural work is sometimes referred to as "work travel".

"Autonomous driving" means that the movement of the agricultural machine is controlled by the action of a control device, not by manual operation by the driver. An agricultural machine that performs automatic driving is sometimes called an "automatic driving agricultural machine" or a "robot agricultural machine". During automatic driving, not only the movement of the agricultural machine but also the operation of agricultural work (for example, the operation of the working machine) may be automatically controlled. When the agricultural machine is a vehicle-type machine, the traveling of the agricultural machine by automatic driving is called "automatic driving". The control device may control at least one of steering, adjustment of the moving speed, and starting and stopping of the movement required for the movement of the agricultural machine. When controlling a work vehicle equipped with a working machine, the control device may control operations such as raising and lowering the working machine, starting and stopping the operation of the working machine. The movement by automatic driving may include not only the movement of the agricultural machine toward the destination along a predetermined route, but also the movement of the agricultural machine following a tracking target. An agricultural machine that performs automatic driving may move partially based on the instructions of a user. In addition, an agricultural machine that performs automatic driving may operate in a manual driving mode in which the agricultural machine moves by manual operation of the driver in addition to the automatic driving mode. Steering an agricultural machine by the action of a control device, rather than manually, is called "automatic steering." A part or all of the control device may be external to the agricultural machine. Communication of control signals, commands, data, etc. may take place between the agricultural machine and a control device external to the agricultural machine. An agricultural machine that performs automatic driving may move autonomously while sensing the surrounding environment, without a human being being involved in controlling the movement of the agricultural machine. An agricultural machine capable of autonomous movement can travel unmanned in a field or outside the field (e.g., on a road). During autonomous movement, the machine may detect obstacles and perform obstacle avoidance operations.

A "work plan" is data that schedules one or more agricultural works to be performed by an agricultural machine. The work plan may include, for example, information indicating the order of agricultural works to be performed by the agricultural machine and the field on which each agricultural work is to be performed. The work plan may include information on the date and time on which each agricultural work is scheduled to be performed. The work plan may be created by a processing device that communicates with the agricultural machine to manage the agricultural work, or a processing device mounted on the agricultural machine. The processing device may create a work plan based on information input by a user (such as an agricultural manager or farm worker) operating a terminal device. In this specification, a processing device that communicates with the agricultural machine to manage the agricultural work is referred to as a "management device." The management device may manage the agricultural work of multiple agricultural machines. In that case, the management device may create a work plan that includes information on each agricultural work performed by each of the multiple agricultural machines. The work plan may be downloaded by each agricultural machine and stored in a storage device. Each agricultural machine can automatically head to the field to perform the scheduled agricultural work according to the work plan and perform the agricultural work.

An "environmental map" is data that expresses the positions or areas of objects that exist in the environment in which the agricultural machine moves using a specified coordinate system. The coordinate system that defines the environmental map may be, for example, a world coordinate system such as a geographic coordinate system fixed relative to the Earth. The environmental map may also include information other than the positions of objects that exist in the environment (for example, attribute information and other information). Environmental maps include various types of maps, such as point cloud maps or grid maps.

An "underground map" is data that uses a specified coordinate system to represent the positions or distribution of objects that exist underground in a field on which agricultural machinery travels. An example of an object that exists underground is the underground part of a crop. The underground part includes rhizomes and roots, and is also called the root system. The coordinate system that defines the underground map may also be a world coordinate system fixed relative to the Earth, similar to the environmental map. The underground map may include information other than the positions of objects that exist underground (e.g., attribute information and other information). The underground map includes various types of maps, such as point cloud maps or grid maps.

The "field map" is any one of the following data: "environmental map" data, "underground map" data, or integrated data that combines "environmental map" data and "underground map" data. The coordinate system that defines the field map may also be a world coordinate system fixed relative to the Earth, similar to the environmental map or underground map.

A "pose" is the "position and orientation" of an object. A pose in two-dimensional space is defined by three coordinate values, for example (x, y, θ). Here, (x, y) are coordinate values in the XY coordinate system, which is a world coordinate system fixed to the Earth, and θ is the angle with respect to the reference direction. θ corresponds to the yaw angle of the roll angle, pitch angle, and yaw angle in three-dimensional space. The yaw angle represents the amount of rotation around the vertical axis of the agricultural machinery.

(Embodiment)
Hereinafter, an embodiment of the present disclosure will be described. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters and overlapping explanation of substantially the same configuration may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the inventor provides the attached drawings and the following explanation so that those skilled in the art can fully understand the present disclosure, and does not intend to limit the subject matter described in the claims by them. In the following description, components having the same or similar functions are given the same reference numerals.

The following embodiments are illustrative, and the technology of the present disclosure is not limited to the following embodiments. For example, the numerical values, shapes, materials, steps, order of steps, layout of the display screen, and the like shown in the following embodiments are merely examples, and various modifications are possible as long as no technical contradictions arise. In addition, one aspect can be combined with another aspect as long as no technical contradictions arise.

The crop detection system in the embodiment of the present disclosure may be mounted on a work vehicle such as an agricultural machine or a construction machine. In the embodiment of the present disclosure, the crop detection system is mounted on an agricultural machine. The crop detection system is a system for acquiring information on not only the above-ground parts of a crop planted in a field, but also the underground parts. The crop detection system includes one or more sensors mounted on the agricultural machine. The crop detection system may use one or more sensors to sense the above-ground and underground (or underground) parts of a field in which a crop is planted, and integrate the sensing data obtained by sensing the above-ground and underground parts. The crop detection system makes it possible to estimate or understand the growth state of the entire crop, including the above-ground and underground parts, based on the integrated sensing data. The state of the crop means, for example, the growth state of the crop or information on the growth of the crop. The state of the crop is represented, for example, by the shape, size (thickness of the root, thickness of the stem), root density, or position information in the field of the above-ground and/or underground parts.

[1. Above-ground and below-ground parts of crops]
A crop is composed of a part existing above ground and a part existing underground. The former is called the aboveground part, and the latter is called the underground part or root system. The underground part includes roots and rhizomes. The roots of a crop grow irregularly underground as the crop grows. In addition, the way the roots grow may vary from year to year due to various factors that affect the growth of the roots, such as pruning techniques, irrigation, and the year. In the embodiment of the present disclosure, grapes are used as an example of a crop to explain the schematic structure of the underground part.

1 is a schematic diagram of the above-ground and underground parts of a grapevine. A grapevine consists of the crown, which is observed above the ground, the above-ground part including the trunk, and the underground part including the root system. Most of the underground part of the grapevine is located within a depth of 100 cm from the ground, and individual roots can grow to a depth of 9 m. Most of the fine roots are located at a depth of 10 cm to 60 cm, with the highest root density occurring at a depth of about 20 cm from the ground. The taproot is usually found at a depth of 18 cm to 80 cm from the ground. The distribution of the roots may depend on the soil characteristics, the impermeable layer, and the variety of the root stock. Grapevine roots are known to be highly branched and can grow to various depths and directions. According to several studies, grapevine roots grow mainly between flowering and véraison (i.e., as the grapes mature and color), and typically do not grow near harvest time, and very little after harvest or in early spring. Even during the dormant period, roots can grow if the soil temperature is high enough. Also, because the above-ground parts are influenced by the underground parts, the growth of the crown and trunk can vary from year to year.

In this way, the distribution of the above-ground and below-ground parts of a field in which crops are planted is specific to each individual crop, forming a unique pattern of the below-ground parts. In particular, each of the multiple crops planted in the field forms a unique pattern of the below-ground parts. Therefore, by combining the below-ground parts of multiple crops, a unique pattern of the entire below-ground part of the field is formed as a whole. The distribution of the entire below-ground part of the field can change with the growth of roots due to the various factors mentioned above. In this respect, the dynamic below-ground part differs from static structures that may exist underground, such as pipes or rocks.

2. System configuration example
Hereinafter, an embodiment in which the technology of the present disclosure is applied to a work vehicle such as a tractor will be mainly described. However, the technology of the present disclosure is not limited to work vehicles such as tractors, and can also be applied to other types of agricultural machinery.

FIG. 2 is a diagram for explaining an overview of an agricultural management system according to an exemplary embodiment of the present disclosure. The agricultural management system shown in FIG. 2 includes a work vehicle 100, a terminal device 400, and a management device 600. The agricultural management system may further include an edge computer. The terminal device 400 is a computer used by a user who remotely monitors the work vehicle 100. The management device 600 is a computer managed by a business operator who operates the agricultural management system. The work vehicle 100, the terminal device 400, and the management device 600 can communicate with each other via a network 80. Although one work vehicle 100 is illustrated in FIG. 2, the agricultural management system may include multiple work vehicles or other agricultural machines.

In an embodiment of the present disclosure, a first sensor that acquires above-ground information, which is information about the above-ground parts of the crop, a second sensor that acquires underground information, which is information about the underground parts of the crop, and an information processing device that estimates the state of the crop, including the above-ground parts of the crop and the underground parts of the crop, based on the above-ground information and the underground information, work together to function as a crop detection system. The information processing device is, for example, a processing device provided in the work vehicle 100, a processor in the management device 600, or an edge computer. An example of the first sensor is a camera or a LiDAR sensor. An example of the second sensor is an underground radar device.

The multiple devices included in the crop detection system may be distributed and mounted on one or multiple work vehicles. For example, the first sensor and the second sensor may be mounted on a single agricultural machine, or may be mounted separately on multiple agricultural machines. The crop detection system may also be retrofitted to a work vehicle that does not have these functions. Such a system may be manufactured and sold independently of the work vehicle. The computer program used in such a system may also be manufactured and sold independently of the work vehicle. The computer program may be provided, for example, stored in a computer-readable non-transitory storage medium. The computer program may also be provided by downloading via a telecommunications line (e.g., the Internet).

The work vehicle 100 in the embodiment of the present disclosure is a tractor. The work vehicle 100 can be equipped with a work implement at either the rear or the front, or both. The work vehicle 100 can travel within a field while performing agricultural work according to the type of work implement. The work vehicle 100 may also travel within or outside a field without a work implement attached.

The work vehicle 100 in the embodiment of the present disclosure has an automatic driving function. That is, the work vehicle 10 can travel by the action of a control device, not manually. However, the automatic driving function is not essential. The control device in the embodiment of the present disclosure is provided inside the work vehicle 100, and can control both the speed and steering of the work vehicle 100. The work vehicle 100 can travel automatically not only in a field, but also outside the field (e.g., on a road).

The work vehicle 100 is equipped with a device such as a GNSS receiver used for positioning. The control device of the work vehicle 100 automatically drives the work vehicle 100 based on the position of the work vehicle 100 and information on the target route or driving route. In this specification, the target route may be referred to as the "planned driving route". In addition to controlling the driving of the work vehicle 100, the control device also controls the operation of the work machine. This allows the work vehicle 100 to perform agricultural work using the work machine while automatically driving in the field. Furthermore, the work vehicle 100 can automatically drive along a road outside the field (e.g., a farm road or a general road) along the target route. The work vehicle 100 automatically drives along a road outside the field while utilizing data output from a sensing device such as a camera or a LiDAR sensor. The sensing device functions as a distance measurement sensor. The distance measurement sensor can function as a first or second sensor.

The management device 600 is a computer that manages agricultural work by the work vehicle 100. The management device 600 may be, for example, a server computer that centrally manages information about a field on the cloud and supports agriculture by utilizing data on the cloud. The management device 600, for example, creates a work plan for the work vehicle 100 and causes the work vehicle 100 to perform agricultural work according to the work plan. The management device 600 generates a target route in the field based on information input by a user using the terminal device 400 or other devices. The management device 600 may further generate and edit an environmental map based on data collected by the work vehicle 100 or other moving bodies using a sensing device such as a LiDAR sensor. The management device 600 transmits the generated work plan, target route, and data of the environmental map or field map to the work vehicle 100. The work vehicle 100 automatically moves and performs agricultural work based on the data.

The terminal device 400 is a computer used by a user located away from the work vehicle 100. The terminal device 400 shown in FIG. 2 is a laptop computer, but is not limited to this. The terminal device 400 may be a stationary computer such as a desktop PC (personal computer), or a mobile terminal such as a smartphone or tablet computer. The terminal device 400 can be used to remotely monitor the work vehicle 100 or remotely operate the work vehicle 100. For example, the terminal device 400 can display on a display an image captured by one or more cameras (imaging devices) equipped on the work vehicle 100. The terminal device 400 can also display on a display a setting screen for a user to input information required to create a work plan for the work vehicle 100 (for example, a schedule for each agricultural work). When a user inputs the required information on the setting screen and performs a transmission operation, the terminal device 400 transmits the input information to the management device 600. The management device 600 creates a work plan based on the information. The terminal device 400 may further have a function for displaying a setting screen on the display for the user to input information necessary to set a target route.

Below, an example of the system configuration according to an embodiment of the present disclosure is described in more detail.

Figure 3A is a side view that shows a schematic example of a work vehicle 100 and a work implement 300 coupled to the work vehicle 100. The work vehicle 100 in the embodiment of the present disclosure can operate in both a manual driving mode and an automatic driving mode. In the automatic driving mode, the work vehicle 100 can travel unmanned. The work vehicle 100 can be automatically driven both in and outside of a field.

[3. Example of work vehicle configuration]
As shown in Fig. 3A, the work vehicle 100 includes a vehicle body 101, a prime mover (engine) 102, and a transmission 103. The vehicle body 101 is provided with wheels 104 with tires, and a cabin 105. The wheels 104 include a pair of front wheels 104F and a pair of rear wheels 104R. A driver's seat 107, a steering device 106, an operation terminal 200, and a group of switches for operation are provided inside the cabin 105. When the work vehicle 100 travels for work in a field, crawlers instead of tires may be attached to one or both of the front wheels 104F and the rear wheels 104R.

The work vehicle 100 may be equipped with at least one sensing device that senses the environment surrounding the work vehicle 100 and the underground, and an information processing device that processes sensing data output from the at least one sensing device. In the example shown in FIG. 3A, the work vehicle 100 is equipped with multiple sensing devices. The sensing devices include multiple cameras 120, multiple obstacle sensors 130, an underground radar device 140, and a LiDAR sensor 220.

The LiDAR sensor 220 is an example of a distance measurement sensor, and detects the distance to the above-ground parts of crops that exist within a measurement range set around the work vehicle 100 as ground information. The underground radar device 140 is also an example of a distance measurement sensor, and detects the distance to the underground parts of crops that exist within the measurement range as underground information.

The camera 120 may be installed, for example, on the front, rear, left and right sides of the work vehicle 100. The camera 120 captures the environment around the work vehicle 100 and generates image data. The image acquired by the camera 120 may be output to an information processing device mounted on the work vehicle 100 and transmitted to a terminal device 400 for remote monitoring. The image may be used to monitor the work vehicle 100 during unmanned driving. The camera 120 may also be used to generate images for recognizing surrounding features or obstacles, white lines, signs, or indications when the work vehicle 100 travels on roads outside the field (farm roads or general roads). In the embodiment of the present disclosure, the camera 120 is an RGB camera. While the work vehicle 100 is traveling in the field, the camera 120 detects images of the aboveground parts of crops present within a measurement range set around the work vehicle 100 as ground information.

In the example of FIG. 3A, the underground radar device 140 is disposed under the front of the vehicle body 101. The underground radar device 140 may be disposed in other positions, such as the lower center of the vehicle body 101 or the underside of the rear of the vehicle body 101. The number of underground radar devices used is not limited to one, and may be two or more. FIG. 3B is a side view showing another configuration example of the work vehicle 100. The work vehicle 100 may be equipped with, for example, two underground radar devices 140. In this case, one may be disposed under the front of the vehicle body 101, and the other may be disposed under the rear of the vehicle body 101. Alternatively, in cases where it is difficult to attach the underground radar device to the rear of the vehicle body due to interference with the work machine, the underground radar device may be attached to the work machine that performs ground work on the ground, or the underground radar device may be attached to the bottom surface of the vehicle body 101.

The underground radar device 140 is configured to radiate electromagnetic waves to the ground and receive reflected waves from the ground. The underground radar device 140 detects the distance to the underground part of the crop present in the measurement range set around the work vehicle 100 by transmitting electromagnetic waves. The underground radar device 140 is an exploration device that explores and detects the presence of an object underground by utilizing the phenomenon in which electromagnetic waves are reflected at a physical boundary surface in the ground due to a change in the relative dielectric constant. The underground radar device 140 repeatedly acquires sensing data including information on the spatial distribution of structures present underground. The spatial distribution may be a distribution in a two-dimensional or three-dimensional space. Hereinafter, the spatial distribution will be simply referred to as "distribution". The underground radar device 140 in the embodiment of the present disclosure repeatedly acquires sensing data including information on the distribution of the underground part of the crop in the ground in the field while the work vehicle 100 is traveling in the field. In this specification, information on the underground part of the crop, more specifically, information on the distribution of the underground part of the crop in the ground in the field will be referred to as "underground information". The underground information is represented by underground sensing data, and includes, for example, information on the shape, size, and location of the underground parts in the field. On the other hand, information on the above-ground parts of the crop, more specifically, information on the distribution of the above-ground parts of the crop above ground in the field, is called "above-ground information." The above-ground information is represented by point cloud data, and includes, for example, information on the shape, size, and location of the above-ground parts in the field.

Figure 7A is a diagram that shows a schematic diagram of an underground radar device 140 exploring underground. Figure 7B is a block diagram of an example of the configuration of the underground radar device 140. Figure 7C is a diagram that shows a schematic diagram of an example of the arrangement of the antenna unit 141 of the underground radar device 140.

The underground radar device 140 in the embodiment of the present disclosure includes a plurality of antenna units (multi-channel) 141 each including a transmitting antenna T and a receiving antenna R, a receiver 142 connected to each of the plurality of receiving antennas R, a pulse generator 143 connected to each of the plurality of transmitting antennas T, a control device 144 that controls the operation of the receiver 142 and the pulse generator 143, and a storage device 145. The underground radar device 140 illustrated in FIG. 7B and FIG. 7C is a three-dimensional underground radar device including a plurality of antenna units 141. The length of the underground radar device 140 along the width direction of the vehicle body 101 (Y direction shown in FIG. 7C) is, for example, 100 cm to 200 cm. The plurality of transmitting antennas T are arranged at equal intervals (for example, tens of mm) along the width direction of the vehicle body 101, and the plurality of receiving antennas R can also be arranged at equal intervals (for example, tens of mm) along the width direction of the vehicle body 101. The three-dimensional underground radar device makes it possible to simultaneously obtain underground exploration data for multiple underground planes parallel to the XZ plane shown in FIG. 7A at intervals between the receiving antennas R. Based on the underground exploration data, the distribution of the underground part of the ground can be obtained as high-resolution three-dimensional information and visualized. This three-dimensional information can be visualized, thereby making it possible to obtain three-dimensional image data. In addition, the two-dimensional or three-dimensional shape or size of the underground part can be estimated based on the three-dimensional information showing the distribution of the underground part of the ground.

The underground radar device 140 can transmit and receive electromagnetic waves in a frequency band from 200 MHz to 3000 MHz, for example, according to a step frequency method. However, the radar method or modulation method is not limited to this and may be any method. For example, an impulse method may be used as the radar method. The underground radar device 140 can generate a time domain waveform corresponding to the reflected wave from the ground by, for example, emitting electromagnetic waves to the ground while gradually increasing the sine wave frequency between the above frequency bands, receiving the reflected wave from the ground, and performing an inverse Fourier transform on the frequency spectrum data obtained. Since the reflected waveform depends on the position or shape of the reflector 77, it is possible to estimate or identify the position (depth in the Z direction shown in FIG. 7C) or shape of the reflector 77 based on the received signal.

The sensing data output from the underground radar device 140 can be processed by the control system of the work vehicle 100, the management device 600, or an edge computer.

The measurement range of the underground radar device 140 can be determined by the length of the underground radar device 140 along the width direction of the vehicle body (Y direction shown in FIG. 7C), as shown in FIG. 7C. This measurement range is called the "second measurement range."

The work vehicle 100 may be equipped with one or more LiDAR sensors. The LiDAR sensor repeatedly outputs sensor data indicating the distance and direction to each measurement point of an object in the surrounding environment, or the two-dimensional or three-dimensional coordinate values of each measurement point. The LiDAR sensor detects the distance to the above-ground part of a crop that is present within a measurement range set around the work vehicle 100 by emitting a laser pulse while the work vehicle 100 is traveling mainly within a field. The sensor data output from the LiDAR sensor may be processed by the control system of the work vehicle 100.

In the example shown in FIG. 3A, the LiDAR sensor 220 is disposed at the front of the cabin 105 of the work vehicle 100. The LiDAR sensor 220 is not limited to being disposed at the front of the cabin 105, but may be disposed on the top of the hood of the work vehicle 100 or at the rear of the cabin 105, etc. The number of LiDAR sensors used is not limited to one, but may be two or more.

The LiDAR sensor 220 successively emits pulses of a laser beam (hereinafter abbreviated as "laser pulses") while changing the emission direction, and can measure the distance and angle to the position of each reflection point based on the time difference between the emission time and the time when the reflected pulse of each laser pulse is acquired. The LiDAR sensor 220 repeatedly acquires sensing data including information regarding the spatial distribution of structures present in the vicinity. The spatial distribution can be a distribution in two-dimensional or three-dimensional space. The LiDAR sensor 220 in the embodiment of the present disclosure repeatedly acquires sensing data including information regarding the distribution of the above-ground parts of crops on the ground in the field while the work vehicle 100 is traveling in the field.

FIG. 8A is a side view of the work vehicle 100, which shows a schematic diagram of the direction and range of the laser pulse emitted from the LiDAR sensor 220. The laser pulse emitted from the LiDAR sensor 220 in the direction of a negative elevation angle is represented by a thick dashed line, and the laser pulse emitted from the LiDAR sensor 220 in the direction of a positive elevation angle is represented by a thin dashed line. Of the multiple laser pulses emitted from the LiDAR sensor 220 at different elevation angles, the laser pulse emitted in the direction of a negative elevation angle may be reflected by the ground if it does not hit the crop. On the other hand, the laser pulse emitted in the direction of a positive elevation angle does not form a reflection point if it does not hit the crop, and the reflection point if it hits the crop may be the stem, branches, leaves, etc. of the crop. For this reason, in the embodiment of the present disclosure, ground information, which is information on the above-ground part of the crop, is obtained by utilizing reflections from the stem, branches, and leaves of the crop. Specifically, ground information can be obtained based on the reflection points of laser pulses whose elevation angles are within a predetermined range among multiple laser pulses emitted from the LiDAR sensor 220 at different elevation angles. Also, ground information can be obtained based on reflection points whose heights are lower than the average height of the crops.

There are multiple types of LiDAR sensor 220 depending on the method of emitting laser pulses. For example, there is a scan type sensor that acquires information on the distance distribution of objects in space by scanning laser pulses, and a flash type sensor that acquires information on the distance distribution of objects in space by using light that is diffused over a wide area. A scan type LiDAR sensor uses light with a higher intensity than a flash type LiDAR sensor, and can acquire distance information from farther away. On the other hand, a flash type LiDAR sensor has a simple structure and can be manufactured at a relatively low cost, making it suitable for applications that do not require strong light. In the embodiments of the present disclosure, an example in which a scan type LiDAR sensor is mainly used is described, but the present disclosure is not limited thereto.

Figure 8B is a schematic diagram of LiDAR sensor 220 as viewed vertically from above. In an embodiment of the present disclosure, as shown in Figure 8B, a first measurement range from the left wheel to the right wheel side facing the travel direction (X direction) of work vehicle 100 is set as the measurement range of LiDAR sensor 220. LiDAR sensor 220 is disposed in a central position on the left and right sides of the upper part of the front of cabin 105. As a result, a first measurement range of LiDAR sensor 220 is set to a predetermined range on the left, right and front sides of the vehicle body with the left and right center line of work vehicle 100 as the axis of symmetry.

Examples of the structure and operation of LiDAR sensors are described in detail in the applicant's published international application WO 2022/107586, the entire disclosure of which is incorporated herein by reference.

Refer to FIG. 3A again. The obstacle sensors 130 shown in FIG. 3A are provided at the front and rear of the cabin 105. The obstacle sensors 130 may be located at other locations as well. For example, one or more obstacle sensors 130 may be provided at any position on the side, front, and rear of the vehicle body 101. The obstacle sensor 130 may include, for example, a laser scanner or ultrasonic sonar. The obstacle sensor 130 is used to detect surrounding obstacles during autonomous driving and to stop or detour the work vehicle 100.

The work vehicle 100 in the embodiment of the present disclosure further includes a positioning unit 110. The positioning unit 110 outputs a vehicle body position, which is the position of the work vehicle 100. The vehicle body position is represented by position information including, for example, the latitude and longitude of the work vehicle 100 in the world coordinate system. However, the positioning unit is not essential. An example of the positioning unit 110 is a GNSS unit. Hereinafter, the positioning unit 110 is referred to as the GNSS unit 110. The GNSS unit 110 includes a GNSS receiver. The GNSS receiver may include an antenna that receives signals from GNSS satellites and a processor that calculates the position of the work vehicle 100 based on the signals received by the antenna. The GNSS unit 110 receives satellite signals transmitted from multiple GNSS satellites and performs positioning based on the satellite signals. GNSS is a general term for satellite positioning systems such as GPS (Global Positioning System), QZSS (Quasi-Zenith Satellite System, e.g., Michibiki), GLONASS, Galileo, and BeiDou. In the embodiment of the present disclosure, the GNSS unit 110 is provided on the top of the cabin 105, but may be provided in another location.

The GNSS unit 110 may include an inertial measurement unit (IMU). Signals from the IMU may be used to supplement position data. The IMU may measure the tilt and minute movements of the work vehicle 100. Data acquired by the IMU may be used to supplement position data based on satellite signals, improving positioning performance.

The information processing device of the work vehicle 100 in the embodiment of the present disclosure is capable of estimating the condition of the crops using at least one of the positioning results from the GNSS unit 110, the image data acquired by the camera 120, and the sensor data acquired by the LiDAR sensor 220, in addition to the sensing data acquired by the underground radar device 140. The information processing device of the work vehicle 100 may also perform self-position estimation based on these data and the positioning results. This allows the position of the work vehicle 100 to be identified with greater accuracy.

The prime mover 102 may be, for example, a diesel engine. An electric motor may be used instead of a diesel engine. The transmission 103 can change the propulsive force and travel speed of the work vehicle 100 by changing gears. The transmission 103 can also switch the work vehicle 100 between forward and reverse.

The steering device 106 includes a steering wheel, a steering shaft connected to the steering wheel, and a power steering device that assists steering by the steering wheel. The front wheels 104F are steered wheels, and the driving direction of the work vehicle 100 can be changed by changing the turning angle (also called the "steering angle"). The steering angle of the front wheels 104F can be changed by operating the steering wheel. The power steering device includes a hydraulic device or an electric motor that supplies an auxiliary force to change the steering angle of the front wheels 104F. When automatic steering is performed, the steering angle is automatically adjusted by the force of the hydraulic device or the electric motor under the control of a control device arranged in the work vehicle 100.

A coupling device 108 is provided at the rear of the vehicle body 101. The coupling device 108 includes, for example, a three-point support device (also called a "three-point link" or "three-point hitch"), a PTO (Power Take Off) shaft, a universal joint, and a communication cable. The coupling device 108 can attach and detach the working machine 300 to the work vehicle 100. The coupling device 108 can raise and lower the three-point link using, for example, a hydraulic device, to change the position or attitude of the working machine 300. In addition, power can be sent from the work vehicle 100 to the working machine 300 via the universal joint. The work vehicle 100 can make the working machine 300 perform a predetermined task while pulling the working machine 300. The coupling device may be provided at the front of the vehicle body 101. In that case, the working machine can be connected to the front of the work vehicle 100.

The working machine 300 shown in FIG. 3A is a sprayer that sprays a chemical on crops, but the working machine 300 is not limited to a sprayer. For example, any working machine, such as a seeder, spreader, transplanter, mower, rake, baler, harvester, rotary tiller, or harrow, can be connected to the work vehicle 100 and used.

The work vehicle 100 shown in FIG. 3A is capable of manned operation, but may also be capable of unmanned operation only. In that case, components necessary only for manned operation, such as the cabin 105, steering device 106, and driver's seat 107, may not be provided in the work vehicle 100. The unmanned work vehicle 100 can travel autonomously or by remote control by a user.

Figure 4 is a block diagram showing an example configuration of the work vehicle 100 and the work machine 300. The work vehicle 100 and the work machine 300 can communicate with each other via a communication cable included in the coupling device 108. The work vehicle 100 can communicate with the terminal device 400 and the management device 600 via the network 80.

In the example of FIG. 4, the work vehicle 100 includes a GNSS unit 110, a camera 120, an obstacle sensor 130, an underground radar device 140, an operation terminal 200, and a LiDAR sensor 220, as well as a sensor group 150 for detecting the operating state of the work vehicle 100, a control system 160, a communication device 190, an operation switch group 210, and a drive unit 240. These components are connected to each other so that they can communicate with each other via a bus. The GNSS unit 110 includes a GNSS receiver 111, an RTK receiver 112, an inertial measurement unit (IMU) 115, and a processing circuit 116. The sensor group 150 includes a steering wheel sensor 152, a turning angle sensor 154, and an axle sensor 156. The control system 160 includes an information processing device 161, a storage device 170, and a control device 180. The control device 180 includes multiple electronic control units (ECUs) 181 to 185. The work machine 300 includes a drive device 340, a control device 380, and a communication device 390. Note that FIG. 4 shows components that are relatively highly related to the operation of the autonomous driving of the work vehicle 100, and does not show other components.

The GNSS receiver 111 in the GNSS unit 110 receives satellite signals transmitted from multiple GNSS satellites and generates GNSS data based on the satellite signals. The GNSS data is generated in a predetermined format, such as the NMEA-0183 format. The GNSS data may include, for example, values indicating the identification number, elevation angle, azimuth angle, and reception strength of each satellite from which the satellite signal is received.

The GNSS unit 110 shown in FIG. 4 uses RTK (Real Time Kinematic)-GNSS to perform positioning of the work vehicle 100. FIG. 5 is a conceptual diagram showing an example of a work vehicle 100 performing positioning using RTK-GNSS. In positioning using RTK-GNSS, in addition to satellite signals transmitted from multiple GNSS satellites 50, correction signals transmitted from a reference station 60 are used. The reference station 60 may be installed near the field where the work vehicle 100 performs work travel (for example, within 10 km from the work vehicle 100). The reference station 60 generates a correction signal, for example in RTCM format, based on the satellite signals received from the multiple GNSS satellites 50 and transmits it to the GNSS unit 110. The RTK receiver 112 includes an antenna and a modem, and receives the correction signal transmitted from the reference station 60. The processing circuit 116 of the GNSS unit 110 corrects the positioning results obtained by the GNSS receiver 111 based on the correction signal. By using RTK-GNSS, it is possible to perform positioning with an accuracy of, for example, a few centimeters. Position information including latitude, longitude, and altitude information is obtained by highly accurate positioning using RTK-GNSS. The GNSS unit 110 calculates the position of the work vehicle 100 at a frequency of, for example, about 1 to 10 times per second.

The positioning method is not limited to RTK-GNSS, and any positioning method (such as interferometric positioning or relative positioning) that can obtain position information with the required accuracy can be used. For example, positioning may be performed using a Virtual Reference Station (VRS) or a Differential Global Positioning System (DGPS). If position information with the required accuracy can be obtained without using a correction signal transmitted from the reference station 60, the position information may be generated without using a correction signal. In that case, the GNSS unit 110 does not need to be equipped with an RTK receiver 112.

Even when RTK-GNSS is used, in places where a correction signal from the reference station 60 cannot be obtained (for example, on a road far from a field), the position of the work vehicle 100 is estimated by other methods rather than relying on the signal from the RTK receiver 112. For example, the position of the work vehicle 100 can be estimated by matching the data output from the camera 120 and/or the LiDAR sensor 220 with a highly accurate environmental map. Also, when the work vehicle 100 travels through a field, the position of the work vehicle 100 can be estimated by matching the sensing data output from the underground radar device 140 with a highly accurate underground map.

The GNSS unit 110 in the embodiment of the present disclosure further includes an IMU 115. The IMU 115 may include a three-axis acceleration sensor and a three-axis gyroscope. The IMU 115 may include an orientation sensor such as a three-axis geomagnetic sensor. The IMU 115 functions as a motion sensor and can output signals indicating various quantities such as the acceleration, speed, displacement, and attitude of the work vehicle 100. The processing circuit 116 can estimate the position and orientation of the work vehicle 100 with higher accuracy based on the signal output from the IMU 115 in addition to the satellite signal and the correction signal. The signal output from the IMU 115 can be used to correct or complement the position calculated based on the satellite signal and the correction signal. The IMU 115 outputs signals at a higher frequency than the GNSS receiver 111. Using the high-frequency signal, the processing circuit 116 can measure the position and orientation of the work vehicle 100 at a higher frequency (e.g., 10 Hz or more). Instead of the IMU 115, a three-axis acceleration sensor and a three-axis gyroscope may be provided separately. The IMU 115 may be provided as a device separate from the GNSS unit 110.

The camera 120 is an imaging device that captures the environment around the work vehicle 100. The camera 120 includes an image sensor, such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The camera 120 may also include an optical system including one or more lenses, and a signal processing circuit. The camera 120 captures the environment around the work vehicle 100 while the work vehicle 100 is traveling, and generates image (e.g., video) data. The camera 120 can capture video at a frame rate of, for example, 3 frames per second (fps) or more. The images generated by the camera 120 can be used when a remote observer uses the terminal device 400 to check the environment around the work vehicle 100, and can be used for positioning or obstacle detection. In particular, the images generated by the camera 120, including the above-ground parts of the crop, are used to estimate the condition of the crop.

As shown in FIG. 3A, multiple cameras 120 may be provided at different positions on the work vehicle 100, or a single camera may be provided. A visible camera that generates visible light images and an infrared camera that generates infrared images may be provided separately. Both a visible camera and an infrared camera may be provided as cameras that generate images for surveillance. The infrared camera may also be used to detect obstacles at night.

The obstacle sensor 130 detects objects present around the work vehicle 100. The obstacle sensor 130 may include, for example, a laser scanner or an ultrasonic sonar. When an object is present closer than a predetermined distance from the obstacle sensor 130, the obstacle sensor 130 outputs a signal indicating the presence of an obstacle. Multiple obstacle sensors 130 may be provided at different positions on the work vehicle 100. For example, multiple laser scanners and multiple ultrasonic sonars may be arranged at different positions on the work vehicle 100. By providing such a large number of obstacle sensors 130, blind spots in monitoring obstacles around the work vehicle 100 can be reduced.

The steering wheel sensor 152 measures the rotation angle of the steering wheel of the work vehicle 100. The turning angle sensor 154 measures the turning angle of the front wheels 104F, which are the steered wheels. The measurements by the steering wheel sensor 152 and the turning angle sensor 154 are used for steering control by the control device 180.

The axle sensor 156 measures the rotational speed of the axle connected to the wheel 104, i.e., the number of rotations per unit time. The axle sensor 156 may be, for example, a sensor that uses a magnetoresistive element (MR), a Hall element, or an electromagnetic pickup. The axle sensor 156 outputs, for example, a numerical value indicating the number of rotations per minute (unit: rpm) of the axle. The axle sensor 156 is used to measure the speed of the work vehicle 100.

The drive device 240 includes various devices necessary for the travel of the work vehicle 100 and the driving of the work equipment 300, such as the prime mover 102, the transmission 103, the steering device 106, and the coupling device 108 described above. The prime mover 102 may be equipped with an internal combustion engine such as a diesel engine. The drive device 240 may be equipped with an electric motor for traction instead of or in addition to the internal combustion engine.

An example of the information processing device 161 is a microprocessor or a microcontroller. The information processing device 161 processes, for example, sensing data output from the underground radar device 140. The information processing device 161 in the embodiment of the present disclosure refers to a plurality of pieces of ground information detected by the first sensor and a plurality of pieces of underground information detected by the second sensor, extracts predetermined underground information common to a predetermined piece of ground information from the plurality of pieces of ground information, associates the extracted predetermined underground information with the predetermined ground information, and estimates the state of the crop from the associated predetermined ground information and predetermined underground information. The estimation of the state of the crop will be described in detail later. In addition, the information processing device 161 in the embodiment of the present disclosure can create a crop shape based on the ground information and the underground information, and create a field map by expanding the created crop shape in a field indicating the field. The data of the field map may be generated or edited by a processor or an edge computer in the management device 600. It is also possible to cause the control device 180, which will be described later, to execute a series of processes performed by the information processing device 161.

The storage device 170 includes one or more storage media such as a flash memory or a magnetic disk. The storage device 170 stores various data generated by the GNSS unit 110, the camera 120, the obstacle sensor 130, the underground radar device 140, the sensor group 150, the information processing device 161, and the control device 180. The data stored in the storage device 170 may include a field map in the field in which the work vehicle 100 travels, an environmental map in the environment in which the work vehicle 100 travels, an underground map, and data on a target route for automatic driving. The environmental map includes, for example, information on a plurality of fields in which the work vehicle 100 performs agricultural work and roads in the surrounding area. The environmental map and the target route may be generated by a processor in the management device 600. The control device 180 may have a function of generating or editing the environmental map and the target route. The control device 180 can edit the environmental map and the target route acquired from the management device 600 according to the traveling environment of the work vehicle 100. The storage device 170 also stores the work plan data received by the communication device 190 from the management device 600.

The storage device 170 also stores computer programs that cause the information processing device 161 and each ECU in the control device 180 to execute various operations described below. Such computer programs may be provided to the work vehicle 100 via a storage medium (e.g., a semiconductor memory or an optical disk) or an electric communication line (e.g., the Internet). Such computer programs may be sold as commercial software.

The control device 180 includes multiple ECUs. The multiple ECUs include, for example, an ECU 181 for speed control, an ECU 182 for steering control, an ECU 183 for implement control, an ECU 184 for automatic driving control, and an ECU 185 for route generation.

The ECU 181 controls the speed of the work vehicle 100 by controlling the prime mover 102, the transmission 103, and the brakes included in the drive unit 240.

The ECU 182 controls the steering of the work vehicle 100 by controlling the hydraulic device or electric motor included in the steering device 106 based on the measurement value of the steering wheel sensor 152.

The ECU 183 controls the operation of the three-point link and the PTO shaft included in the coupling device 108 to cause the working machine 300 to perform a desired operation. The ECU 183 also generates a signal that controls the operation of the working machine 300 and transmits the signal from the communication device 190 to the working machine 300.

The ECU 184 performs calculations and control to realize autonomous driving based on data output from the GNSS unit 110, the camera 120, the obstacle sensor 130, the information processing device 161, and the sensor group 150. For example, the ECU 184 identifies the position of the work vehicle 100 based on data output from at least one of the GNSS unit 110, the camera 120, and the information processing device 161. The ECU 184 may identify the position of the work vehicle 100 based further on data output from the LiDAR sensor 220. When the work vehicle 100 travels particularly in a field, the information processing device 161 may perform self-position estimation based on sensing data output from the underground radar device 140, either alone or in cooperation with the ECU 184. The ECU 184 may correct the position of the work vehicle 100 estimated by the information processing device 161 based on data acquired by the camera 120 or the LiDAR sensor 220. Outside the field, the ECU 184 estimates the position of the work vehicle 100 using data output from the camera 120 or the LiDAR sensor 220. For example, the ECU 184 may estimate the position of the work vehicle 100 by matching the data output from the camera 120 or the LiDAR sensor 220 with an environmental map. During automatic driving, the ECU 184 performs calculations necessary for the work vehicle 100 to travel along the target route based on the estimated position of the work vehicle 100. The ECU 184 sends a speed change command to the ECU 181 and a steering angle change command to the ECU 182. In response to the speed change command, the ECU 181 changes the speed of the work vehicle 100 by controlling the prime mover 102, the transmission 103, or the brakes. In response to the steering angle change command, the ECU 182 changes the steering angle by controlling the steering device 106.

While the work vehicle 100 is traveling, the ECU 185 recognizes obstacles that exist around the work vehicle 100 based on data output from the camera 120 and the obstacle sensor 130. The ECU 185 can also determine the destination of the work vehicle 100 based on the work plan stored in the storage device 170, and determine a target route from the start point of the movement of the work vehicle 100 to the destination point.

Through the operation of these ECUs, the control device 180 cooperates with the information processing device 161 to realize autonomous driving. During autonomous driving, the control device 180 controls the drive device 240 based on the measured or estimated position of the work vehicle 100 and the target route. In this way, the control device 180 can drive the work vehicle 100 along the target route.

The multiple ECUs included in the control device 180 can communicate with each other according to a vehicle bus standard such as CAN (Controller Area Network). A faster communication method such as in-vehicle Ethernet (registered trademark) may be used instead of CAN. In FIG. 4, the information processing device 161 and the ECUs 181 to 185 are shown as individual blocks, but their respective functions may be realized by multiple ECUs. An in-vehicle computer that integrates at least some of the functions of the information processing device 161 and the ECUs 181 to 185 may be provided. The control device 180 may include ECUs other than the ECUs 181 to 185, and any number of ECUs may be provided depending on the functions. For example, the information processing device 161 may be implemented in the control device 180 as an ECU. Each ECU includes a processing circuit including one or more processors.

The communication device 190 is a device including a circuit for communicating with the work machine 300, the terminal device 400, and the management device 600. The communication device 190 includes a circuit for transmitting and receiving signals conforming to the ISOBUS standard such as ISOBUS-TIM between the communication device 390 of the work machine 300. This allows the work machine 300 to perform a desired operation and obtain information from the work machine 300. The communication device 190 may further include an antenna and a communication circuit for transmitting and receiving signals via the network 80 between the communication devices of the terminal device 400 and the management device 600. The network 80 may include, for example, a cellular mobile communication network such as 3G, 4G, or 5G, and the Internet. The communication device 190 may have a function for communicating with a mobile terminal used by a supervisor near the work vehicle 100. Communication between such mobile terminals can be performed in accordance with any wireless communication standard, such as Wi-Fi (registered trademark), cellular mobile communications such as 3G, 4G, or 5G, or Bluetooth (registered trademark).

The operation terminal 200 is a terminal through which a user performs operations related to the traveling of the work vehicle 100 and the operation of the work machine 300, and is also called a virtual terminal (VT). The operation terminal 200 may include a display device such as a touch screen, and/or one or more buttons. The display device may be, for example, a liquid crystal or organic light-emitting diode (OLED) display. By operating the operation terminal 200, a user can perform various operations such as switching the automatic driving mode on/off, recording or editing an environmental map, setting a target route, and switching the work machine 300 on/off. At least some of these operations can also be realized by operating the operation switch group 210. The operation terminal 200 may be configured to be removable from the work vehicle 100. A user at a location away from the work vehicle 100 may operate the detached operation terminal 200 to control the operation of the work vehicle 100. The user may control the operation of the work vehicle 100 by operating a computer on which necessary application software is installed, such as a terminal device 400, instead of the operation terminal 200.

Figure 6 is a diagram showing an example of an operation terminal 200 and an operation switch group 210 provided inside the cabin 105. Inside the cabin 105, an operation switch group 210 including a plurality of switches that can be operated by a user is arranged. The operation switch group 210 may include, for example, a switch for selecting a main or sub-transmission gear shift stage, a switch for switching between an automatic driving mode and a manual driving mode, a switch for switching between forward and reverse, and a switch for raising and lowering the work machine 300. Note that if the work vehicle 100 only performs unmanned driving and does not have a function for manned driving, the work vehicle 100 does not need to have the operation switch group 210.

The drive unit 340 in the work machine 300 shown in FIG. 4 performs the operations necessary for the work machine 300 to perform a specified task. The drive unit 340 includes devices according to the application of the work machine 300, such as a hydraulic device, an electric motor, or a pump. The control device 380 controls the operation of the drive unit 340. The control device 380 causes the drive unit 340 to perform various operations in response to signals transmitted from the work vehicle 100 via the communication device 390. In addition, a signal according to the state of the work machine 300 can also be transmitted from the communication device 390 to the work vehicle 100.

Figure 9 is a diagram showing a schematic distribution of the underground parts of crops planted in a field. Figure 9 shows an example focusing on the underground parts 700 of nine crops among the crops. The underground parts of each crop are spread throughout the ground of the field to form the overall underground part. The distribution or shape of the underground parts differs for each individual crop. Each root extends irregularly in three-dimensional space. For example, the standard root size (diameter) of grapes is about 0.5 cm to 2 cm, with thick roots having a diameter of 2 cm or more and thin roots having a diameter of 0.5 cm or less. Grape roots can reach a point, for example, 2 m to 3 m away from the trunk in the horizontal direction (direction perpendicular to the depth direction of the ground).

While the work vehicle 100 is traveling in the field, the underground radar device 140 acquires sensing data including information about the underground part 700 of the crop underground. When the work vehicle 100 repeatedly travels back and forth along the X direction or Y direction shown in FIG. 9, for example, the underground radar device 140 acquires sensing data including three-dimensional information showing the distribution of the underground part 700 underground. Each piece of sensing data is observation data obtained based on the reflected waves from the ground received by the receiving antenna of the underground radar device 140. For example, by using microwaves of 1500 MHz to 2000 MHz, it is possible to detect roots that exist at a depth of about 30 cm from the ground and have a diameter of 5 mm or more. By using microwaves of a relatively low frequency of about 400 MHz, it is possible to detect roots that exist at a depth of about 2 m from the ground if the roots have a diameter of 2 cm to 4 cm.

7A and 7C are referred to again. When the work vehicle 100 travels, for example, in the X direction, each of the multiple receiving antennas R arranged at a predetermined interval in the width direction (Y direction) of the work vehicle 100 receives reflected waves from the ground, and thus the vertical section profiles of the underground part in the vertical section parallel to the XZ plane shown in FIG. 9 are obtained as many times as the number of receiving antennas R. The information processing device 161 can generate a three-dimensional data group showing the distribution of the underground part in the entire field by connecting the multiple vertical section profiles in the Y direction. The three-dimensional data group can be modeled and visualized. The information processing device 161 can estimate the shape of the underground part based on the data group obtained by observation. The image generated based on the data group obtained by observation is called a radar image. The radar image includes, for example, a part or all of the shape of the underground part.

When the work vehicle 100 is equipped with a pair of underground radar devices at the front and rear of the vehicle body 101, one may be shifted in the width direction of the work vehicle (Y direction shown in FIG. 7C) relative to the other by, for example, half the distance between the antenna units 141. This makes it possible to obtain longitudinal section profile data with twice the resolution in the width direction of the work vehicle compared to when a single underground radar device is used. Obtaining a large amount of data at high resolution is useful for estimating the position or shape of underground parts in the ground, or for constructing highly accurate underground maps or farm field maps showing the distribution of underground parts, as described below.

When the work vehicle 100 has completed traveling through the entire field or a section, a three-dimensional data set is obtained that includes information regarding the distribution of the underground parts of the entire field or a section. The three-dimensional data set can be sliced in the depth direction underground parallel to the XY plane. By slicing the three-dimensional data set in the depth direction, distribution information of the underground parts in a two-dimensional plane (horizontal cross section) at a desired depth is obtained, and a horizontal cross section profile Sn of the underground parts in the horizontal cross section is obtained based on the distribution information. The information processing device 161 can generate a composite radar image in the depth direction by combining multiple radar images corresponding to multiple horizontal cross section profiles S0 to Sn at different depths. Each radar image includes root distribution information that reflects the shape of the roots at the respective depths.

The information processing device 161 may analyze the distribution pattern of data points distributed in two-dimensional or three-dimensional space using, for example, a spatial analysis method for the data group obtained by observation. The distribution of the data points may be represented as raster data, or may be visualized using, for example, a kernel method. Alternatively, the information processing device 161 may calculate the density of the point distribution and generate an image that visualizes them. The image generated in this way is also an example of the radar image mentioned above.

[4. Acquisition of ground information]
Next, an example of a method for acquiring ground information while the work vehicle 100 travels through a farm field will be described with reference to Fig. 10. In the example shown in Fig. 10, the work vehicle 100 travels through a vineyard and acquires ground information of a row 20 of growing grape vines. However, the work vehicle 100 is not limited to grape vines and can acquire ground information of any crop.

As shown in FIG. 10, while the work vehicle 100 is traveling manually or automatically between the rows of trees 20 in a vineyard, repeatedly moving straight ahead and turning, the LiDAR sensor 220 acquires ground information about the rows of trees 20.

In the example shown in FIG. 10, when the work vehicle 100 travels along one of a number of tree rows 20, the above-ground parts of the left side of the tree row 20 facing the side of the work vehicle 100 (the part within the dotted rectangle LU in FIG. 10) are detected, but the above-ground parts of the right side of the tree row 20 opposite the left side (the part within the dashed rectangle RU in FIG. 10) are not detected. In this specification, the left and right directions referred to by the terms right and left as explained with reference to the drawings are absolute directions as seen from the world coordinate system when looking down on the vineyard, and are not relative directions as seen from the mobile body coordinate system fixed to the traveling work vehicle 100.

In the example of FIG. 10, while the work vehicle 100 is traveling along the left side of a certain tree row, the LiDAR sensor 220 emits a laser pulse, acquires sensor data showing the distribution of reflection points of the emitted laser pulse, and further acquires ground information on the above-ground part of the left part of the tree row 20 based on the sensor data. Next, while the work vehicle 100 turns at the end point of the tree row 20 and travels along the right part of the tree row 20, the LiDAR sensor 220 emits a laser pulse, acquires sensor data showing the distribution of reflection points of the emitted laser pulse, and further acquires ground information on the above-ground part of the right part of the tree row 20 based on the sensor data. While the work vehicle 100 is traveling between adjacent tree rows 20, the LiDAR sensor 220 can simultaneously acquire ground information on the above-ground part of the right part of one of the adjacent tree rows 20 and ground information on the above-ground part of the left part of the other adjacent tree row 20. In other words, the LiDAR sensor 220 can obtain ground information about the above-ground parts of the tree rows 20 located within the first measurement range on both sides of the work vehicle 100.

In this way, the LiDAR sensor 220 obtains ground information on the above-ground parts of multiple tree rows 20 for each tree row, and obtains ground information on the above-ground parts of all of the tree rows 20 in the vineyard by repeatedly traveling straight and turning. While the work vehicle 100 travels along the tree rows 20, the LiDAR sensor 220 can continuously obtain ground information on the tree rows 20 along the actual travel path P.

In the above example, a method for acquiring ground information regarding the entire circumference of the above-ground parts of a row of grapevines was described, but this method is merely one example, and acquiring ground information regarding the entire circumference of the above-ground parts of a row of trees is not essential. For example, ground information regarding the above-ground parts of at least one tree included in the row of trees can be acquired on a tree-by-tree basis.

[5. Acquisition of point cloud data and coordinate system transformation]
Fig. 11 is a schematic diagram of a vineyard in which there are five tree rows. Fig. 11 illustrates the above-ground parts of the five tree rows. In the example shown in Fig. 11, there are five tree rows T1 to T5 in the vineyard, but the number of tree rows is of course not limited to this. Furthermore, the left parts of the five tree rows T1 to T5 in Fig. 11 will be referred to as the "left side parts" LU1 to LU5, respectively, and the right parts of the five tree rows 20 will be referred to as the "right side parts" RU1 to RU5, respectively.

The work vehicle 100 proceeds from the starting point S, and while traveling on the outside of the tree row T1, acquires point cloud data of the left side portion LU1 of the ground part of the tree row T1 using the LiDAR sensor 220. After turning at the end point of the tree row T1, the work vehicle 100 enters between the adjacent tree rows T1 and T2, and acquires point cloud data of the right side portion RU1 of the ground part of the tree row T1 and the left side portion LU2 of the ground part of the tree row T2 using the LiDAR sensor 220. The work vehicle 100 proceeds to the end point G while repeatedly traveling along the tree rows and turning, and acquires point cloud data of the left side portions LU3 to LU5 of the ground part of the tree rows T3 to T5 and point cloud data of the right side portions RU2 to RU4 of the ground part of the tree rows T2 to T4 using the LiDAR sensor 220. Finally, the work vehicle 100, while traveling outside the tree row T5, acquires point cloud data of the right side portion RU5 of the above-ground portion of the tree row T5 using the LiDAR sensor 220. In this way, the LiDAR sensor 220 acquires point cloud data of the above-ground portions of the tree rows T1 to T5 throughout the entire vineyard.

The information processing device 161 can treat the point cloud data acquired for each of the left side portions LU1-LU5 on the ground and the right side portions RU1-RU5 on the ground as lateral units in the above-ground portion of each tree row, and store information related to the point cloud data in lateral units in the storage device. In an embodiment of the present disclosure, the GNSS unit 110 equipped in the work vehicle 100 has an IMU 115 and detects the difference in operation between the work vehicle 100 traveling along the tree row and turning. This detection makes it possible to identify the point cloud data acquired continuously by the LiDAR sensor 220 while the work vehicle 100 travels from the start point S to the end point G in lateral units.

The point cloud data of the left side portion LU1 to LU5 of the ground part of the tree row and the point cloud data of the right side portion RU1 to RU5 of the ground part can be distinguished using the positioning data output from the GNSS unit 110. For example, the work vehicle 100 runs from the start point S along the left side portion LU1 of the ground part of the tree row T1, and turns after reaching the end of the left side portion LU1. The information processing device 161 can detect that the direction of travel of the work vehicle 100 changes due to the turning of the work vehicle 100 based on the positioning data output from the GNSS unit 110. According to this detection result, the information processing device 161 can recognize that the work vehicle 100 has reached the end of the left side portion LU1 of the ground part of the tree row T1 from one end (starting point S) to the other end, and that it has reached the end of the right side portion RU1 from the other end of the left side portion LU1. In this way, by utilizing the change in the traveling direction of the work vehicle 100, the information processing device 161 can recognize that sensing has shifted from the left side portion LU1 of the tree row T1 to the right side portion RU1, and can separate the point cloud data of the left side portion LU1 from the point cloud data of the right side portion RU1. Similarly, for the tree rows T2 to T5, the information processing device 161 can separate the point cloud data of the left side portions LU2 to LU5 on the ground from the point cloud data of the right side portions RU2 to RU5 on the ground each time the work vehicle 100 turns.

The information processing device 161 in the embodiment of the present disclosure assigns identification information to the point cloud data of the divided left side portions LU1 to LU5 of the ground portion and the right side portions RU1 to RU5 of the ground portion, and identifies the divided point cloud data by side. The identification information will be described later.

In an embodiment of the present disclosure, the point cloud data is data including information on the three-dimensional coordinates (u1, v1, w1) of each reflection point expressed in a sensor coordinate system fixed to the LiDAR sensor 220, or information indicating the distance to each reflection point and the direction of the reflection point. However, each reflection point may be expressed, for example, in a two-dimensional coordinate system. The point cloud data of the left side portion LU1-LU5 of the ground part of the tree row and the right side portion RU1-RU5 of the ground part can be expressed in a three-dimensional coordinate system (X, Y, Z), which is the world coordinate system. In other words, the coordinates of each reflection point in the three-dimensional coordinates (u1, v1, w1) can be converted to coordinates in the three-dimensional coordinate system (X, Y, Z). For example, the point cloud data of the left side portion LU1 of the ground part of the tree row T1 can be expressed in a three-dimensional coordinate system (X, Y, Z) based on the position information of the work vehicle 100 in the world coordinate system indicated by the positioning data output from the GNSS unit 110 and the coordinate information of each reflection point indicated by the point cloud data output from the LiDAR sensor 220. In other words, the point cloud data of the left side portion LU1 of the ground part of the tree row T1 can be expressed in a three-dimensional coordinate system (X, Y, Z) based on the coordinate information of an arbitrary point through which the work vehicle 100 passes while the LiDAR sensor 220 is sensing the tree row T1 and the distance from the arbitrary point to the tree row T1 (strictly speaking, the distance from the LiDAR sensor 220 to the reflection point). Similarly, the point cloud data of the left side portions LU2 to LU5 of the ground part and the right side portions RU2 to RU5 of the ground part can also be expressed in a three-dimensional coordinate system (X, Y, Z). For example, the start point S or the end point G may be the origin of the three-dimensional coordinate system (X, Y, Z), or another position may be the origin.

[6. Association of point cloud data of left and right parts of ground]
Next, the information processing device 161 extracts or identifies point cloud data for the left and right lateral parts of the ground portion of the tree rows T1 to T5, which are divided by lateral unit, from a plurality of point cloud data for the left lateral parts LU1 to LU5 and the right lateral parts RU1 to RU5 of the ground portion of the tree rows T1 to T5. For example, the information processing device 161 refers to the point cloud data for the left lateral parts LU1 to LU5 of the ground portion and the point cloud data for the right lateral parts RU1 to RU5 of the ground portion in a three-dimensional coordinate system (X, Y, Z), and associates, from the plurality of point cloud data, point cloud data including the closest or similar data groups, or point cloud data including matching data groups. In other words, the information processing device 161 associates the identification information of the point cloud data that includes the closest or matching data group among the multiple identification information assigned to each of the point cloud data of the left side portion LU1 to LU5 of the ground portion and the multiple identification information assigned to each of the point cloud data of the right side portion RU1 to RU5 of the ground portion.

The information processing device 161 in the embodiment of the present disclosure associates point cloud data including the closest or most similar data groups from among the multiple point cloud data described above by matching the point cloud data. The matching can be performed using any matching algorithm, such as NDT (Normal Distribution Transform) or ICP (Iterative Closest Point).

Of the two associated point cloud data, the information processing device 161 determines that the point cloud data located in the relatively left space in the three-dimensional coordinate system (X, Y, Z) is the point cloud data related to the left side of the ground, and determines that the point cloud data located in the relatively right space in the three-dimensional coordinate system is the point cloud data related to the right side of the ground.

Figure 12 is a diagram showing the result of associating point cloud data for the left and right sides of the ground that make up the same tree row from multiple point cloud data divided by side units. Figure 12 shows the result of associating point cloud data for the left and right sides of the ground that make up each of the tree rows T1 to T5 shown in Figure 11. In the example shown in Figure 12, identification information 0001U to 0010U is assigned to multiple (10) point cloud data acquired by side units when the work vehicle 100 moves from the start point S to the end point G shown in Figure 11. The point cloud data assigned with identification information 0001U is the first acquired data, and the point cloud data assigned with identification information 0010U is the most recent data acquired last.

The information processing device 161 associates, by matching, the point cloud data of the left side portion of the ground portion that constitutes the tree row T1 with the point cloud data of the right side portion from the point cloud data assigned with the identification information 0001U to 0010U. In the example shown in Figure 12, the information processing device 161 determines that the point cloud data assigned with the identification information 0001U and the point cloud data assigned with the identification information 0002U are the most similar data groups. In other words, the information processing device 161 associates the identification information 0001U with the identification information 0002U. Then, the information processing device 161 refers to the coordinates of the point cloud data assigned with the two associated identification information 0001U and 0002U, and determines that the point cloud data assigned with the identification information 0001U and located in the relatively left space in the three-dimensional coordinate system (X, Y, Z) is the point cloud data associated with the left side portion LU1 of the ground, and determines that the point cloud data assigned with the identification information 0002U and located in the relatively right space in the three-dimensional coordinate system is the point cloud data associated with the right side portion RU1 of the ground.

The information processing device 161, like the tree row T1, associates the point cloud data of the left side portion of the ground portion constituting each of the tree rows T2 to T5 with the point cloud data of the right side portion by matching. The information processing device 161 determines the point cloud data assigned with identification information 0003U, 0005U, 0007U, and 0009U, which is located in a relatively left space in the three-dimensional coordinate system (X, Y, Z), as the point cloud data associated with the left side portion LU2 to LU5 of the ground portion, respectively, and determines the point cloud data assigned with identification information 0004U, 0006U, 0008U, and 0010U, which is located in a relatively right space in the three-dimensional coordinate system, as the point cloud data associated with the right side portion RU2 to RU5 of the ground portion, respectively.

[7. Generation of Ground Information]
The information processing device 161 can estimate ground information from point cloud data of the left side portion and the right side portion of the ground portion constituting the same extracted tree row. For example, the information processing device 161 extracts one or more point cloud data including the closest or identical data group from point cloud data determined to be point cloud data of the left side portion LU1 of the ground portion of the tree row T1 and assigned with identification information 0001U, and point cloud data determined to be point cloud data of the right side portion RU1 of the ground portion of the same tree row T1 as the tree row and assigned with identification information 0002U. The extracted one or more point cloud data are called "common ground point cloud data".

The information processing device 161 can estimate ground information related to the above-ground parts of the tree row by expressing the point cloud data to which the identification information 0001U and 0002U have been assigned in the same three-dimensional coordinate system (X, Y, Z) using the common above-ground part point cloud data as a reference. The information processing device 161 then integrates the two point cloud data to which the identification information 0001U and 0002U have been assigned, and assigns the identification information U0001 of the above-ground part to the integrated point cloud data (see FIG. 12). The point cloud data to which the identification information U0001 has been assigned can indicate ground information related to the above-ground parts of the tree row T1, such as shape, size, or coordinate position. Similarly, the information processing device 161 can estimate ground information about the above-ground parts of tree rows T2 to T5 from the point cloud data of the left side section assigned with identification information 0003U, 0005U, 0007U, and 0009U, and the point cloud data of the right side section assigned with identification information 0004U, 0006U, 0008U, and 0010U, respectively.

The information processing device 161 may, for example, refer to the ground information for each tree row and the distance from the origin in a three-dimensional coordinate system of the reflection points included in the point cloud data indicating each piece of ground information, and assign identification information U0001 to U0005 to the ground information, thereby identifying the ground information divided for each tree row. Each of the identification information U0001 to U0005 and the ground information to which each identification information has been assigned may be stored in a storage device.

[8. Obtaining Underground Information]
Next, an example of a method for acquiring underground information of growing grape vines by running a work vehicle in a vineyard will be described with reference to Fig. 13. As shown in Fig. 11, while the work vehicle 100 is running manually or automatically between the rows of trees in the vineyard, repeating straight ahead and turning between adjacent rows of trees, the underground radar device 140 acquires underground information relating to the underground portions of the rows of trees.

In the example shown in FIG. 13, when the work vehicle 100 travels along one of a number of tree rows, the left side of the underground portion of the tree row facing the side of the work vehicle 100 (the portion within the dotted rectangle LD in FIG. 13) is explored, but the underground portion of the right side of the tree row opposite the left side (the portion within the dotted rectangle RD in FIG. 13) is not explored.

In the example of FIG. 13, while the work vehicle 100 is traveling along the left side of a certain tree row, the underground radar device 140 repeatedly acquires sensing data by transmitting and receiving electromagnetic waves, and further acquires underground information of the left side of the tree row based on the sensing data. Next, while the work vehicle 100 turns at the end point of the tree row and travels along the right side of the tree row, the underground radar device 140 repeatedly acquires sensing data by transmitting and receiving electromagnetic waves, and further acquires underground information of the right side of the tree row based on the sensing data. While the work vehicle 100 is traveling between adjacent tree rows, the underground radar device 140 can simultaneously acquire underground information of the right side of one of the adjacent tree rows and underground information of the left side of the other adjacent tree row. In other words, the underground radar device 140 can acquire underground information regarding the underground parts of the tree rows located within the second measurement range on both sides of the work vehicle 100.

In this way, the underground radar device 140 obtains underground information for each tree row regarding the underground portions of the entire tree row, and obtains underground information for the entire underground portions of multiple tree rows in the vineyard while repeatedly traveling straight and turning. While the work vehicle 100 travels along the tree rows, the underground radar device 140 can continuously obtain underground information for multiple tree rows along the travel path P.

In the above example, a method for acquiring underground information about the entire circumference of the underground portion of a grapevine was described, but this method is merely one example, and like above-ground information, it is not essential to acquire underground information about the entire circumference of the underground portion of a tree row. For example, underground information about the underground portion of at least one tree included in a tree row can be acquired on a tree-by-tree basis.

[9. Acquisition of underground sensing data and coordinate system transformation]
Figure 14 is a schematic diagram of a vineyard with five tree rows. Figure 14 shows the underground portions of the five tree rows. The underground portions of the five tree rows T1 to T5 shown in Figure 14 correspond to the above-ground portions of the five tree rows T1 to T5 shown in Figure 11. The left portions of the five tree rows T1 to T5 in Figure 14 will be referred to as "left lateral portions" LD1 to LD5, respectively, and the right portions of the five tree rows 20 will be referred to as "right lateral portions" RD1 to RD5, respectively.

The work vehicle 100 proceeds from the start point S, and while traveling outside the tree row T1, acquires sensing data (hereinafter referred to as "underground sensing data") including information on the spatial distribution of the left side LD1 of the underground part of the tree row T1 using the underground radar device 140. The work vehicle 100 turns at the end point of the tree row T1, then enters between the adjacent tree rows T1 and T2, and acquires underground sensing data of the right side RD1 of the underground part of the tree row T1 and the left side LD2 of the underground part of the tree row T2 using the underground radar device 140. The work vehicle 100 proceeds to the end point G while repeatedly traveling and turning along the tree rows, and acquires underground sensing data of the left side LD3-LD5 of the underground parts of the tree rows T3-T5 and the right side RD2-RD4 of the underground parts of the tree rows T2-T4 using the underground radar device 140. Finally, the work vehicle 100, while traveling outside the tree row T5, acquires underground sensing data of the right side portion RD5 of the underground portion of the tree row T5 using the underground radar device 140. In this way, the underground radar device 140 acquires underground sensing data of the underground portions of the tree rows T1 to T5 throughout the vineyard.

The information processing device 161 can treat the underground sensing data of the left side portions LD1-LD5 and the right side portions RD1-RD5 of the underground portion as lateral units in the underground portion of each tree row, and store information related to the sensing data in a storage device in lateral units. The GNSS unit 110 equipped in the work vehicle 100 has an IMU 115 and detects the difference in operation between the work vehicle 100 traveling along the tree row and turning. This detection makes it possible to identify the underground sensing data continuously acquired by the underground radar device 140 while the work vehicle 100 travels from the start point S to the end point G in lateral units.

The underground sensing data of each of the left side parts LD1 to LD5 of the underground part of the tree row and the underground sensing data of each of the right side parts RD1 to RD5 of the underground part can be distinguished using the positioning data output from the GNSS unit 110. For example, the work vehicle 100 travels along the left side part LD1 of the underground part of the tree row T1 from the starting point S, and turns after reaching the end of the left side part LD1. The information processing device 161 can detect that the direction of travel of the work vehicle 100 changes due to the turning of the work vehicle 100 based on the positioning data output from the GNSS unit 110. According to this detection result, the information processing device 161 can recognize that the work vehicle 100 has reached the other end of the left side part LD1 of the underground part of the tree row T1 from one end (starting point S) and the other end of the left side part LD1, and has reached the end of the right side part RD1 from the other end of the left side part LD1. In this way, by utilizing the change in the traveling direction of the work vehicle 100, the information processing device 161 can recognize that sensing has shifted from the left side LD1 of the tree row T1 to sensing the right side RD1, and can separate the respective underground sensing data. Similarly, for the tree rows T2 to T5, the information processing device 161 can separate the underground sensing data of the left side LD2 to LD5 of the underground part from the underground sensing data of the right side RD2 to RD5 of the underground part each time the work vehicle 100 turns.

In an embodiment of the present disclosure, the information processing device 161 assigns identification information to the underground sensing data for each of the divided left side sections LD1 to LD5 and right side sections RD1 to RD5 of the underground section, in the same way as for the aboveground section, to identify the divided underground sensing data by side.

In an embodiment of the present disclosure, the underground sensing data is a data group related to the distribution of the three-dimensional space of the underground part in the ground, and may include information indicating the distance to each data point of the data group and the direction of the data point. Each data of the data group may be data including information on three-dimensional coordinates (u2, v2, w2) which are a sensor coordinate system fixed to the underground radar device 140. Therefore, the underground sensing data of the left side part LD1 to LD5 of the underground part of the tree row and the right side part RD1 to RD5 of the underground part can be expressed in a three-dimensional coordinate system (X, Y, Z) which is a world coordinate system, similar to the point cloud data. For example, the underground sensing data of the left side part LD1 of the underground part of the tree row T1 can be expressed in a three-dimensional coordinate system (X, Y, Z) based on the position information of the work vehicle 100 in the world coordinate system indicated by the positioning data output from the GNSS unit 110 and the coordinate information of each data point included in the underground sensing data output from the underground radar device 140. In other words, the underground sensing data of the left side LD1 of the underground part of the tree row T1 can be represented in a three-dimensional coordinate system (X, Y, Z) based on the coordinate information of any point that the work vehicle 100 passes while the underground radar device 140 is sensing the tree row T1 and the distance from the any point to the tree row T1 (strictly speaking, the distance from the underground radar device 140 to each data point). Similarly, the underground sensing data of the left side LD2 to LD5 of the underground part and the right side RD2 to RD5 of the underground part can also be represented in a three-dimensional coordinate system (X, Y, Z). Note that the origin of this three-dimensional coordinate system (X, Y, Z) coincides with the origin of the three-dimensional coordinate system in which the underground sensing data group indicating the underground information exists.

[10. Association of underground sensing data from the left and right sides of the underground section]
Next, the information processing device 161 extracts or identifies data groups of the left and right underground sections constituting the same tree row from a plurality of underground sensing data of the left and right underground sections LD1-LD5 and RD1-RD5 of the tree rows T1-T5, which are divided by side. For example, the information processing device 161 refers to the underground sensing data group of the left underground section LD1-LD5 and the underground sensing data group of the right underground section RD1-RD5 in a three-dimensional coordinate system (X, Y, Z), and associates data groups with some data that are closest to each other or similar to each other, or data groups with matching data, from among the plurality of data groups. In other words, the information processing device 161 associates the identification information of the data group that is closest to or matches a part of the underground sensing data group among the multiple identification information assigned to each underground sensing data group of the left side portion LD1 to LD5 of the underground portion and the multiple identification information assigned to each underground sensing data group of the right side portion RD1 to RD5 of the underground portion.

The information processing device 161 determines, of the two associated underground sensing data groups, the data group located relatively to the left in the three-dimensional coordinate system as the data group relating to the left side of the underground area, and determines the data group located relatively to the right in the three-dimensional coordinate system as the data group relating to the right side of the underground area.

Figure 15 is a diagram showing the result of associating data groups of the left and right sides of the underground parts that make up the same tree row from multiple underground sensing data groups divided by side. Figure 15 shows the result of associating underground sensing data groups of the left and right sides of the underground parts that make up each of the tree rows T1 to T5 shown in Figure 14. In the example shown in Figure 15, identification information 0001D to 0010D are assigned to multiple (10) underground sensing data groups acquired by side when the work vehicle 100 moves from the start point S to the end point G shown in Figure 14. The underground sensing data group assigned with identification information 0001D is the first data acquired, and the underground sensing data group assigned with identification information 0010D is the latest data acquired last.

The information processing device 161 associates, by matching, a data group for the left side portion of the underground portion that constitutes the tree row T1 with a data group for the right side portion from among the underground sensing data groups that have been assigned identification information 0001D to 0010D. In the example shown in FIG. 15, the information processing device 161 determines that the underground sensing data group assigned identification information 0001D and the underground sensing data group assigned identification information 0002D are the most similar data groups. In other words, the information processing device 161 associates the identification information 0001D with the identification information 0002D. Then, the information processing device 161 refers to the coordinates of the underground sensing data groups assigned with the two associated identification information 0001D and 0002D, and determines that the data group assigned with the identification information 0001D and located in the relatively left space in the three-dimensional coordinate system (X, Y, Z) is the underground sensing data group associated with the left side portion LD1 of the underground section, and determines that the data group assigned with the identification information 0002D and located in the relatively right space in the three-dimensional coordinate system is the underground sensing data group associated with the right side portion RD1 of the underground section.

The information processing device 161, like the tree row T1, associates the data groups of the left and right sides of the underground portions constituting each of the tree rows T2 to T5 by matching. The information processing device 161 determines the point cloud data assigned with identification information 0003D, 0005D, 0007D, and 0009D, which are located in the relatively left space in the three-dimensional coordinate system (X, Y, Z), as underground sensing data groups related to the left side portions LD2 to LUD of the underground portions, respectively, and determines the data groups assigned with identification information 0004D, 0006D, 0008D, and 0010D, which are located in the relatively right space in the three-dimensional coordinate system, as underground sensing data groups related to the right side portions RD2 to RD5 of the underground portions, respectively.

[11. Generation of underground information]
The information processing device 161 may estimate underground information from an underground sensing data group of the left side portion and an underground sensing data group of the right side portion constituting the extracted same tree row. For example, the information processing device 161 extracts one or more data groups whose part of the data is closest to or matches from an underground sensing data group determined to be a data group of the left side portion LD1 of the underground part of the tree row T1 and to which identification information 0001D is assigned, and an underground sensing data group determined to be a data group of the right side portion RD1 of the underground part of the same tree row T1 as the tree row and to which identification information 0002D is assigned. The extracted one or more data groups are called a "common underground data group".

The information processing device 161 can estimate underground information about the underground part of the tree row by expressing the data groups each assigned with the identification information 0001D and 0002D in the same three-dimensional coordinate system (X, Y, Z) using the common underground data group as a reference. The information processing device 161 then integrates the two underground sensing data groups assigned with the identification information 0001D and 0002D, and assigns underground identification information D0001 to the integrated underground sensing data group (see FIG. 15). The underground sensing data group assigned with the identification information D0001 can indicate underground information about the underground part of the tree row T1, such as the shape, size, or coordinate position. Similarly, the information processing device 161 can estimate underground information about the underground portions of tree rows T2 to T5 from a group of data for the left side section assigned with identification information 0003D, 0005D, 0007D, and 0009D, and a group of data for the right side section assigned with identification information 0004D, 0006D, 0008D, and 0010D, respectively.

The information processing device 161 may, for example, refer to the underground information for each tree row and the distance from the origin in a three-dimensional coordinate system of each data point included in the underground sensing data group indicating each piece of underground information, and assign identification information D0001 to D0005 to the underground information, thereby identifying the underground information divided for each tree row. Each of the identification information D0001 to D0005 and the underground information to which each identification information has been assigned may be stored in the storage device.

[12. Integrating aboveground and belowground information to estimate crop information]
The information processing device 161 may estimate crop information by referring to a plurality of above-ground information and a plurality of underground information. In an embodiment of the present disclosure, the information processing device 161 estimates the state of the crop by referring to a plurality of above-ground information indicated by point cloud data to which a plurality of identification information U0001 to U0005 has been assigned, and underground information indicated by underground sensing data to which identification information D0001 to D0005 has been assigned. In the following description, for example, above-ground information indicated by point cloud data to which identification information U0001 has been assigned is referred to as above-ground information to which identification information U0001 has been assigned. Underground information indicated by underground sensing data to which identification information D0001 has been assigned is referred to as underground information to which identification information D0001 has been assigned.

The information processing device 161 extracts specific underground information that is common to specific ground information from among the multiple pieces of ground information. In other words, the information processing device 161 associates point cloud data indicating one of the multiple pieces of ground information with underground sensing data indicating one of the multiple pieces of underground information.

In the above-mentioned example, among the multiple pieces of ground information each having the identification information U0001 to U0005 assigned thereto, the ground information assigned with the identification information U0002 is assumed to be the specified ground information. In this case, the information processing device 161 extracts underground information common to the ground information assigned with the identification information U0002 from the multiple pieces of underground information each having the identification information D0001 to D0006 assigned thereto. Specifically, the information processing device 161 searches for an underground sensing data group that is closest to or matches the point cloud data indicating the ground information assigned with the identification information U0002 from among the underground sensing data groups indicating the underground information each having the identification information D0001 to D0005 assigned thereto.

When the information processing device 161 determines that it is unable to extract specific underground information common to specific ground information among multiple pieces of ground information, it may display a message on the display device to notify the user that the specific underground information cannot be extracted. An example of a display device is the screen of the operation terminal 200 or the terminal device 400 described above. By displaying such a message, it is possible to, for example, prompt the user of the system to reacquire sensing data.

When extracting common combinations from multiple aboveground information and multiple underground information, i.e., when extracting specific underground information common to specific aboveground information, it is preferable that the information processing device 161 makes a judgment by comparing, in a three-dimensional coordinate system (X, Y, Z), a data group located at or near the boundary between aboveground and underground parts, which is included in the point cloud data representing the aboveground information, with a data group located at or near the boundary between aboveground and underground parts, which is included in the underground sensing data representing the underground information.

Figure 16 is a plan view showing an enlarged cross section of a trunk 710, which is part of the underground portion 700 of a tree. The cross-sectional shape or size of a tree trunk varies from individual tree to individual tree. The trunk portion of a tree includes a portion that is common to both above-ground and underground portions at the boundary between them. Therefore, in an embodiment of the present disclosure, the information processing device 161 focuses on the cross-sectional shape or size of the above-ground trunk portion and the underground trunk portion, and searches among multiple underground sensing data groups for an underground sensing data group that includes a trunk data group that is closest to or matches a trunk data group included in one of the multiple point cloud data.

Figure 17 shows a table listing identification information for the entire crop, obtained by integrating the table listing identification information for the above-ground parts shown in Figure 12 with the table listing identification information for the underground parts shown in Figure 15. The information processing device 161 in the embodiment of the present disclosure references the table shown in Figure 12 and the table shown in Figure 15, associates above-ground and below-ground information for the same crop, and creates a table listing identification information for the entire crop.

The information processing device 161 can associate the aboveground information U0001 and the underground information D0001, and assign crop identification information UD001 for identifying the entire crop to the tree row T1. In other words, the information processing device 161 can integrate the aboveground point cloud data and the underground sensing data, which have different attributes, in a common coordinate system. With respect to the trunk of a tree, a partial data group of the point cloud data to which the aboveground information U0001 has been assigned and a partial data group of the underground sensing data to which the underground information D0001 has been assigned may be closest, similar, or identical in a three-dimensional coordinate system (X, Y, Z). In this case, the information processing device 161 can determine that the predetermined underground information common to the aboveground information U0001 is the underground information D0001. Similarly, the information processing device 161 can associate the aboveground information U0002 to U0005 with the belowground information D0002 to D0005, respectively, and assign the crop identification information UD002 to UD005 to the tree rows T2 to T5, respectively. The information processing device 161 can determine that predetermined belowground information common to the aboveground information U0002 to U0005 is the belowground information D0002 to D0005, respectively.

The information processing device 161 links the aboveground information U0002 and the underground information D0002 extracted as a combination common to, for example, tree row T2. Specifically, the information processing device 161 assigns new crop identification information UD002 by linking the aboveground information U0002 and the underground information D0002 as multiple pieces of information related to tree row T2. This distinguishes tree row T2 from other tree rows. The information processing device 161 can determine which of tree rows T1 to T5 a combination of aboveground and underground information common to a certain tree row corresponds to, for example, based on the distance from a representative point of the point cloud data or underground sensing data (for example, the position of the center of gravity of the data group) to the origin in a three-dimensional coordinate system (X, Y, Z). Similarly, the information processing device 161 links the aboveground information U0001, U0003 to U0005 extracted as a combination common to the tree rows T1, T3 to T5 with the belowground information D0001, D0003 to D0005, and assigns the crop identification information UD001, UD003 to UD005 to the tree rows T1, T3 to T5. The information processing device 161 can associate each of the crop identification information UD001 to UD005 of the tree rows T1 to T5 with the aboveground and belowground information included in each identification information, and store it in the storage device.

The information processing device 161 in an embodiment of the present disclosure may estimate the shape or size of the entire tree row, or one or more trees included in the tree row, from the associated above-ground information and below-ground information. The associated above-ground information and below-ground information are sensor data obtained based on distances detected by a first sensor and a second sensor, respectively. The information processing device 161 estimates the shape or size of the trees, which are crops, from the sensor data. For example, the information processing device 161 may estimate the size or shape of the tree row T2 based on above-ground information U0002 and below-ground information D0002 included in crop identification information UD002, out of the combinations of above-ground information and below-ground information included in each of the crop identification information UD001 to UD005. Specifically, the information processing device 161 can access the storage device and estimate the size or shape of the tree row T2 based on the ground information U0002 and underground information D0002 associated with the tree row T2, i.e., the point cloud data representing the ground information U0002 and the coordinate information in a three-dimensional coordinate system (X, Y, Z) of the underground sensing data representing the underground information D0002.

[13. Estimation of tree shape and size]
In one aspect, the information processing device 161 estimates the shape, size, or growth condition of the crop based on ground information acquired by a first sensor (e.g., a LiDAR sensor) during autonomous driving and underground information acquired by a second sensor (e.g., an underground radar device) during autonomous driving.

With reference to Figures 18 and 19, we will explain how to estimate the size and shape of a tree in a tree row.

Figure 18 is a diagram showing a schematic shape of a portion of tree row T2. Figure 18 shows a portion of four trees out of tree row T2 (see Figure 11) which includes ten trees. Figure 19 is a diagram showing a schematic appearance of one tree T2a included in tree row T2. When a given tree row includes multiple trees, the information processing device 161 can extract one tree out of multiple trees that have been manually or automatically specified. For example, the tree is specified by the user operating a terminal device.

In an embodiment of the present disclosure, the information processing device 161 can grasp the shape of the tree row T2 including the above-ground and underground parts as shown in FIG. 18 by expanding the point cloud data indicating a specific tree row T2 into a virtual field represented by a three-dimensional coordinate system (X, Y, Z). The information processing device 161 can grasp the shape of at least one tree from among the multiple trees included in the tree row T2 on a tree-by-tree basis based on the obtained shape of the tree row T2 and the coordinates of the data group representing the shape of the tree row T2. For example, when a tree T2a is specified by the user, as shown in FIG. 19, the information processing device 161 extracts the shape of the specified tree row T2 including the above-ground and underground parts from the shape of the tree row T2. The specified tree may be a tree located at the end of the tree row, or may be a tree located other than the end of the tree row. The information processing device 161 estimates, for example, the size and shape of the actual tree T2a from the shape representing the tree T2a.

According to an embodiment of the present disclosure, it is possible to integrate point cloud data with different attributes and underground sensing data, and estimate the size and shape of tree rows or trees on a tree-row or tree-by-tree basis.

<Estimating the size of the above-ground part of a tree>
The information processing device 161 can, for example, recognize the trunk of the tree T2a from the shape of the tree T2a and obtain the size of the trunk (height H1 and width W1 shown in FIG. 19). Alternatively, the information processing device 161 can recognize the crown of the tree T2a from the shape of the tree T2a and obtain the size of the crown (height H2 and width W2 shown in FIG. 19). For example, the height H1 of the trunk of the tree T2a can be defined by the distance from the reflection point (e.g., the part corresponding to the ground) located at the lowest position in the Y direction in the point cloud data of the ground part of the tree T2a in a three-dimensional coordinate system (X, Y, Z) to the reflection point of the branch part located at the lowest position in the Y direction in the point cloud data of the crown. The width W1 of the trunk of the tree T2a can be defined by the horizontal distance of the parallel cross sections of the trunk on the XY plane shown in FIG. The width W1 of the trunk can be determined by extracting a plurality of horizontal (X direction) widths or distances in the height direction (Y direction) of the trunk in the horizontal direction and calculating the average of the extracted widths. Alternatively, the stem width W1 may be the horizontal width of the stem at a given height from the ground.

The crown shown in FIG. 19 corresponds to the layer of branches and leaves formed above the trunk. The height H2 of the crown of tree T2a can be determined, for example, by the distance from the reflection point located highest in the Y direction in the point cloud data of the trunk in a three-dimensional coordinate system (X, Y, Z) to the reflection point of the branch or leaf portion located highest in the Y direction in the point cloud data of the crown. The width W2 of the crown can be determined by the maximum distance in the horizontal direction (X direction) of the branch or leaf portion included in the crown. Furthermore, the information processing device 161 may determine the height H4 of tree T2a by adding the height H1 of the trunk and the height H2 of the crown, or may determine it as the distance from the ground to the highest point of the crown.

Figure 19 illustrates the heights H1, H2, H4, and widths W1 and W2 of a cross section of tree T2a parallel to the XY plane, but the information processing device 161 is capable of determining the height and width of the trunk and/or crown of tree T2a in a cross section parallel to the YZ plane or XZ plane.

<Estimating the size of the underground part of a tree>
The information processing device 161 can recognize the root portion (or root system) of the tree T2a from the shape of the tree T2a, as well as the above-ground part of the tree T2a, and can obtain the size of the root portion (depth H3 and width W3 shown in FIG. 19). For example, the root depth H3 of the tree T2a can be defined by the distance in the Y direction from a data point located on the ground to a data point of the root portion located lowest in the Y direction among the underground sensing data of the underground part in a three-dimensional coordinate system (X, Y, Z). The root width W3 can be defined by the maximum distance of the root in the horizontal direction (X direction) underground.

Considering the characteristics of a vineyard, the ground directly below or near the tree T2a may be inclined or uneven. For this reason, the position of the virtual ground that serves as the reference for height or depth may be estimated by finding the average of the minimum Y coordinates of some of the data included in the ground data group among the multiple point cloud data of the above-ground portion of the tree T2a.

The estimation of the size or shape of each of the trunk, crown, and root parts of a tree may be performed on a tree-by-tree or tree-row basis. For example, the size of each of the trunk, crown, and root parts of a tree T2a may be determined based on features obtained from the shape of the tree T2a obtained by expanding point cloud data to which identification information U0002 related to crop identification information UD002 is assigned into a field indicating a farm field represented by a three-dimensional coordinate system (X, Y, Z), or may be determined by pattern matching of the shape of the tree T2a. Note that the method of estimating the size or shape of each of the trunk, crown, and root parts of a tree is not limited to the method exemplified in this specification. Furthermore, the information processing device 161 can estimate the growth condition of the tree T2a based on features determined from the shape of the tree T2a obtained by expanding the point cloud data to which the identification information U0002 related to the crop identification information UD002 has been assigned and/or the group of underground sensing data to which the identification information D0002 has been assigned, into a field representing the farm field represented in a three-dimensional coordinate system (X, Y, Z).

In this way, it is possible to detect the above-ground and underground parts of crops in a field and estimate the condition of the crops based on the information obtained from each of the above-ground and underground parts. In particular, even in cases where it is difficult to understand the condition of the crops from above-ground information alone, it is possible to comprehensively understand the condition of the crops by adding underground information.

[14. Generation of farm field map]
The information processing device 161 can create a crop shape based on above-ground information and underground information, and can create a farm field map in which a travel route (or a target route) is set by expanding the created crop shape into a field showing the farm field. The created crop shape is not limited to a reproduction of an actual crop shape, and may be an imitation of the crop shape, for example, a rough shape of the crop obtained by deforming the crop shape.

The information processing device 161 can create a field map showing the shape of each of the tree rows T1 to T5 by expanding a plurality of point cloud data of the aboveground parts associated with the crop identification information UD001 to UD005, respectively, onto a virtual field represented in a three-dimensional coordinate system (X, Y, Z). An example of a field map is shown in FIG. 20. The illustrated field map reflects the coordinate information, relative distance, orientation, etc. of the point cloud data of the aboveground parts of the tree rows T1 to T5 in the three-dimensional coordinate system (X, Y, Z). Furthermore, the field map can reflect the coordinate information, relative distance, orientation, etc. of the underground sensing data group of the underground parts of the tree rows T1 to T5.

The information processing device 161 can, for example, determine the latitude and longitude of the crop from the position of the work vehicle 100 (i.e., the vehicle position) expressed in latitude and longitude and the distance from the work vehicle 100 to a specified crop obtained by at least the first sensor, and create a field map with position information that associates the shape of the crop created based on the crop estimated while the work vehicle 100 is traveling with the latitude and longitude of the crop. For example, the information processing device 161 creates a field map with position information that includes the shapes of the tree rows T1 to T5 and the latitude and longitude position information of the tree rows T1 to T5. The information processing device 161 converts the coordinates of each data point of the point cloud data of the tree rows T1 to T5 in three-dimensional coordinates (u2, v2, w2) into latitude and longitude position information of the tree rows T1 to T5 in a three-dimensional coordinate system (X, Y, Z) based on the distance from the work vehicle 100 to each of the tree rows T1 to T5 and the vehicle body position information detected by the GNSS unit 110. The information processing device 161 can create a three-dimensional field map with position information by arranging the three-dimensional shapes of the tree rows T1 to T5 at the positions indicated by the converted position information. The information processing device 161 may also create a two-dimensional field map with position information by arranging the two-dimensional shapes of the tree rows T1 to T5.

The above-mentioned field map with location information can be used for the automatic driving of the work vehicle 100. For example, the information processing device 161 or the management device 600 may generate a planned driving route in advance on the created field map with location information.

The work vehicle 100 may automatically travel within the field using the created field map with position information. Specifically, the route generation ECU 185 shown in FIG. 4 determines a planned travel route for the work vehicle 100, which means a predetermined travel route, based on the field map with position information. The planned travel route may be automatically determined by the ECU 185 based on the field map with position information, or may be set by a user operation.

Figure 21 is a diagram showing a schematic of the distribution of trunks in a row of trees, generated based on a field map with location information. In the two-dimensional distribution of trunks in the row of trees shown in Figure 21, the trunks of each tree are represented by circles of a certain size. A group of data showing such a two-dimensional distribution of trunks is called "stem distribution data". The trunk distribution data may include, for example, latitude and longitude position information of the center of each trunk. In addition to the latitude and longitude position information, the trunk distribution data may include information on the width W1 of the trunk shown in Figure 19. A user may use an operation terminal to input information showing the arrangement of multiple rows of trees, such as the distance between the trunks of adjacent trees or the position information of the trunks. The input information is recorded in a storage device or other storage medium.

21 also shows a planned travel route P determined based on the trunk distribution data. For example, the route generation ECU 185 may determine a curve connecting the trunks of multiple trees included in each tree row. The ECU 185 may determine, for example, a curve or broken line passing through the center of the curve connecting the trunks of multiple trees included in each of two adjacent tree rows (e.g., tree rows T1 and T2) as the planned travel route. The control device 180 (see FIG. 4) may control steering so that the position information output by the GNSS unit matches the planned travel route. The control device 180 does not need to strictly control the planned travel route generated in the center of the trunk curves of the two adjacent tree rows so that the center of the body of the work vehicle 100 passes through it. The planned travel route may be shifted from the center of the trunk curves of the two adjacent tree rows.

For simplicity, a method for generating a planned driving route based on two-dimensional trunk distribution data has been shown, but the ECU 185 may use the created farm field map with position information to determine a planned driving route based on the coordinates in a three-dimensional coordinate system (X, Y, Z) of each data point of the point cloud data of the trunks of multiple trees included in a tree row.

The work vehicle 100 may sense any row of trees in the field while autonomously traveling along a planned route determined based on the stem distribution data or the field map. While autonomously traveling, the work vehicle 100 can estimate any row of trees by having the LiDAR sensor 220 acquire ground information and the underground radar device 140 acquire underground information.

According to an embodiment of the present disclosure, it is possible to create a field map by expanding the acquired tree information into a virtual field. This field map can contribute to the visualization of information and can also be used to generate planned driving routes for the automatic driving of work vehicles.

FIG. 22 is a flowchart showing an outline of the process flow of an estimation method for estimating crop information. The estimation method in the embodiment of the present disclosure includes step S10 of acquiring point cloud data of the aboveground part and underground sensing data of the underground part for each side of the tree row, step S20 of assigning identification information to the point cloud data and underground sensing data, step S30 of determining whether the work vehicle has completed traveling from the start point to the end point, step S40 of applying coordinate transformation processing to the point cloud data and underground sensing data, step S50 of associating point cloud data of the left and right sides of the aboveground part and generating aboveground information, step S60 of associating underground sensing data of the left and right sides of the underground part and generating underground information, and step S70 of integrating the aboveground information and underground information to estimate crop information. Details of the processing in each step are as described above.

The work vehicle 100 can estimate crop information for one or more tree rows in a field on a tree row basis by following the flow shown in FIG. 22. Alternatively, the work vehicle 100 can estimate crop information for at least one or more trees of interest among multiple trees included in one or more tree rows in a field. When estimating crop information for a single tree, for example, the operator of the work vehicle may determine whether acquisition of sensing data for the left and right sides of the tree has been completed.

In the processing of step S50 or step S60, if the information processing device 161 cannot associate the data for the left and right sides, for example, if there is no data for the right side related to the left side, an error message indicating that either the data for the left or right side could not be detected may be displayed, for example, on the screen of the operation terminal 200 of the work vehicle 100.

[15. Steering control during automatic driving]
An example of the operation of steering control during automatic driving will be described in more detail below.

Figure 23 is a flowchart showing an example of the operation of steering control during automatic driving executed by the control device 180. The control device 180 performs automatic steering by executing the operations of steps S121 to S125 shown in Figure 23 while the work vehicle 100 is traveling. The speed is maintained at a preset speed, for example. The control device 180 acquires data indicating the position of the work vehicle 100 estimated by the information processing device 161 while the work vehicle 100 is traveling (step S121). Next, the control device 180 calculates the position deviation between the position of the work vehicle 100 and the target route (step S122). The position deviation represents the distance between the position of the work vehicle 100 at that time and the target route. The control device 180 determines whether the calculated position deviation exceeds a preset threshold value (step S123). If the position deviation exceeds the threshold value, the control device 180 changes the steering angle by changing the control parameters of the steering device included in the drive device 240 so that the position deviation is reduced. If the position deviation does not exceed the threshold in step S123, the operation of step S124 is omitted. In the following step S125, the control device 180 determines whether or not a command to end the operation has been received. The command to end the operation may be issued, for example, when a user remotely instructs the automatic driving to be stopped or when the work vehicle 100 reaches the destination. If the command to end the operation has not been issued, the process returns to step S121, and the same operation is performed based on the newly estimated position of the work vehicle 100. The control device 180 repeats the operations of steps S121 to S125 until a command to end the operation is issued. The above operations are executed by the ECUs 182 and 184 in the control device 180.

23, the control device 180 controls the drive device 240 based only on the position deviation between the position of the work vehicle 100 estimated by the information processing device 161 and the target route, but the control may also take into account the azimuth deviation. For example, when the azimuth deviation, which is the angular difference between the orientation of the work vehicle 100 estimated by the information processing device 161 or the orientation of the work vehicle 100 identified by the GNSS unit 110 having an IMU and the direction of the target route, exceeds a preset threshold, the control device 180 may change the control parameter (e.g., steering angle) of the steering device of the drive device 240 in accordance with the deviation.

Below, an example of steering control by the control device 180 will be described in more detail with reference to Figures 24A to 24D.

Figure 24A is a diagram showing an example of a work vehicle 100 traveling along a target route P. Figure 24B is a diagram showing an example of a work vehicle 100 in a position shifted to the right from the target route P. Figure 24C is a diagram showing an example of a work vehicle 100 in a position shifted to the left from the target route P. Figure 24D is a diagram showing an example of a work vehicle 100 facing in an inclined direction with respect to the target route P. In these figures, a reference line is set along the target route, and a reference point is set on the target route. For example, a pose indicating the position and orientation of the work vehicle 100 estimated by the information processing device 161 is expressed as r (x, y, θ). (x, y) are coordinates representing the position of the reference point of the work vehicle 100 in the XY coordinate system, which is a world coordinate system. In the examples shown in Figures 24A to 24D, the reference point of the work vehicle 100 is at the position where the GNSS antenna on the cabin is installed, but the position of this reference point is arbitrary. θ is an angle representing the estimated or measured orientation of the work vehicle 100. In the illustrated example, the target path P is parallel to the Y axis, but generally the target path P is not necessarily parallel to the Y axis.

As shown in FIG. 24A, if the position and orientation of the work vehicle 100 do not deviate from the target path P, the control device 180 maintains the steering angle and speed of the work vehicle 100 unchanged.

As shown in FIG. 24B, when the position of the work vehicle 100 shifts to the right from the target route P, the control device 180 changes the steering angle so that the travel direction of the work vehicle 100 tilts to the left and approaches the target route P. At this time, in addition to the steering angle, the speed may also be changed. The magnitude of the steering angle may be adjusted, for example, according to the magnitude of the position deviation Δx.

As shown in FIG. 24C, when the position of the work vehicle 100 shifts to the left from the target route P, the control device 180 changes the steering angle so that the travel direction of the work vehicle 100 tilts to the right and approaches the target route P. In this case, the speed may also be changed in addition to the steering angle. The amount of change in the steering angle may be adjusted, for example, according to the magnitude of the position deviation Δx.

As shown in FIG. 24D, when the position of the work vehicle 100 is not far from the target route P but the direction is different from the direction of the target route P, the control device 180 changes the steering angle so that the azimuth deviation Δθ is reduced. In this case, the speed may also be changed in addition to the steering angle. The magnitude of the steering angle may be adjusted, for example, according to the magnitudes of the position deviation Δx and the azimuth deviation Δθ. For example, the smaller the absolute value of the position deviation Δx, the larger the amount of change in the steering angle according to the azimuth deviation Δθ may be. When the absolute value of the position deviation Δx is large, the steering angle is changed significantly to return to the target route P, so the absolute value of the azimuth deviation Δθ inevitably becomes large. Conversely, when the absolute value of the position deviation Δx is small, it is necessary to bring the azimuth deviation Δθ closer to zero. For this reason, it is appropriate to relatively increase the weight of the azimuth deviation Δθ (i.e., the control gain) for determining the steering angle.

Control techniques such as PID control or MPC control (model predictive control) can be applied to the steering control and speed control of the work vehicle 100. By applying these control techniques, it is possible to smooth the control that brings the work vehicle 100 closer to the target path P.

This specification discloses the solutions described in the following items.

[Item 1]
A crop detection system for detecting a crop in a field, comprising:
A first sensor for detecting ground information, which is information about an above-ground part of the crop;
A second sensor for detecting underground information, which is information about the underground part of the crop;
an information processing device that estimates a crop including an above-ground part of the crop and an underground part of the crop based on the above-ground information and the underground information;
A crop detection system comprising:

[Item 2]
The information processing device includes:
By referring to the plurality of pieces of ground information detected by the first sensor and the plurality of pieces of underground information detected by the second sensor, predetermined underground information common to predetermined pieces of ground information is extracted from the plurality of pieces of ground information;
Associating the extracted predetermined underground information with the predetermined ground information;
2. The crop detection system of claim 1, wherein the crop is estimated from the predetermined above-ground information and the predetermined underground information associated with each other.

[Item 3]
The associated predetermined ground information and the associated predetermined underground information are sensing data obtained from distances detected by the first sensor and the second sensor, respectively;
3. The crop detection system according to claim 2, wherein the information processing device estimates a size of the crop from the sensing data.

[Item 4]
4. The crop detection system according to claim 2, wherein the information processing device creates a shape of the crop based on the above-ground information and the underground information, and creates a field map by expanding the created shape of the crop into a field representing the field.

[Item 5]
5. The crop detection system according to any one of claims 1 to 4, wherein the first sensor and the second sensor are mounted on a single agricultural machine or separately on multiple agricultural machines.

[Item 6]
the first sensor is a distance measuring sensor that detects a distance to an above-ground part of the crop present in a measurement range set around the agricultural machine as the ground information, or a camera that detects an image of the above-ground part of the crop present in the measurement range as the ground information,
4. The crop detection system according to any one of claims 1 to 3, wherein the second sensor is a distance measuring sensor that detects the distance to the underground part of the crop present in the measurement range as the underground information.

[Item 7]
7. The crop detection system of item 6, wherein the distance measurement sensor in the first sensor is a LiDAR sensor that detects the distance to the above-ground part of the crop present in the measurement range by emitting a laser pulse.

[Item 8]
8. The crop detection system according to claim 6, wherein the second sensor is a ground penetrating radar device that detects the distance to an underground portion of the crop present in the measurement range by transmitting electromagnetic waves.

[Item 9]
The agricultural machine includes:
a positioning unit that outputs a vehicle body position that is the position of the agricultural machine;
a control device that controls steering so that the position output by the positioning unit coincides with a predetermined travel route;
9. The crop detection system according to any one of claims 6 to 8, comprising:

[Item 10]
10. The crop detection system according to item 9, wherein the information processing device creates a shape of the crop based on the above-ground information and the underground information, and creates a field map on which the travel route is set by expanding the created shape of the crop into a field representing the field.

[Item 11]
The agricultural machine includes a control device for controlling automatic traveling,
11. The crop detection system according to any one of claims 6 to 10, wherein the information processing device estimates the crop based on the ground information acquired by the first sensor during the autonomous driving and the underground information acquired by the second sensor during the autonomous driving.

[Item 12]
The agricultural machine includes a positioning unit that outputs a vehicle body position, which is a position of the agricultural machine,
The information processing device includes:
4. The crop detection system according to item 2 or 3, wherein the latitude and longitude of the crop are calculated from the vehicle position expressed in latitude and longitude and the distance from the agricultural machine to the specified crop obtained by at least the first sensor, and a field map is created in which the shape of the crop, which is created based on the crop estimated while the agricultural machine is traveling, corresponds to the latitude and longitude of the crop.

[Item 13]
Further comprising a display device,
The crop detection system described in item 2, wherein when the information processing device determines that it is unable to extract the specified underground information that is common to a specified piece of ground information among the multiple pieces of ground information, it displays a message on the display device to notify the user that the specified underground information cannot be extracted.

[Item 14]
the ground penetrating radar device is provided at a front portion of the agricultural machine,
Item 14. The crop detection system of any one of items 1 to 13, further comprising another ground penetrating radar device provided at a rear of the agricultural machine.

[Item 15]
A crop detection system according to any one of claims 1 to 14,
A drive device for driving the agricultural machine;
Agricultural machinery equipped with

[Item 16]
A work machine for performing ground work on the ground is provided,
Item 16. The agricultural machine according to item 15, wherein the ground penetrating radar device is attached to the work machine.

The technology disclosed herein can be applied to the operation of agricultural machinery such as tractors, harvesters, rice transplanters, riding tillers, vegetable transplanters, grass cutters, seed sowing machines, fertilizer applicators, or agricultural robots.

50: GNSS satellite, 60: reference station, 80: network, 100: work vehicle, 101: vehicle body, 102: engine, 103: transmission, 104: wheels, 105: cabin, 106: steering device, 107: driver's seat, 108: coupling device, 110: positioning device, 111: GNSS receiver, 112: RTK receiver, 115: inertial measurement unit (IMU), 116: processing circuit, 120: camera, 130: obstacle sensor, 140: underground radar device, 141: antenna na unit, 150...sensor group, 152...steering wheel sensor, 154...turning angle sensor, 156...rotation sensor, 160...control system, 161...processing device, 170...storage device, 180...control device, 181-185...ECU, 190...communication device, 200...operation terminal, 210...operation switch group, 220...LiDAR sensor, 240...driving device, 250...receiver, 300...work machine, 340...driving device, 380...control device, 390...communication device, 400...terminal device, 600...management device

Claims (12)

A crop detection system for detecting a crop in a field, comprising:
A first sensor for detecting ground information, which is information about an above-ground part of the crop;
A second sensor for detecting underground information, which is information about the underground part of the crop;
an information processing device that estimates a crop including an above-ground part of the crop and an underground part of the crop based on the above-ground information and the underground information;
A crop detection system comprising: The information processing device includes:
By referring to the plurality of pieces of ground information detected by the first sensor and the plurality of pieces of underground information detected by the second sensor, predetermined underground information common to predetermined pieces of ground information is extracted from the plurality of pieces of ground information;
Associating the extracted predetermined underground information with the predetermined ground information;
The crop detection system of claim 1 , wherein the crop is estimated from the predetermined above-ground information and the predetermined below-ground information associated with each other. The associated predetermined ground information and the associated predetermined underground information are sensing data obtained from distances detected by the first sensor and the second sensor, respectively;
The crop detection system according to claim 2 , wherein the information processing device estimates a size of the crop from the sensing data. The crop detection system according to claim 2 or 3, wherein the information processing device creates a shape of the crop based on the aboveground information and the underground information, and creates a field map by expanding the created shape of the crop into a field representing the field. The crop detection system according to claim 1 or 2, wherein the first sensor and the second sensor are mounted on a single agricultural machine or separately mounted on multiple agricultural machines. the first sensor is a distance measuring sensor that detects a distance to an above-ground part of the crop present in a measurement range set around the agricultural machine as the ground information, or a camera that detects an image of the above-ground part of the crop present in the measurement range as the ground information,
The crop detection system according to claim 1 , wherein the second sensor is a distance measuring sensor that detects a distance to an underground part of the crop present within the measurement range as the underground information. The crop detection system according to claim 6, wherein the distance measurement sensor in the first sensor is a LiDAR sensor that detects the distance to the above-ground part of the crop present in the measurement range by emitting a laser pulse. The crop detection system according to claim 6, wherein the second sensor is a ground-penetrating radar device that detects the distance to the underground part of the crop present within the measurement range by transmitting electromagnetic waves. The agricultural machine includes:
a positioning unit that outputs a vehicle body position that is the position of the agricultural machine;
a control device that controls steering so that the position output by the positioning unit coincides with a predetermined travel route;
The crop sensing system of claim 6 , comprising: The crop detection system according to claim 9, wherein the information processing device creates a shape of the crop based on the above-ground information and the underground information, and creates a field map on which the travel route is set by expanding the created shape of the crop into a field representing the field. The agricultural machine includes a control device for controlling automatic traveling,
The crop detection system of claim 6, wherein the information processing device estimates the crop based on the ground information acquired by the first sensor during the autonomous driving and the underground information acquired by the second sensor during the autonomous driving. The agricultural machine includes a positioning unit that outputs a vehicle body position, which is a position of the agricultural machine,
The information processing device includes:
4. The crop detection system according to claim 2 or 3, further comprising: determining the latitude and longitude of the crop from the vehicle position expressed in latitude and longitude and the distance from the agricultural machine to the specified crop obtained by at least the first sensor; and creating a field map that associates the shape of the crop, created based on the crop estimated while the agricultural machine is traveling, with the latitude and longitude of the crop.

Images