CN115909733B - Driving intention prediction method based on cross-domain perception and mental theory - Google Patents

Abstract

The invention discloses a driving intention prediction method based on a cross-domain perception and mental theory, which comprises the following steps: firstly, integrating multi-mode sensing data; secondly, constructing a cross-domain perception theory; thirdly, constructing a driver cognitive structure model based on a mental theory; fourthly, constructing a man-machine collaborative driving intention model; the beneficial effects are that: information support is provided for the system to better cooperate with a driver to finish driving tasks; the use difficulty of the man-machine co-driving system is reduced, so that the practicability of the man-machine co-driving system is improved; the universality and the effectiveness of driving intention prediction are greatly improved; the interpretability of the man-machine co-driving system can be obviously improved; the driver acceptance and the trust of the man-machine co-driving system can be improved; the whole system has good universality for common drivers, and meanwhile, the iterative updating characteristic is easy to conduct personalized retraining.

Description

A driving intention prediction method based on cross-domain perception and theory of mind

Technical Field

The present invention relates to a driving intention prediction method, and in particular to a driving intention prediction method based on cross-domain perception and theory of mind.

Background Art

With the continuous development of automobile technology, the level of vehicle intelligence is getting higher and higher. Smart cars have become one of the main development directions of automobile technology, and have been studied and applied in all aspects of vehicles. The vehicle operation process of smart cars can be roughly summarized as: perception, decision-making and planning, and control. These three aspects have now become recognized research hotspots in the automotive industry. According to the classification standards of the International Society of Automotive Engineers (SAE International), the current main technical breakthroughs are concentrated on the research related to conditional autonomous driving and highly autonomous driving at levels L3 and L4. Human-machine co-driving technology is a highly feasible and practical smart car solution that was produced under such a background.

Due to the human-in-the-loop nature of human-machine co-driving technology, an excellent human-machine co-driving system perception layer should, in addition to meeting the common autonomous driving perception layer requirements, add additional perception modules for the driver of the main vehicle. As the "other driver" who understands each other, they can work together to complete the overall driving perception task, thereby significantly reducing the driver's perception burden. In the above perception module, a very important task is to predict the driving intention of the driver and even the entire human-machine co-driving system.

Since the human-machine co-driving system has the characteristics of collaborative driving between humans and autonomous driving systems, the driving intention of the human-machine co-driving system has the following characteristics. First, duality. Unlike the autonomous driving system under the usual definition, the human-machine co-driving system has two "drivers" at the same time, the human driver and the autonomous driving system. After the perception, planning, decision-making, and control of the two, they are organically combined through the human-machine co-driving system and then act on the vehicle. Therefore, the prediction of the driving intention of the human-machine co-driving system should include information in two dimensions: the prediction of the human driver's driving intention and the understanding of the driving intention of the autonomous driving system. Second, partial knownness. For the human-machine co-driving system, although the driving intention of the human driver is hidden, the driving intention of the "driver" of the autonomous driving system can be directly obtained by reading the key output signals of the decision-making layer and the control layer and then calculating in reverse.

At present, domestic and foreign teams have explored relevant methods for predicting driving intentions, but existing technologies still have certain shortcomings. First, there is a lack of research on collaborative driving intentions for human-machine co-driving systems. Relevant research on driving intentions is still mainly focused on the prediction of drivers' driving intentions, which cannot be well applied to collaborative human-machine co-driving systems. Secondly, there are few technologies that use the dynamic microscopic traffic flow of vehicle changes as an influencing factor, and most predictions of driving intentions are based on limited traffic scenarios. Finally, the exploration of theory of mind modeling based on driver cognitive mechanisms in driving intention prediction is still not in-depth enough, and most technologies still rely solely on data-driven driving intention predictions, with poor interpretability.

Chinese patent CN202210243030.X discloses a driving intention recognition method and system for mixed traffic flow, which recognizes the driver's lane change intention by sensing the traffic information of the mixed traffic flow and the status information of the surrounding traffic participants through the calculation of the deep neural network. Chinese patent CN202210637423.9 discloses a driving intention recognition method based on fuzzy theory and big data analysis, which clusters and constructs a driving intention recognition rule base based on historical data, and implements the driving intention corresponding to the driving data based on fuzzy rule recognition. Chinese patent CN202210784309.9 provides a vehicle intention prediction method, system, electronic device and medium, which determines the driving status information of the target vehicle by collecting the environmental image information and lane change status information of the vehicle to be controlled, and then predicts the driving intention of the target vehicle. The above three patents can only predict the driver's driving intention based on the limited perception domain of the on-board sensor system of the intelligent networked vehicle, and are not well applicable to the human-machine collaborative driving system, and cannot well cope with the surrounding dynamically changing micro-traffic flow feature domain and micro-traffic flow situation.

Summary of the invention

The main purpose of the present invention is to accurately assess the traffic flow situation by fusing information from the ground domain and the low-altitude domain for cross-domain perception;

Another object of the present invention is to establish a cognitive structure model of the driver based on the theory of mind, and to accurately identify the driver's driving intention by conducting a detailed analysis of the driver's cognitive structure and combining it with the current traffic flow situation;

Another object of the present invention is to calculate the human-machine collaborative driving intention based on the human-machine collaborative mechanism of the human-machine co-driving system;

In order to achieve the above-mentioned purpose, the present invention provides a driving intention prediction method based on cross-domain perception and theory of mind.

The driving intention prediction method based on cross-domain perception and theory of mind provided by the present invention comprises the following steps:

The first step is to integrate multimodal perception data. The specific process is as follows:

Step 1: Collecting ground-air data. Cross-domain perception refers to using ubiquitous communication equipment to collect data from different information domains and derive the state of other perception domains of the perceived target. Cross-domain perception mainly refers to perceiving information across the ground domain and low-altitude domain, and deriving microscopic traffic flow and traffic situation information within the perception range.

Step 2: Collecting long-term driving data of drivers. Collecting the operation, expression and driving intention information of several different types of drivers for a long time to form an offline database to support the training and verification of the driver cognitive structure model based on the theory of mind in the third step;

Step 3: Driver short-term driving data collection: When the system is actually running, the driver's status information within the current period is collected for online prediction of the driver's driving intention in the current state. Since the system training is based on the data collected by the driving simulator and the actual use is based on the actual vehicle conditions, it is necessary to perform initial value adjustment based on the steering clearance, brake clearance and hardware parameters of the installation position of each sensor of the actual vehicle to solve the zero drift problem caused by different vehicle hardware.

Step 4: Multimodal data preprocessing. In this step, the data collected in the first three steps are preprocessed as follows:

The multi-sensor information collected by the intelligent vehicle-mounted sensor system in step one is subjected to sensor fusion to obtain the perception map of the intelligent driving system and fit the driver's perception map; the image information collected by the drone aerial survey is identified to obtain the information of traffic participants in the traffic flow. The information collected in step one is processed and mainly output to the second step for cross-domain perception;

The collected information is stored in a one-to-one correspondence between the scene driver's driving intention, microscopic traffic flow situation, driver's operation and driver's facial information to reflect their relevance. The information collected in step 2 is processed and output to step 3 for training to form a driver's cognitive structure model;

The data collected in step 3 is processed in a similar manner to the data in step 2, and after processing, it is input into step 1 of step 4 to predict the driver's driving intention;

The second step is to build a cross-domain perception theory. The specific process is as follows:

Step 1: Domain similarity comparison. First, the data obtained from the aerial survey is clustered, and the data in the offline database is divided into several traffic environment domains with the same characteristics. The clustering methods include manual clustering and algorithm-based automatic clustering. The advantage of manual clustering is that it can easily handle high-level features that are difficult or even impossible for computers to understand. The aerial survey data is divided into safe urban road domains, safe highway domains, safe rural road domains, dangerous urban road domains, dangerous highway domains, and dangerous rural road domains according to the characteristics. The advantage of algorithm-based automatic clustering is that it is extremely efficient and has significant advantages in dealing with the estimation problem of microscopic traffic flow trends. The method used is based on the K-means clustering method for automatic classification in this step.

Step 2: Domain deviation calibration: bring the information collected by drone aerial survey when the system is online into the clustering result of step 1 for classification, and calculate the deviation between the position of the microscopic traffic flow where the main vehicle is located in the cluster at the current moment and the center of the classification result, which is used for adjustment and calibration in step 3;

If the microscopic traffic flow where the main vehicle is located and the clustering result are not successfully matched in this step, manual clustering is required;

Step 3: cross-domain information phase calibration;

Step 4: Estimation of microscopic traffic flow situation. In this step, the online traffic situation estimation of microscopic traffic flow is performed based on the cellular transmission model and the macroscopic basic diagram of traffic flow. The cellular transmission model is based on the principle of flow transmission conservation. After being corrected by the actual traffic flow characteristics, the cellular transmission model is used to discretize the flow in time.

The position of the cell where the main vehicle is located in the graph is found by searching, and its traffic volume and density are brought into the calibrated macroscopic basic graph of traffic flow. According to its position in the graph, its traffic situation is divided in the form of stage rating, and the curve is divided into equal parts according to the arc length and then projected to the horizontal coordinate. Then, the final microscopic traffic flow situation is obtained according to the interval of the horizontal coordinate of the cell where the main vehicle is located in the graph.

The third step is to build a driver cognitive structure model based on the theory of mind. The specific process is as follows:

Step 1: Driver meta-component structure model. Meta-component is the most advanced and important part of cognitive structure. It is the individual's cognition of his own cognition. It is used to execute plans, make decisions and monitor to guide actions. It is the most critical factor in solving problems. The operational definition of metacognition is: metacognition is the individual's monitoring, regulation, control, evaluation and reflection of his own cognitive processing;

Step 2: Driver manipulation model. Manipulation in theory of mind is also called operation, which refers to the psychological processing process of a series of activities in which the system executes instructions from the control center and encodes, processes, converts and transfers input information, extracted information and working memory. Manipulation mainly involves two levels: basic operation and method strategy. The driver manipulation model mainly models the method strategy level.

Step 3: Driver Schema Library. The diagram in the theory of mind refers to the organic structure of the representation of thoughts, that is, the knowledge and experience in the individual's mind, which is used to represent reality. The driver's schema library refers to the representation actions related to the driver's driving behavior.

The above three steps form a complete driver cognitive structure model, in which the driver meta-component model established in step 1 serves as the driver's cognition of himself, guides the driver's operation, and outputs the representation as the driver diagram library. The whole process is logically reversible, that is, the driver diagram library reflects the driver's operation and acts on the driver meta-component model, which provides a basis for using the driver cognitive structure model to reason about the driver's driving intention.

Step 4: Build a human-machine collaborative driving intention model. The specific process is as follows:

Step 1: Driver intention identification: the driver's short-term driving data collected and preprocessed in the first step, the micro-traffic flow feature domain obtained by clustering in the second step, and the estimated micro-traffic flow situation are used as input to the driver's cognitive structure model established in the third step. According to the input of the graphic library, it is processed by the driver's meta-component model and then fed back into the driver's operation model, thereby inferring the driver's driving intention;

Step 2: Human-machine driving rights assessment. The driving intention of the human-machine co-driving system consists of two parts, namely the driver's driving intention and the automatic driving system's driving intention. In step 1, the online prediction of the driver's driving intention is completed, while the driving intention of the automatic driving system is a known quantity for the human-machine co-driving system and can be directly obtained by reading the output result of the decision layer of the system at the previous moment. The driving intention of the human-machine co-driving system is obtained by superimposing the two according to driving rights;

Step three: predicting the human-machine collaborative driving intention. By safely superimposing the calculation logic of the decision-making layer of the human-machine collaborative driving system, the final required human-machine collaborative driving intention can be obtained. In terms of expression, the human-machine collaborative driving intention ultimately exists in the form of a preview point, that is, within the next few steps, the human-machine collaborative driving system believes that the vehicle should track the driving trajectory position.

The first step of step 1 includes the following steps:

Link 1: Ground traffic environment information perception: The intelligent connected vehicle sensor system collects traffic environment information around the main vehicle, mainly including image information collected by the front-view camera and surround-view camera, point cloud information collected by the laser radar, target information collected by the millimeter-wave radar, and distance information collected by the ultrasonic radar, as well as the kinematic information of the vehicle position and speed collected by the inertial navigation and CAN bus interface. By collecting all the above-mentioned on-board sensor information, the ground traffic environment information is formed;

Link 2: Perception of low-altitude traffic flow information: Collect natural traffic data around the main vehicle through drone aerial survey. Drone aerial survey data mainly refers to the image stream data obtained by drone aerial photography. Due to its data source, the data has extremely comprehensive information on medium and large traffic participants, but the information on small traffic participants is relatively poor.

The steps in step 2 of step 1 are as follows:

Step 1: Set up the collection scene. Since the main purpose of the data collected in this step is training and verification, it is necessary to collect various information under the premise of knowing the traffic situation and the driver's driving intention. Therefore, a driving simulator will be used to collect data in a known scene. The source of the known scene is the known scene library established in the early stage. The advantage of this is that there are more alternatives for the same scene, which increases the randomness of the scene for the driver and makes the driver's operation closer to the real situation.

Known scenarios are defined in expected functional safety and refer to a single scenario in the scenario library established in the early stage. The scenario includes information about traffic participants, background environment and road conditions. Based on this information, the microscopic traffic flow situation in the current scenario can be accurately calculated.

Step 2: Collect the driver's control command information by installing sensors on the steering wheel, pedals, shifter and light controller of the driving simulator to collect the driver's vehicle control command information in known scenarios;

Step three: Collecting the driver's facial information. The camera sensor installed directly in front of the driver continuously collects the driver's facial information, mainly including the driver's eye movement information and head movement information. These two types of information are strongly correlated with the driver's cognitive behavior and can better reflect the driving intention.

Step 3 of the second step includes the following steps:

Step 1: Calculate the relative deviation. First, calculate the deviation between the center position and the boundary of the cluster according to the clustering result to get the maximum deviation, and calculate the ratio of the current deviation value to the maximum deviation to get the relative deviation.

Step 2: Calibrate the macro basic map of traffic flow. The micro traffic flow situation estimation is completed based on the macro basic map of traffic flow. In order to solve the specific problems in different traffic scenario domains, the macro basic map of traffic flow is calibrated according to the relative deviation calculated in step 1, and the macro basic map of traffic flow is appropriately scaled locally or overall according to the category and relative deviation.

The first step of the third step includes the following steps:

Link 1: Monitoring: The driver will monitor and observe the overall driving behavior during the driving process. If the driver component model is abstracted into a control theory model, this link can be completed by setting an observer;

Step 2: Adjustment: modify the parameters of the meta-component model based on the deviation between the previous round of system operation results and the expected operation results of the meta-component model, which is simplified to the parameter adjustment process;

Link 3, control, is directly related to the manipulation in the next step. From the perspective of the meta-component, it sends out signals to guide the manipulation model. When operating, the driver has a certain perceptual cognition of the object of control, that is, the human-machine co-driving system. The control method will be different from that of traditional cars or fully automatic driving systems. When establishing the model of this link, this factor needs to be taken into consideration and the model structure needs to be adjusted;

Step 4, evaluation, by calling the monitoring results of step 1 to calculate the errors and errors generated during the operation of the system, which are used as input to guide the adjustment process of step 2;

Step 5: Reflection, internal evolution of the model. By applying the completed model input of this round to the revised model adjusted in step 2, the adjustment effect of the model can be verified, so that the model can be gradually optimized and improved to solve the optimization problem of the model.

The steps in step 2 of step 3 are as follows:

Step 1: Cognitive strategy for solving the problem. Cognitive strategy is the driver's driving intention. The driver's driving intention has different forms of expression, including semantic level driving intention, operation instruction level driving intention and preview point level driving intention. In order to balance accuracy and simplicity of expression, the preview point level driving intention expression is adopted, that is, the position where the driver intends the vehicle to pass at the next moment;

Step 2: Heuristic strategy, establishing a mapping relationship between driving intention and basic operations. The training data used to establish the mapping relationship comes from the driver's long-term driving data collected and pre-processed in the first step and the driver's perception map formed by ground domain information.

The steps in step 3 of the third step are as follows:

Stage 1: Basic operations: Driver basic operations refer to the complete set of driver's operating actions, including steering wheel angle, clutch, brake and accelerator pedal positions, gear positions and lighting status, as well as the driver's eye and head movements;

Part 2: Basic driving knowledge. Basic driving knowledge refers to the relationship between basic operations under normal driving conditions due to physical conditions or driving rules, such as the absolute value of the steering wheel angle has a maximum, the brake pedal and accelerator pedal usually cannot be pressed at the same time, and the clutch pedal is usually pressed when changing gears.

Beneficial effects of the present invention:

1) The driving intention prediction method based on cross-domain perception and theory of mind described in the present invention can accurately predict the collaborative driving intention of the human-machine co-driving system, and provide information support for the system to better cooperate with the driver to complete the driving task;

2) The output result of the driving intention prediction method based on cross-domain perception and theory of mind described in the present invention is intuitive and clear, which can effectively reduce the driver's perceptual load and reduce the difficulty of using the human-machine co-driving system, thereby improving its practicality;

3) The driving intention prediction method based on cross-domain perception and theory of mind described in the present invention establishes a cross-domain perception fusion method that integrates low-altitude information and ground domain information based on cross-domain perception theory, which can solve the problem of perception domain deviation, thereby greatly improving the versatility and effectiveness of driving intention prediction;

4) The driving intention prediction method based on cross-domain perception and theory of mind described in the present invention establishes a cognitive structure model of the driver based on the theory of mind, systematically explains the generation mechanism and logical deduction process of the driver's driving intention, and can significantly improve the interpretability of the human-machine co-driving system;

5) The driving intention prediction method based on cross-domain perception and theory of mind described in the present invention has simple and clear logic, is easy for drivers to understand, and can improve the driver's acceptance and trust in the human-machine co-driving system;

6) The driving intention prediction method based on cross-domain perception and theory of mind described in the present invention forms statistical conclusions based on big data, so that the overall system has good universality for ordinary drivers. At the same time, due to its iterative update characteristics, it is easy to perform personalized retraining.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of the overall steps of the driving situation reasoning method of the present invention.

FIG. 2 is a schematic diagram of the overall architecture of the driving situation reasoning method of the present invention.

FIG3 is a schematic diagram of the overall architecture of the first step of the present invention.

FIG. 4 is a schematic diagram of the overall architecture of the second step of the present invention.

FIG5 is a schematic diagram of the overall architecture of the third step of the present invention.

FIG. 6 is a schematic diagram of the overall architecture of the fourth step of the present invention.

FIG. 7 is a diagram of a cellular transmission model described in the second step of the present invention.

FIG8 is an example of a macro basic diagram of traffic flow described in the second step of the present invention.

FIG. 9 is a structural diagram of the driver's cognitive structure model in the third step of the present invention.

FIG. 10 is a diagram showing an example of the calculation results of step three in the third step of the present invention.

FIG. 11 is a diagram showing an example of the calculation results of step three in the fourth step of the present invention.

DETAILED DESCRIPTION

Please refer to Figures 1 to 11:

The driving intention prediction method based on cross-domain perception and theory of mind provided by the present invention comprises the following steps:

The first step is to integrate multimodal perception data;

The second step is to build a cross-domain perception theory;

The third step is to build a driver cognitive structure model based on theory of mind;

Step 4: Build a human-machine collaborative driving intention model.

The process of integrating multimodal perception data in the first step is as follows:

Step 1: Collecting ground-air data. Cross-domain perception refers to using ubiquitous communication equipment to collect data from different information domains and derive the state of other perception domains of the perceived target. The cross-domain perception described in this patent mainly refers to the perception of information across the ground domain and the low-altitude domain, and the derivation of microscopic traffic flow and traffic situation information within the perception range. This step mainly includes the following two links:

Link 1: Perception of ground traffic environment information: The intelligent connected vehicle sensor system collects traffic environment information around the main vehicle, mainly including image information collected by the front-view camera and surround-view camera, point cloud information collected by the lidar, target information collected by the millimeter-wave radar and distance information collected by the ultrasonic radar, as well as kinematic information such as vehicle position and speed collected by the inertial navigation and CAN bus interface. By collecting all the above-mentioned vehicle-mounted sensor information, the ground traffic environment information is formed.

Link 2: Perception of low-altitude traffic flow information: Collect natural traffic data around the main vehicle through drone aerial survey. Drone aerial survey data mainly refers to the image stream data obtained by drone aerial photography. Due to its data source, the data has extremely comprehensive information on medium and large traffic participants, but the information on small traffic participants is relatively poor.

Step 2: Collect long-term driving data of drivers. Collect the operation, expression and driving intention information of multiple different types of drivers for a long time to form an offline database to support the training and verification of the driver cognitive structure model based on the theory of mind in the third step. It specifically includes the following three steps:

Step 1: Set up the collection scene. Since the main purpose of the data collected in this step is training and verification, we need to collect various information under the premise of knowing the traffic situation and the driver's driving intention. Therefore, here we use a driving simulator to collect data in a known scene. The source of the known scene is the known scene library established in the early stage. The advantage of this is that there are more alternatives for the same scene, which increases the randomness of the scene for the driver and makes the driver's operation closer to the real situation.

In particular, the known scenario mentioned here refers to the definition in the Safety of Intended Functionality (SOTIF), and in this patent, it refers to a single scenario in the scenario library established in the early stage. The scenario includes information such as traffic participants, background environment and road conditions. Based on this information, the micro-traffic flow situation in the current scenario can be accurately calculated.

Step 2: Collecting driver control command information: By installing sensors on the steering wheel, pedals, gear shifter and light controller of the driving simulator, the driver's vehicle control command information in known scenarios is collected.

Step 3: Collecting the driver's facial information. The camera sensor installed in front of the driver continuously collects the driver's facial information, mainly including the driver's eye movement information and head movement information. These two types of information are strongly related to the driver's cognitive behavior and can better reflect the driving intention.

Step 3: Collect the driver's short-term driving data. When the system is actually running, the driver's status information within the current period of time is collected for online prediction of the driver's driving intention in the current state. The collection equipment and method are basically the same as those of step 2, link 2 and link 3. It should be noted that since the system training is based on the data collected by the driving simulator and the actual use is based on the actual vehicle conditions, it is necessary to perform initial value adjustment based on the hardware parameters such as the steering clearance, braking clearance and installation position of each sensor of the actual vehicle to solve the zero drift problem caused by different vehicle hardware.

Step 4: Multimodal data preprocessing. In this step, the data collected in the first three steps are preprocessed. The details are as follows:

The multi-sensor information collected by the intelligent vehicle-mounted sensor system in step one is fused to obtain the perception map of the intelligent driving system and fit the driver's perception map; the image information collected by the drone aerial survey is identified to obtain the information of traffic participants in the traffic flow. The information collected in step one is processed and mainly output to the second step for cross-domain perception.

The driver's driving intention and microscopic traffic flow situation information described in step 2 are calculated by the known scene information selected in step 2, link 1; the driver's operation information is a time-series state vector sequence composed of the data collected by each sensor in step 2, link 2; the driver's facial information is a time-series eye pupil position vector sequence and head angle vector sequence formed by recognizing the image collected by the camera sensor in step 2, link 3. Furthermore, the information collected in the above three links should be stored in a one-to-one correspondence relationship between the scene driver's driving intention-microscopic traffic flow situation-driver operation-driver facial information to reflect its relevance. After processing, the information collected in step 2 is mainly output to the third step for training to form a driver's cognitive structure model.

The data collected in step three is processed in a similar way to the data in step two. After processing, it is input into step one of step four to predict the driver's driving intention.

An exemplary embodiment of the first step is shown in FIG. 3 .

The process of constructing the cross-domain perception theory in the second step is as follows:

Step 1, domain similarity comparison. First, cluster the data obtained from the aerial survey, and divide the data in the offline database into multiple traffic environment domains with the same characteristics. The clustering methods include manual clustering and algorithm-based automatic clustering. The advantage of manual clustering is that it can easily handle high-level features that are difficult or even impossible for computers to understand, and divide the aerial survey data into safe urban road domains, safe highway domains, safe rural road domains, dangerous urban road domains, dangerous highway domains, and dangerous rural road domains according to the characteristics. The advantage of algorithm-based automatic clustering is that it is extremely efficient and has significant advantages in dealing with the estimation problem of microscopic traffic flow trends. The method described in this patent is based on the K-means clustering method for automatic classification in this step.

Step 2: Domain deviation calibration. The information collected by the drone aerial survey when the system is online is brought into the clustering result of step 1 for classification, and the deviation between the position of the microscopic traffic flow where the main vehicle is located in the cluster at the current moment and the center of the classification result is calculated for adjustment and calibration in step 3.

In particular, if the microscopic traffic flow where the main vehicle is located and the clustering result are not successfully matched in categories in this step, manual clustering should be performed.

Step 3: Cross-domain information phase calibration. Calibration based on deviation specifically includes the following two steps:

Step 1: Calculate relative deviation. First, calculate the deviation between the center position and the boundary of the cluster according to the clustering result, that is, the maximum deviation, and calculate the ratio of the current deviation value to the maximum deviation to obtain the relative deviation.

Step 2: Calibrate the macro basic map of traffic flow. The micro traffic flow situation estimation described in this patent is completed based on the macro basic map of traffic flow. In order to solve the specificity problem in different traffic scene domains, the macro basic map of traffic flow is calibrated according to the relative deviation calculated in step 1, and the macro basic map of traffic flow is appropriately scaled locally or overall according to the category and relative deviation.

Step 4: Estimation of microscopic traffic flow situation. In this step, the online traffic situation estimation of microscopic traffic flow is performed based on the cellular transmission model and the macroscopic basic diagram of traffic flow. The cellular transmission model is based on the principle of flow transmission conservation. After being corrected by the actual traffic flow characteristics, the cellular transmission model after the flow is discretized by time can be expressed as the following formula:

n _i (k+1)＝n _i (k)+y _i (k)-y _i+1 (k)

q _i (k)=min{q _max,i ,v _i ρ _i-1 (k),w(ρ _jam -ρ _i (k))}

_yi (k)＝ _qi (k)ε

In the formula, i represents the cell number; k represents the discrete time step; y represents the traffic flow; n represents the number of vehicles in the cell; ε represents the cell interval, that is, the cell size; q represents the flow rate of the cell; ρ represents the traffic density; v represents the traffic speed under normal conditions; q _max,i represents the maximum flow rate of traffic flow; ρ _jam represents the blocking density; and w represents the reaction propagation speed of the congestion wave.

The position of the cell where the main vehicle is located in the graph is found by searching, and its traffic volume and density are brought into the calibrated macro basic graph of traffic flow. According to its position in the graph, its traffic situation is divided in the form of stage rating. Generally, the curve can be equally divided according to the arc length and then projected to the horizontal coordinate (density), and then the final micro traffic flow situation can be obtained according to the interval of the horizontal coordinate of the cell where the main vehicle is located in the graph.

FIG. 4 shows an exemplary embodiment of the second step.

FIG. 7 shows the cellular transport model described in step 4 of the second step.

FIG8 shows an exemplary curve of the traffic flow macro basic diagram described in the second step four.

The process of constructing the driver's cognitive structure model based on the theory of mind in the third step is as follows:

Step 1: Driver meta-component structure model. Meta-component is the most advanced and important part of cognitive structure. It is the individual's cognition of his own cognition. It is used to execute plans, make decisions and monitor to guide actions. It is the most critical factor in solving problems. According to the previous research on theory of mind, the operational definition of metacognition can be defined as: metacognition is the individual's monitoring, regulation, control, evaluation and reflection of his own cognitive processing. Therefore, the driver meta-component model can be divided into the following five links:

Step 1: Monitoring. The driver will monitor and observe the overall driving behavior during the driving process. If the driver component model is abstracted into a control theory model, this step can be completed by setting an observer.

Step 2: Adjustment: Based on the deviation between the last round of system operation results and the expected operation results of the meta-component model, the meta-component model parameters are modified, which is simplified to the parameter adjustment process.

Link 3: Control. This is directly related to the manipulation in the next step. From the perspective of the meta-component, it sends out signals to guide the manipulation model. In particular, since the driver has a certain perceptual cognition of the object he controls, that is, the human-machine co-driving system, when operating, the control method will be different from that of traditional cars or fully automatic driving systems. This factor needs to be taken into consideration when establishing the model of this link to adjust the model structure.

Phase 4: Evaluation: By calling the monitoring results of Phase 1, the errors and errors generated during the operation of the system are calculated and used as input to guide the adjustment process of Phase 2.

Step 5: Reflection. The internal evolution of the model is to verify the adjustment effect of the model by applying the completed model input of this round to the revised model adjusted in step 2, so that the model is gradually optimized and improved, solving the optimization problem of the model.

The first step in modeling the driver's cognitive structure model based on the theory of mind is to model the driver according to the five links mentioned above. Obviously, the driver meta-component model established consists of five modules corresponding to the five links mentioned above, and the relationship between each link is shown in the meta-component part in Figure x.

Step 2: Driver manipulation model. Manipulation in the theory of mind is also called operation, which refers to the psychological processing process in which the system executes instructions from the control center and encodes, processes, converts and transfers input information, extracted information and working memory. Manipulation mainly involves two levels: basic operation and method strategy. For ease of use, the driver manipulation model described in this patent mainly models the method strategy level, which specifically includes the following two links:

Section 1: Cognitive strategy for solving the problem. The cognitive strategy described in this section is the driver's driving intention. The driver's driving intention has different forms of expression, including semantic level driving intention, operation instruction level driving intention, and preview point level driving intention. In order to balance accuracy and simplicity of expression, this patent adopts the preview point level driving intention expression, that is, the location where the driver intends the vehicle to pass at the next moment.

Step 2: Heuristic strategy. Establish a mapping relationship between the driving intention described in step 1 and the basic operation in step 3, step 1. The training data used to establish the mapping relationship comes from the driver's long-term driving data collected and preprocessed in the first step and the driver's perception map formed by the ground domain information.

Specifically, we use the Bayesian network to establish the driver control model. The Bayesian network can be established by the following formula:

P(X ₁ ,X ₂ ,…,X _n )=P( _{X 1} |X _{2 ,} X ₃ ,…,X _n )·P( _{X 2} | P(X _n )

Where P represents the probability (or frequency) of an event; _Xi represents the event that occurs.

The driver manipulation model based on Bayesian network has the following advantages: First, Bayesian network can reveal the causal relationship between variables. Second, Bayesian network reveals the dependency between variables, which can avoid the bias when predicting missing data. Moreover, learning causal relationship can help data analysis of specific problems on the one hand, and help prediction under interference on the other hand.

Step 3: Driver Schema Library. The diagram in the theory of mind refers to the organic structure of the representation of thoughts, that is, the knowledge and experience of the individual mind, which is used to represent reality. The driver schema library described in this patent only refers to the representation actions related to the driver's driving behavior, specifically including the following two links:

Link 1: Basic operations. Driver basic operations refer to the complete set of driver's manipulation actions, which specifically include steering wheel angle, clutch, brake and accelerator pedal positions, gear positions and lighting conditions, as well as the driver's eye and head movements for this patent.

Part 2: Basic driving knowledge. Basic driving knowledge rules refer to the relationship between basic operations constrained by physical conditions or driving rules under normal driving conditions. For example, the absolute value of the steering wheel angle has a maximum value, the brake pedal and the accelerator pedal cannot usually be pressed at the same time, and the clutch pedal is usually pressed when the gear changes.

The connection between the steps in the third step is shown in Figure 9. The three steps form a complete driver cognitive structure model, in which the driver meta-component model established in step 1 serves as the driver's cognition of himself, guides the driver's manipulation, and outputs the representation as the driver diagram library. Furthermore, the whole process is logically reversible, that is, the driver diagram library reflects the driver's manipulation and acts on the driver meta-component model, which provides a basis for using the driver cognitive structure model to reason about the driver's driving intention.

FIG. 5 shows an exemplary embodiment of the third step.

The process of building a human-machine collaborative driving intention model in the fourth step is as follows:

Step 1: Identify the driver's intention. The driver's short-term driving data collected and preprocessed in the first step, the microscopic traffic flow feature domain obtained by clustering in the second step, and the estimated microscopic traffic flow situation are used as input to the driver's cognitive structure model established in the third step. The input from the graphic library is processed by the driver's meta-component model and then fed back into the driver's operation model, thereby inferring the driver's driving intention.

Step 2: Human-machine driving rights assessment. The driving intention of the human-machine co-driving system consists of two parts, namely the driver's driving intention and the driving intention of the automatic driving system. In step 1, the online prediction of the driver's driving intention was completed, and the driving intention of the automatic driving system is a known quantity for the human-machine co-driving system, which can be directly obtained by reading the output results of the decision layer of the system at the previous moment. Usually, the driving intention of the human-machine co-driving system is obtained by superimposing the two according to driving rights.

Step 3: Prediction of human-machine collaborative driving intention. The final required human-machine collaborative driving intention can be obtained by safely superimposing the calculation logic of the decision-making layer of the human-machine collaborative driving system. Furthermore, in terms of expression, the human-machine collaborative driving intention ultimately exists in the form of a preview point, that is, the human-machine collaborative driving system believes that the vehicle should track the driving trajectory position in the next few steps.

FIG. 6 shows an exemplary embodiment of the fourth step.

Claims (7)

1. A driving intention prediction method based on a cross-domain perception and mental theory is characterized by comprising the following steps of: the method comprises the following steps:

the first step, the multi-mode sensing data are integrated, and the specific process is as follows:

step one, ground-air data acquisition, namely acquiring data of different information domains by using ubiquitous communication equipment, deducing the state of a perception domain of a perception target, sensing information of a ground-crossing domain and a low airspace domain by using the cross-domain perception, and deducing microscopic traffic flow traffic situation information in a sensing range;

Step two, long-time driving data acquisition is carried out on a driver, operation, emotion and driving intention information of the driver with different digital types are acquired for a long time, an offline state database is formed, and the offline state database is used for supporting training and verification of a driver cognitive structure model based on the mental theory in the third step;

Step three, short-time driving data acquisition of a driver, namely acquiring state information of the driver in a current period of time when the system is actually operated, and predicting the driving intention of the driver in the current state on line, wherein the training of the system aims at the actual vehicle condition when the system is actually used based on the data acquired by a driving simulator, and initial value teaching is required according to the steering clearance and the braking clearance of the actual vehicle and the hardware parameters of the installation positions of all sensors, so that the zero drift problem caused by different vehicle hardware is solved;

Step four, multi-mode data preprocessing, in which some data preprocessing is performed on the data acquired in the first three steps, specifically as follows:

carrying out sensor fusion on the multi-sensor information acquired by the intelligent vehicle-mounted sensor system in the first step to obtain an intelligent driving system perception map and fitting the intelligent driving system perception map to obtain a driver perception map; identifying image information acquired by aerial survey of the unmanned aerial vehicle to obtain traffic participant information in traffic flow, and outputting the acquired information to a second step for cross-domain sensing after processing;

Storing the acquired information by adopting a one-to-one correspondence relation of scene driver driving intention, microscopic traffic flow situation, driver operation and driver face information during storage so as to show the relevance of the scene driver driving intention, microscopic traffic flow situation, driver operation and driver face information, processing the acquired information in the second step and outputting the processed information to a third step for training to form a driver cognitive structure model;

The data acquired in the third step is similar to the data acquired in the second step, and the data are input into the first step of the fourth step after being processed to predict the driving intention of the driver;

secondly, constructing a cross-domain perception theory, wherein the specific process is as follows:

Firstly, comparing domain similarity, namely firstly clustering data obtained by aerial survey, dividing the data in an offline database into a plurality of traffic environment domains with the same characteristics, wherein the clustering method comprises manual clustering and algorithm automatic clustering, the manual clustering has the advantages of easily processing advanced characteristics which are difficult to understand or even impossible to understand by a computer, dividing the aerial survey data into a safe urban road domain, a safe expressway domain, a safe rural road domain, a dangerous urban road domain, a dangerous expressway domain and a dangerous rural road domain according to the characteristics, and the algorithm automatic clustering has the advantages of extremely high efficiency and remarkable advantage in the aspect of processing the estimation problem of microscopic traffic flow situation, and the method is used for carrying out the automatic classification of the steps based on the K-means clustering;

Step two, domain deviation calibration, namely introducing information acquired by unmanned aerial vehicle aerial survey during online operation of the system into the clustering result of the step one for classification, and calculating the deviation between the position of the host vehicle in the clustering and the center of the classification result at the current time of the microscopic traffic flow, wherein the position is used for adjustment and calibration in the step three;

If the microscopic traffic flow of the host vehicle is not successfully matched with the clustering result still in category in the step, manual clustering is carried out;

step three, cross-domain information phase calibration;

estimating the situation of the microscopic traffic flow, wherein in the step, the online traffic situation estimation of the microscopic traffic flow is carried out based on a cell transmission model and a traffic flow macroscopic basic diagram, the cell transmission model is based on the principle of conservation of traffic flow transmission, and after the actual traffic flow characteristic is corrected, the traffic is discretized according to time;

searching to find the position of the cell of the host vehicle in the graph, bringing the traffic flow and the traffic density into a calibrated traffic flow macroscopic basic graph, dividing the traffic situation by the position of the cell in the graph in a stage grading mode, equally dividing the curve according to the arc length, projecting the curve to the abscissa, and obtaining the final microscopic traffic flow situation according to the region of the abscissa of the cell of the host vehicle in the graph;

Thirdly, constructing a driver cognitive structure model based on a mental theory, wherein the specific process is as follows:

Step one, a driver meta-component structure model, wherein meta-components are the highest level in a cognitive structure and are the most important parts, and are the cognition of individuals on the driver, so as to execute planning, make decisions and monitor to guide actions, and are the most critical factors for solving the problems, and the operability of meta-cognition is defined as follows: meta cognition is the monitoring, regulation, control, evaluation and disfigurement of an individual to its cognitive processes;

step two, a driver manipulates a model, which is also called operation in mental theory, and refers to a psychological processing process of a series of activities of coding, processing, converting and transferring input information, extracted information and working memory by a system executing instructions from a control center, wherein the manipulation involves two levels of basic operation and method strategy, and the driver manipulates the model to model the method strategy level;

Step three, a driver pattern library, wherein the graphic representation in the mental theory refers to a represented organic composition structure of thought, namely knowledge and experience of individual brains, for representing reality, and the driver graphic representation library refers to a representation action related to driving behaviors of a driver;

The three steps form a complete driver cognitive structure model, wherein the driver meta-component model established in the first step is taken as a driver to realize the driver, guide the driver to operate, and output and represent a driver graphic library, and the whole process has logic reversibility, namely the driver graphic library reflects the driver to operate and acts on the driver meta-component model, so that a basis is provided for the driver intention reasoning by using the driver cognitive structure model;

fourth, constructing a man-machine cooperative driving intention model, wherein the specific process is as follows:

Step one, identifying the intention of a driver, taking the micro traffic flow characteristic domain acquired and preprocessed by the driver in the first step and obtained by clustering in the second step and the estimated micro traffic flow situation as inputs, taking the micro traffic flow characteristic domain and the estimated micro traffic flow situation into a driver cognitive structure model established in the third step, and recharging the micro traffic flow characteristic domain and the estimated micro traffic flow situation into a driver operation model through the processing of a driver element component model according to the input of a graphic library, so as to reversely output the driving intention of the driver;

Step two, evaluating the driving intention of the man-machine co-driving system, wherein the driving intention of the man-machine co-driving system consists of two parts, namely, the driving intention of a driver and the driving intention of an automatic driving system, the on-line prediction of the driving intention of the driver is completed in step one, the driving intention of the automatic driving system is known quantity for the man-machine co-driving system and can be directly obtained by reading the output result of a decision layer of the last time system, and the driving intention of the man-machine co-driving system is obtained by superposing the two parts according to the driving intention;

Thirdly, predicting the human-computer co-driving intention, and carrying out safe superposition according to the calculation logic of a decision layer of the human-computer co-driving system to obtain the finally required human-computer co-driving intention, wherein in the aspect of expression form, the finally required human-computer co-driving intention exists in a form of a pre-aiming point, namely the human-computer co-driving system considers that the vehicle tracks the position of the running track in the next several steps.

2. The driving intention prediction method based on cross-domain perception and mental theory according to claim 1, wherein the method comprises the following steps: the first step comprises the following steps:

Step one, sensing the traffic environment information of the land area: the method comprises the steps of acquiring traffic environment information around a host vehicle through an intelligent network-connected automobile sensor system, wherein the traffic environment information comprises image information acquired by a front-view camera and a surrounding camera, point cloud information acquired by a laser radar, target object information acquired by a millimeter wave radar and distance information acquired by an ultrasonic radar, and kinematic information of the position and the speed of the vehicle acquired by inertial navigation and a CAN bus interface, and forming traffic environment information of a ground area through acquiring the information;

Step two, low airspace traffic flow information perception: natural traffic data around the main vehicle is collected through an unmanned aerial vehicle aerial survey mode, the unmanned aerial vehicle aerial survey data refer to image stream data obtained through unmanned aerial vehicle aerial photography, and the data has extremely comprehensive medium-and-large-sized traffic participant information due to data sources, and is relatively poor for small-sized traffic participant information.

3. The driving intention prediction method based on cross-domain perception and mental theory according to claim 1, wherein the method comprises the following steps: the second step in the first step comprises the following steps:

Firstly, setting an acquisition scene, wherein the acquired data is used for training and verification, and various information is acquired on the premise of knowing traffic situation and driving intention of a driver, so that a driving simulator is adopted for acquisition under the known scene, and the set known scene source is a known scene library established in the early stage;

the known scene is defined in the expected functional safety, and refers to a single scene in a scene library established in advance, wherein the scene comprises information of traffic participants, background environments and road conditions, and the microcosmic traffic flow situation under the current scene can be accurately calculated according to the information;

step two, acquiring control instruction information of a driver, namely acquiring the control instruction information of the vehicle of the driver in a known scene by additionally arranging sensors on a steering wheel, a pedal, a gear shifter and a light controller of a driving simulator;

and step three, collecting face information of a driver, wherein the face information of the driver is continuously collected through a camera sensor arranged right in front of the driver, and the face information comprises eye movement information and head movement information of the driver.

4. The driving intention prediction method based on cross-domain perception and mental theory according to claim 1, wherein the method comprises the following steps: the step three in the second step comprises the following links:

Calculating relative deviation, namely calculating the deviation between the central position of the cluster and the boundary according to the clustering result to obtain maximum deviation, and calculating the ratio of the current deviation value to the maximum deviation to obtain the relative deviation;

And step two, calibrating the traffic flow macroscopic basic diagram, namely completing microscopic traffic flow state potential estimation based on the traffic flow macroscopic basic diagram, calibrating the traffic flow macroscopic basic diagram according to the relative deviation calculated by the step one, and carrying out proper local or whole scaling on the traffic flow macroscopic basic diagram according to the difference of category and relative deviation in order to solve the specificity problem under different traffic scene domains.

5. The driving intention prediction method based on cross-domain perception and mental theory according to claim 1, wherein the method comprises the following steps: the step one in the third step comprises the following steps:

firstly, monitoring, namely, monitoring and observing the whole driving behavior by a driver in the driving behavior process, and if a driver element component model is abstracted into a control theory model, completing the link by setting an observer;

Step two, adjusting, namely modifying parameters of the meta-component model based on the deviation between the running result of the previous wheel system and the expected running result of the meta-component model, namely correspondingly simplifying the parameter adjusting process;

Step three, control, directly related to the control in the next step, from the perspective of element, sending out a signal for guiding the control model, wherein the driver has a certain perceptual cognition on the control object, namely the man-machine co-driving system, when in operation, the control mode is different from the control mode of the traditional automobile or the full-automatic driving system, and when the link model is established, the factor needs to be taken into consideration to adjust the model structure;

Step four, evaluating, namely calculating errors and errors generated in the running process of the system by calling the monitoring result of the step one, and using the errors and errors as an adjustment process of the step two;

And step five, back thinking, and model internal evolution, and verifying the adjusting effect of the model by inputting the model completed in the round to the modified model adjusted in the step two, so that the model is gradually optimized and perfected, and the optimization problem of the model is solved.

6. The driving intention prediction method based on cross-domain perception and mental theory according to claim 1, wherein the method comprises the following steps: the second step in the third step comprises the following steps:

The first link is a cognitive strategy for solving the problem, wherein the cognitive strategy is the driving intention of a driver, the driving intention expression forms of the driver are different, the driving intention of a semantic level, the driving intention of an operation instruction level and the driving intention of a pretightening point level are provided, and in order to consider the accuracy and the simplicity of the expression form, the driving intention expression form of the pretightening point level, namely the position where the driver deliberately plans the vehicle to pass next time is adopted;

And establishing a mapping relation between the driving intention and the basic operation by using a link II and heuristic strategy, wherein the training data source used for establishing the mapping relation is a driver perception map formed by long-time driving data of the driver and ground area information acquired and preprocessed in the first step.

7. The driving intention prediction method based on cross-domain perception and mental theory according to claim 1, wherein the method comprises the following steps: the third step comprises the following steps:

Step one, basic operation, wherein the basic operation of a driver refers to a complete set of operation actions of the driver, and comprises steering wheel rotation angle, clutch brake and accelerator pedal position, gear and lamplight state, and eye movement and head movement state of the driver;

And step two, basic driving knowledge, wherein the basic driving knowledge refers to the relation between basic operations constrained by physical conditions or driving rules in a normal driving state, such as the maximum absolute value of steering wheel angle, and the fact that a brake pedal and an accelerator pedal cannot be simultaneously stepped on and the clutch pedal is stepped on when the gear changes.