CN115035502A - Driver's behavior monitoring method, device, electronic device and storage medium - Google Patents

Abstract

The present disclosure provides a driver behavior monitoring method, device, electronic device, and storage medium, which relate to the technical field of artificial intelligence, specifically to the technical fields of image recognition, computer vision, feature recognition, and the like, and can be applied to the scenes of behavior monitoring, such as driver behavior monitoring, behavior analysis, and the like. The specific implementation scheme comprises the following steps: acquiring an image of a driver in a vehicle; determining a first image of a driver-related body frame region in the image; determining a second image of the head area of the driver and a third image of the hand area according to the first image; determining the body movement of the driver according to the first image, determining the angle of the driver relative to the advancing direction of the vehicle according to the second image, and determining the hand movement of the driver according to the third image; the behavior of the driver is recognized based on at least one of the body motion, the angle, and the hand motion. The method and the device have the advantages that the speed of recognizing the behavior of the driver is higher, and the accuracy is higher.

Description

Driver's behavior monitoring method, device, electronic device and storage medium

technical field

The present disclosure relates to the technical field of artificial intelligence, in particular to the technical fields of image recognition, computer vision, feature recognition, etc., and can be applied to scenarios such as data retrieval and information query, and in particular to a driver's behavior monitoring method, device, electronic device, and storage medium.

Background technique

The driving tool travels according to the driver's operation. With the increase of the number of driving tools, driving safety has also become the focus of the prior art.

During the driving process of the driving tool, the driver needs to pay attention to the situation of the driving tool and the road ahead of the vehicle at any time during the driving process of operating the driving tool. Therefore, the monitoring of the driver's behavior is realized to increase the driver's driving tool. Security is a top priority in existing technology.

In recent years, with the development of technology, the general method of monitoring the driver's behavior is to obtain a video or image of the driver's driving tool through a monitoring device, and artificially monitor the driver's video or image to identify Whether the driver has dangerous driving behaviors such as inattention, playing with mobile phones or sleeping.

SUMMARY OF THE INVENTION

The present disclosure provides a driver's behavior monitoring method, device, electronic device and storage medium, which can enrich the content of search suggestions, and recommend more abundant search suggestions more in line with user needs for users.

According to a first aspect of the present disclosure, there is provided a method for monitoring driver behavior, the method comprising: acquiring an image of a driver in a vehicle; determining a first image of a human body frame area related to the driver in the image; According to the first image, a second image of the driver's head area and a third image of the hand area are determined; the driver's body movement is determined according to the first image, and according to the second image determining the angle of the driver relative to the forward direction of the vehicle, and determining the hand movement of the driver according to the third image; according to at least one of the body movement, the angle and the hand movement species, identifying the driver's behavior.

According to a second aspect of the present disclosure, there is provided a device for monitoring driver behavior, the device comprising: the device includes: an acquisition module for acquiring an image of a driver in a vehicle; a first determination module for in the In the image, a first image of the driver-related body frame area is determined; a second determination module is configured to determine, according to the first image, a second image of the driver's head area, and a first image of the driver's hand area. three images; a third determination module, configured to determine the body movement of the driver according to the first image, determine the angle of the driver relative to the forward direction of the vehicle according to the second image, and determine the angle of the driver relative to the forward direction of the vehicle according to the second image The three images determine the hand motion of the driver; the recognition module is configured to recognize the driver's behavior according to at least one of the body motion, the angle and the hand motion.

According to a third aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor. The at least one processor executes to enable the at least one processor to perform the method of the first aspect.

According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method according to the first aspect.

According to a fifth aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method according to the first aspect.

In the present disclosure, an image of the driver in the vehicle is acquired; a first image of the driver-related body frame area is determined in the image; according to the first image, a second image of the driver's head area and a first image of the hand area are determined. Three images; determine the driver's body movement according to the first image, determine the driver's angle relative to the vehicle's forward direction according to the second image, and determine the driver's hand movement according to the third image; according to the body movement, angle and hand movement At least one of the driver's behaviors. It is equivalent to determining the driver's body movement, the angle relative to the vehicle's forward direction, and the body movement according to the first image, the second image, and the third image in the driver's image, and then according to the body movement, relative to the vehicle's forward direction. Angles and body movements to identify the driver's behavior. In turn, the recognition speed of the driver's behavior recognition is faster and the accuracy is higher.

It should be understood that what is described in this section is not intended to identify key or critical features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.

Description of drawings

The accompanying drawings are used for better understanding of the present solution, and do not constitute a limitation to the present disclosure. in:

1 is a schematic flowchart of a driver's behavior monitoring method according to an embodiment of the present disclosure;

2 is another schematic flowchart of a method for monitoring driver behavior provided by an embodiment of the present disclosure;

3 is a schematic diagram of the composition of a driver's behavior monitoring device provided by an embodiment of the present disclosure;

FIG. 4 shows a schematic block diagram of an example electronic device 400 that may be used to implement embodiments of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

It should be understood that, in various embodiments of the present disclosure, the character "/" generally indicates that the contextual object is an "or" relationship. The terms "first", "second", etc. are used for descriptive purposes only, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features.

The driving tool travels according to the driver's operation. With the increase of the number of driving tools, driving safety has also become the focus of the prior art.

Specifically, during the driving process of the driving tool, the driver needs to pay attention to the situation of the driving tool itself and the road ahead of the vehicle at any time during the driving process of operating the driving tool. Therefore, the behavior of the driver is monitored to increase the number of drivers. The safety of driving tools is a top priority in the prior art.

In recent years, with the development of technology, the general method of monitoring the driver's behavior is to obtain a video or image of the driver's driving tool through a monitoring device, and artificially monitor the driver's video or image to identify Whether the driver has dangerous driving behaviors such as inattention, playing with mobile phones or sleeping.

However, the current method for monitoring driver behavior is time-consuming and labor-intensive, and the recognition speed is slow and the accuracy is low. The reliability of the identified results is also poor.

The embodiments of the present disclosure provide a method for monitoring driver behavior, which can be applied to behavior monitoring scenarios such as vehicle driver behavior monitoring, train driver behavior monitoring, and the like. The method can recognize the driver's behavior according to at least one of the driver's body motion, the driver's angle relative to the vehicle's forward direction, and the hand motion when the driver drives the vehicle, so that the method has the recognition speed The advantages of faster and higher accuracy, and also avoid the influence of human subjective opinions, increase the credibility of the identified results.

Optionally, the driver behavior monitoring method provided by the embodiment of the present disclosure may be applied to any behavior monitoring scenarios such as train cab driver behavior monitoring, vehicle cab driver behavior monitoring, and duty behavior monitoring. The application scenarios of this method are not limited.

The execution body of the driver behavior monitoring method provided by the embodiment of the present disclosure may be a computer, or may be a server or other computing device with data processing capability. For example, the aforementioned computer, server, computing device, etc. may be a background service device of any of the aforementioned application scenarios, such as a server in the background of the monitoring system. The present disclosure also does not limit the executive body of the driver's behavior monitoring method.

In some embodiments, the server may be a single server, or may also be a server cluster composed of multiple servers. In some embodiments, the server cluster may also be a distributed cluster. The present disclosure also does not limit the specific implementation of the server.

FIG. 1 is a schematic flowchart of a method for monitoring driver behavior according to an embodiment of the present disclosure. As shown in Figure 1, the method may include:

S101. Acquire an image of a driver in a vehicle.

Exemplarily, further description will be given by taking the monitoring of the driver's behavior in the cab of a train (such as a motor train, a train, etc.) and taking the execution theme as the server as an example. An image acquisition device is set in the cab of the train, and the image acquisition device is connected to a server set in the background. The image of the driver is sent to the server, and the server receives and acquires the image captured by the image acquisition device, thereby completing the purpose of acquiring the image of the driver in the train cab.

In practical applications, before the driver enters the cab, the image acquisition device has been acquiring the image of the preset position in the cab, and sends the image to the server for image recognition. When the server recognizes the image in the image When the driver leaves, the server sends a first instruction to acquire the image of the driver to the image acquisition device, and the image acquisition device acquires the image of the driver in the cab according to the first instruction, until the server detects that there is no driver in the image. When the driver is in the driver's seat, a second instruction is sent to the image acquisition device, and the image acquisition device stops acquiring the image of the driver in the cab according to the second instruction.

It is understandable that the image acquisition device is fixedly arranged at a fixed position in the train cab, and is used to acquire an image of the fixed position in the train cab, that is, in practical applications, the image acquisition device must include the image in the field of view. The driver's seat. Optionally, the field of view of the image acquisition device may be set according to the position of the driver in the train, which is not specifically limited herein.

In practical applications, after the server obtains the image in the vehicle, it needs to identify the image to determine whether there is a person in the image, and then identify the identity information of the person to determine whether the driver is in the cab. If the person in the image is the driver, continue to acquire the image of the driver in the vehicle.

S102. Determine a first image of a driver-related body frame area in the image.

It can be understood that after acquiring the avatar of the driver in the vehicle, the image needs to be further processed in order to monitor the behavior of the driver.

Specifically, image recognition is performed on the image to determine the position corresponding to the driver in the image, that is, the human body frame related to the driver in the image is determined according to the characteristics of the driver through image recognition. The range of the human body frame includes all areas of the driver's human body, and the size of the human body frame can be set according to actual needs, which is not specifically limited here.

In practical applications, the driver's human body detection module is generally used to determine the first image of the driver-related human frame area in the image. There is a human frame detection model in the human body detection module. Identify, select an area in the image that meets the preset driver characteristics, and determine the image of the area as the first image of the driver-related human frame area, that is, the human frame detection model can determine the driver-related image in the input image. The first image of the body frame area.

S103. Determine a second image of the driver's head area and a third image of the hand area according to the first image.

It can be understood that after determining the first image of the body frame area related to the driver in the image of the driver in the vehicle, it is necessary to further analyze the driver's behavior, so it is also necessary to obtain the first image in the first image. Other images that can represent driver behavior.

Specifically, image recognition is performed on the first image, and a head region in the first image that matches the head feature is determined according to a preset head feature, and a second image including the head region corresponding to the head region is obtained, The entire area of the driver's head exists in the second image; the first image can also be characterized, and according to preset hand characteristics, the hand in the first image that matches the hand characteristics can be determined Then, a third image corresponding to the hand region is obtained. In the third image, there are all regions of the driver's hand. Since the human body has both left and right hands, the third image can be one, or It can be two. If the third image is two, the two third images represent the driver's left hand and right hand respectively. The size of the second image and the third image is set according to actual needs, which is not specified here. limited.

In practical applications, a head-hand detection module is generally used to determine the second image of the head area and the third image of the hand area in the first image, respectively, and the head-hand detection module has a head detection model and a hand respectively. a head detection model, the head detection model determines a second image in the first image that conforms to the head-related features by performing head-related feature recognition on the first image; A hand-related feature is identified, and a third image corresponding to the hand-related feature is determined in the first image.

S104. Determine the body movement of the driver according to the first image, determine the angle of the driver relative to the forward direction of the vehicle according to the second image, and determine the hand movement of the driver according to the third image.

It can be understood that after determining the first image, the second image and the third image in the image of the driver in the vehicle, it is also necessary to determine the action of the driver according to the first image, the second image and the third image.

Since the first image is an image of the driver-related body frame area, the body motion of the driver can be determined from the first image; the second image is an image of the driver's head area, and the second image can be Determining the facing direction of the driver means obtaining the angle of the driver relative to the forward direction of the vehicle; the third image is the driver's hand image, and the driver's hand movement can be determined according to the third image.

In practical applications, the first image is input into the preset driver human behavior analysis model, and by analyzing the first image, the driver human behavior analysis model outputs the action type corresponding to the driver. The action type may include sitting, lying down, etc.; the second image is input to a preset head angle analysis model, and by analyzing the second image, the head angle analysis model outputs the driver's heading relative to the vehicle. The third image is input into the preset hand motion recognition model, and by analyzing the third image, the hand motion recognition model outputs the corresponding hand motion of the driver.

S105: Identify the driver's behavior according to at least one of the body motion, the driver's angle relative to the vehicle's forward direction, and the hand motion.

It can be understood that after determining the driver's body movement through the first image, determining the driver's angle relative to the vehicle's forward direction through the second image, and determining the driver's hand movement through the third image, it is also necessary to The driver's body movement, the driver's angle with respect to the vehicle's forward direction, and the hand movement are used to identify the driver's behavior.

In the present disclosure, since the driver's body motion is obtained through the first image related to the driver's body frame area, the driver's angle relative to the vehicle's forward direction is obtained from the second image, and the driver's The hand motion is obtained according to the third image related to the driver's body frame area, so that the driver's behavior can be recognized through the body motion, the driver's angle relative to the vehicle's forward direction, and the hand motion, and then the driver's body The motion, the driver's angle relative to the vehicle's forward direction, and the hand motion are all used as judgment parameters for judging the driver's behavior, thereby increasing the accuracy of the recognized driver's behavior. Moreover, the driver's behavior is recognized through body movements, the driver's angle relative to the vehicle's forward direction, and hand movements. Compared with the traditional judgment of the driver's behavior, it has the advantages of faster recognition speed and higher accuracy.

Optionally, before the human body frame detection model, the method further includes training the human body frame detection model, and the step of training the human body frame detection model includes:

The images marked with the human frame label and the images without the marked human frame are input into the model to be trained, and the to-be-trained model performs feature analysis on the images marked with the human frame and the images without the human frame, and divides the input images according to the markings. In order to mark images with human frame labels and images without marked human frame images, after training, the human frame detection model is obtained. In practical application, the human body frame detection model can input an image, determine the human body frame area in the image through feature recognition, and output the image of the relevant area of the human body frame in the image.

For example, the server inputs the acquired image of the driver in the vehicle into the human body frame detection model, and through the processing of the human body frame detection model, a first image of the driver-related body frame area in the image is obtained.

Optionally, the present disclosure also includes identifying the driver's identity information.

When the server determines the first image of the driver's frame area on the image, it can also obtain the driver's identity feature information, and match the identity feature information with the identity feature information in the preset database according to the identity feature information, thereby Identification information representing the identity of the driver is identified.

In practical applications, after identifying the driver's identity information, the driver's identity information may be marked on the first image.

Optionally, before using the head detection model and the hand detection model, it also includes training the head detection model and the hand detection model, and the steps of training the head detection model and the hand detection model include:

Using a deep neural network model, the images with head area labels and the images without head area are input into the deep neural network model. The image is characterized by analysis, and the deep neural network model divides the input image into an image with a head region label and an image without a head region according to the label, so as to obtain the head detection model. Then, in practical application, the head detection model can input the first image of the driver-related body frame area, determine the head area in the first image through feature recognition, and use the head in the first image to determine the head area in the first image. The region is output as a second image.

The training process of the hand detection model is roughly the same as the above-mentioned head detection model, and the process of training the hand detection model will not be repeated here. After training the hand detection model, the hand detection model can input driver-related In the first image of the human body frame area, the hand area in the first image is determined through feature recognition, and the hand area in the first image is output as a third image.

Optionally, before using the preset driver human behavior analysis model, it is also necessary to train the preset driver human behavior analysis model, and input images marked with different body action labels into the deep neural network model. , the deep neural network model classifies different body movements according to the labels, so as to realize the training of the deep neural network model. In practice, the trained deep neural network model can input the first images of different behaviors. The behavior in the image is analyzed to output the type of body movement in the first image.

The training process of the head angle analysis model and the hand motion recognition model is roughly the same as the above-mentioned driver human behavior analysis model, and the training process of the head angle analysis model and the hand motion recognition model will not be repeated here. After the image is passed through the head angle analysis model, the head angle analysis model can input the second image and output the angle of the driver relative to the direction of the vehicle; after the third image is trained by the hand motion recognition model, the hand angle analysis model The third image may be input, and the driver's hand motion type may be output.

Optionally, the step of S105 includes:

According to the body movement, identify whether the driver has the first behavior.

According to the driver's human behavior analysis model, the action of the driver is determined, and according to the corresponding relationship between the action and the behavior of the driver, it is further determined whether the driver has the first behavior.

In practical applications, the first behavior is set according to actual needs, and there is no specific limitation here. If the first behavior is a sleeping behavior, the body movements corresponding to the sleeping behavior are the driver lying on his stomach or lying down. etc., that is, if the action of the driver is found to be lying on his stomach or lying down through the analysis of the driver's human behavior analysis model, then it is determined that the driver has the first behavior of sleeping. Specifically, the body motion corresponding to the first behavior is set according to actual needs, which is not specifically limited here. And the first act is a correct act, but it can also be a violation.

Due to the long running time of the train, according to the regulations, when the driver continues to drive for more than four hours, he must relax and rest. The behavior of relaxation can also be taken as the first behavior, and the physical actions corresponding to the relaxation behavior are jumping and walking. etc., that is, if the action of the driver is jumping or walking through the analysis of the driver's human behavior analysis model, it is determined that the driver has the first behavior of relaxation.

Optionally, if the first behavior is a sleeping behavior, the step of identifying whether the driver has the first behavior according to the body movement includes: identifying the sleeping behavior of the driver according to the body movement.

According to the driver's human behavior analysis model, the action of the driver is determined, and according to the corresponding relationship between the action and the behavior of the driver, it is further determined whether the driver has sleeping behavior.

In practical applications, the body movements corresponding to the sleeping behavior are the driver lying on his stomach, lying down, etc. That is, if the driver's human behavior analysis model shows that the driver's action is lying or lying down, then It is determined that the driver has sleeping behavior. The body action corresponding to the sleeping behavior is set according to actual needs, which is not specifically limited here.

Optionally, the S105 further includes:

According to the hand motion, identify whether the driver has the second behavior.

Through the hand motion recognition model, the driver's hand motion is determined, and according to the corresponding relationship between the hand motion and the behavior, it is further determined whether the driver has the second behavior.

In practical applications, the second behavior is set according to actual needs. The second behavior can be a violation or a non-violation behavior, which is not specifically limited here. If the second behavior includes a non-violation behavior, for example: Salute, command, etc., if the second behavior includes illegal behavior: such as smoking, holding a mobile phone, etc.

If the second behavior includes salute or command, the hand movements corresponding to the salute or command behavior are raising hand, waving hand, etc., that is, if the hand movement recognition model is used, it is determined that the driver's hand movement is raising hand If the second behavior includes smoking, holding a mobile phone, etc., the hand action corresponding to the smoking or holding a mobile phone is holding, etc. That is, if the hand motion recognition model determines that the driver's hand motion is holding, etc., it is determined that the driver has a second behavior such as smoking or holding a mobile phone.

Specifically, the hand motion corresponding to the second behavior is set according to actual needs, which is not specifically limited here.

Optionally, if the second behavior is a smoking behavior, the step of identifying whether the driver has the second behavior according to the hand motion includes: identifying the smoking behavior of the driver according to the hand motion.

Through the hand motion recognition model, the driver's hand motion is determined, and according to the corresponding relationship between the hand motion and the behavior, it is further determined whether the driver has smoking behavior. In practical applications, the smoking behavior is set according to actual needs.

Optionally, if the second behavior includes salute or command, the hand action corresponding to the salute or command behavior is raising a hand, waving, etc., that is, if the hand action recognition model is used to determine the driver's hand. If the action is raising a hand, waving, etc., it is determined that the driver has a second behavior such as salute or command; if the second behavior includes smoking, holding a mobile phone, etc., the hand action corresponding to the smoking or holding a mobile phone is Holding, etc., that is, if the hand motion recognition model of the driver determines that the driver's hand motion is holding, etc., it is determined that the driver has a second behavior such as smoking or holding a mobile phone.

Specifically, the hand motion corresponding to the second behavior is set according to actual needs, which is not specifically limited here.

Optionally, the S105 further includes:

According to the angle, identify whether the driver has the third behavior.

Determine the driver's angle relative to the vehicle's forward direction through the head motion recognition model, and further determine whether the driver exists according to the corresponding relationship between the driver's driver's angle relative to the vehicle's forward direction and behavior third act.

In practical applications, the third behavior is set according to actual needs. The third behavior can be a non-violation behavior, that is, the head is in the same direction as the vehicle, which means that the driver is attentive, or it can be a violation. That is, if the included angle between the head and the forward direction of the vehicle is relatively large, it means that the driver is inattentive, which is not specifically limited here.

Determine the angle of the driver relative to the vehicle's forward direction through the head motion recognition model, and compare the angle with the preset angle. If the angle is smaller than the preset angle, it means that the driver is looking ahead, indicating that the driver is paying attention. , if the angle is greater than the preset angle, it means that the driver is facing other directions, which means that the driver is inattentive. In practical applications, the preset angle is set according to actual needs, which is not specifically limited here.

Optionally, if the third behavior is an attention behavior, the step of identifying whether the driver has the third behavior according to the angle includes: identifying whether the driver has the attention behavior according to the angle.

Determine the driver's angle relative to the vehicle's forward direction through the head motion recognition model, and further determine whether the driver exists according to the corresponding relationship between the driver's driver's angle relative to the vehicle's forward direction and behavior third act.

In practical applications, the third behavior is set according to actual needs. The third behavior can be a non-violation behavior, that is, the head is in the same direction as the vehicle, which means that the driver is attentive, or it can be a violation. That is, if the included angle between the head and the forward direction of the vehicle is relatively large, it means that the driver is inattentive, which is not specifically limited here.

Determine the angle of the driver relative to the vehicle's forward direction through the head motion recognition model, and compare the angle with the preset angle. If the angle is smaller than the preset angle, it means that the driver is looking ahead, indicating that the driver is paying attention. , if the angle is greater than the preset angle, it means that the driver is facing other directions, which means that the driver is inattentive. In practical applications, the preset angle is set according to actual needs, which is not specifically limited here.

Optionally, in response to the sleeping behavior being recognized, the confidence level of the sleeping behavior is obtained according to at least one of the distance between the head region and the hand region, and the overlapping area.

If the driver's sleeping behavior is identified based on body movements, it is necessary to verify whether the recognition result of the driver's sleeping behavior is accurate according to the distance and/or overlapping area between the head area and the hand area.

In order to increase the accuracy of judging the sleeping behavior, after identifying the driver's sleeping behavior according to the body movement, it is necessary to further verify the recognized behavior. In the first image, the second image and the third image are obtained. Since the second image represents the position of the driver's head and the third image represents the position of the driver's hand, the distance and/or overlapping area between the second image and the third image can be The distance, the overlapping area between the second image and the third image, the distance between the second image and the third image, and the overlapping area are used to verify the confidence that the driver has sleeping behavior.

In practical applications, the distance between the second image and the third image in the first image represents the relative distance between the driver's head and the hand, and the second image and the third image in the first image The overlapping area between the images represents the contact area between the driver's head and the hand. Generally, if the driver sleeps on his stomach, the distance between the head and the hand is short, and the There is a certain overlapping area between the head and the hand, that is, when the distance between the driver's head and the hand is less than the preset distance threshold, the overlapping area between the head and the hand is greater than the preset area threshold, the head When the distance between the hand and the hand is less than the preset distance threshold, and the overlapping area of the head and the hand is greater than the preset area threshold, the obtained confidence of the driver's corresponding sleeping behavior is greater than the preset confidence. The degree threshold, that is, the result of identifying the driver's sleeping behavior is more accurate. vice versa.

Optionally, the confidence of the driver's corresponding sleeping behavior is calculated according to the distance between the driver's head and the hand and the overlapping area of the head and the hand, which is not specifically limited here. The confidence may be a value or an interval, which is not specifically limited here. The confidence threshold is only a value for judging whether the confidence is credible, and details are not described here.

Specifically, since the second image and the third image are generally rectangular frames, the distance between the center point of the second image and the center point of the third image can be used as the distance between the driver's head and hand The distance between the second image and the third image can also be determined by judging the overlapping area of the rectangular frame of the second image and the rectangular frame of the third image to obtain the overlapping area between the second image and the third image. If the distance between them is less than the preset distance threshold, it can be verified that the driver has a high degree of confidence in sleeping behavior; if the overlapping area of the head and the hand is greater than the preset area threshold, it can be verified that the driver has sleeping behavior. The confidence is high; or, if it is determined that the distance between the driver's head and the hand is less than the preset distance threshold, and the overlapping area of the driver's head and the hand is greater than the preset area threshold, it can be verified that the The confidence level of the driver's sleep behavior is high. In practical applications, the preset distance threshold and the preset area threshold are set according to actual needs, which are not specifically limited here.

If the identification result of verifying that the driver has sleeping behavior is accurate, it is further determined that the driver has sleeping behavior.

Optionally, the confidence level of the driver's existing behavior is obtained according to at least one of the distance between the head area and the hand area and the overlapping area.

The above specifically explains how to determine the confidence level of the driver's sleeping behavior. For further explanation, here, the confidence level of the smoking behavior is specifically explained.

When it is recognized that the driver has smoking behavior according to the hand action, the confidence level of the driver's smoking behavior is verified according to the distance and/or overlapping area between the head area and the hand area.

In order to increase the accuracy of judging smoking behavior, further verification is required after the driver's smoking behavior is identified according to hand movements. In the first image, the distance and the distance between the second image and the third image are obtained. / or overlapping area, since the second image represents the position of the driver's head and the third image represents the position of the driver's hand, the distance between the second image and the third image, the distance between the second image and the third image, and the The overlapping area of the third image, the distance between the second image and the third image, and the overlapping area are calculated to obtain the confidence level for verifying that the driver has smoking behavior.

Optionally, the confidence level of the smoking behavior corresponding to the driver is based on the distance between the second image and the third image, the overlapping area between the second image and the third image, and the difference between the second image and the third image. The distance and the overlapping area are calculated, which are not specifically limited here, and the calculated confidence may be a value or an interval, which is not specifically limited here.

Specifically, since the second image and the third image are generally rectangular frames, the distance between the center point of the second image and the center point of the third image can be used as the distance between the driver's head and hand The distance between the second image and the third image can also be determined by judging the overlapping area of the rectangular frame of the second image and the rectangular frame of the third image to obtain the overlapping area between the second image and the third image. If the distance between the head and the hand If the distance is less than the preset distance threshold, it can be verified that the recognition result of the driver's smoking behavior is accurate; if the overlapping area of the head and the hand is greater than the preset area threshold, it can be verified that the driver's smoking behavior recognition result is accurate; Alternatively, if it is determined at the same time that the distance between the driver's head and the hand is less than the preset distance threshold, and the overlapping area of the driver's head and the hand is greater than the preset area threshold, it can be verified that the driver has smoking behavior The identification results are accurate. In practical applications, the preset distance threshold and the preset area threshold are set according to actual needs, which are not specifically limited here.

If it is verified that the confidence level of the driver's smoking behavior is greater than the threshold confidence level threshold, it is further determined that the driver has smoking behavior. The confidence threshold is only a value for judging whether the confidence is credible, and details are not described here.

The calculation method of the confidence level of the attention behavior is similar to the calculation method of the confidence level of the smoking behavior, and will not be repeated here.

FIG. 2 is another schematic flowchart of a method for monitoring driver behavior according to an embodiment of the present disclosure. as shown in picture 2. Optionally, the method further includes:

S201. Identify the driver's identity information according to at least one of the first image and the second image.

It can be understood that the present disclosure is used to monitor the driver's behavior, and the driver can be evaluated based on the result of monitoring the driver's behavior. Therefore, it is also necessary to identify the driver's identity information.

Specifically, the driver's identity information and the driver's characteristic information, as well as the corresponding relationship between the driver's identity information and the information in the figure are pre-stored in the server. When identifying the driver's identity information, you can Feature identification is performed in the first image, the second image, or the first image and the second image, the identified feature information is matched with the pre-stored feature information, and the driver's identity information is obtained according to the matching result.

S202. Mark the recognized behavior of the driver according to the driver's identity information.

According to the obtained driver's behavior and the driver's identity information, association is made, so that the driver's behavior is marked according to the driver's identity information, that is, the driver's behavior can be recorded in the driver's identity information. .

For example, if the driver's identity information is 007, and the monitored behavior of the driver includes sleeping and holding a mobile phone, the identity information of 007 will be marked with the behavior of sleeping and holding a mobile phone, so that the driver's behavior is consistent with that of the driver. The driver's identity information is associated, and the driver's identity information is marked with the driver's behavior.

In the present disclosure, the sleeping behavior of the driver is recognized according to the body motion; the smoking behavior of the driver is recognized according to the hand motion; the attention behavior of the driver is recognized according to the angle, and At least one of the distance and the overlapping area is used to obtain the confidence of the driver's behavior. The driver's body movements, the driver's angle relative to the vehicle's forward direction, and hand movements are all used as judgment parameters for judging the driver's behavior, thereby increasing the accuracy of the identified driver's behavior; Compared with the traditional judgment of the driver's behavior, it has the advantages of faster recognition speed and higher accuracy. And because the confidence of the driver's behavior can be obtained through at least one of the distance between the head area and the hand area and the overlapping area, and the confidence of the driver's behavior can be used to further make the identified driving. Increased accuracy of employee behavior.

FIG. 3 is a schematic diagram of the composition of a driver's behavior monitoring device according to an embodiment of the present disclosure. As shown in FIG. 3 , the apparatus may include: an acquisition module 301 , a first determination module 302 , a second determination module 303 , a third determination module 304 , and an identification module 305 .

an acquisition module 301, configured to acquire an image of a driver in the vehicle;

a first determining module 302, configured to determine the first image of the driver-related body frame area in the image;

The second determination module 303 is configured to determine the second image of the driver's head area and the third image of the hand area according to the first image;

The third determination module 304 is configured to determine the body movement of the driver according to the first image, determine the angle of the driver relative to the forward direction of the vehicle according to the second image, and determine the hand movement of the driver according to the third image;

The identification module 305 is configured to identify the driver's behavior according to at least one of body motion, angle and hand motion.

Optionally, the identification module 305 is specifically configured to identify the driver's sleeping behavior according to body movements;

Identify the driver's smoking behavior based on hand movements;

According to the angle, identify the driver's attention behavior;

Methods also include:

In response to the sleeping behavior being recognized, the confidence level of the sleeping behavior is obtained according to at least one of the distance between the head region and the hand region and the overlapping area.

Optionally, the obtaining module 301 is specifically configured to obtain the confidence of the driver's existing behavior according to at least one of the distance between the head region and the hand region and the overlapping area.

Optionally, the device also includes a marking module, a marking module (not shown in the accompanying drawings) for:

Identify the driver's identity information according to at least one of the first image and the second image;

According to the driver's identity information, the identified driver's behavior is marked.

Optionally, the identification module 305 is specifically configured to identify whether the driver has the first behavior according to the body movement.

Optionally, the identification module 305 is also used to verify that the driver has the behavior of sleeping according to the distance and/or overlapping area of the head area and the hand area when it is recognized that the driver is sleeping according to the body movement. Are the results accurate.

Optionally, the identification module 305 is further configured to identify whether the driver has the second behavior according to the hand action.

Optionally, the identification module 305 is also used for the second behavior including smoking behavior, and the device further includes: when it is recognized that the driver has smoking behavior according to the hand action, according to the distance and/or coincidence of the head area and the hand area. area to verify whether the identification results of the driver's smoking behavior are accurate.

Optionally, the identification module 305 is further configured to identify whether the driver has the third behavior according to the angle of the driver relative to the forward direction of the vehicle.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

In an exemplary embodiment, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, In order to enable at least one processor to perform the method as the above embodiment.

In an exemplary embodiment, the readable storage medium may be a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method according to the above embodiments.

In an exemplary embodiment, the computer program product comprises a computer program which, when executed by a processor, implements the method according to the above embodiment.

FIG. 4 shows a schematic block diagram of an example electronic device 400 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 4 , the electronic device 400 includes a computing unit 401 that can be programmed according to a computer program stored in a read only memory (ROM) 402 or a computer program loaded from a storage unit 404 into a random access memory (RAM) 403 Various appropriate actions and processes are performed. In the RAM 403, various programs and data necessary for the operation of the device 400 can also be stored. The computing unit 401 , the ROM 402 , and the RAM 403 are connected to each other through a bus 404 . An input/output (I/O) interface 405 is also connected to bus 404 .

Various components in the electronic device 400 are connected to the I/O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a magnetic disk, an optical disk etc.; and a communication unit 409, such as a network card, modem, wireless communication transceiver, and the like. The communication unit 409 allows the electronic device 400 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

Computing unit 401 may be various general-purpose and/or special-purpose processing components with processing and computing capabilities. Some examples of computing units 401 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 executes the various methods and processes described above, such as the driver's behavior monitoring method. For example, in some embodiments, a driver's behavior monitoring method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 404 . In some embodiments, part or all of the computer program may be loaded and/or installed on the electronic device 400 via the ROM 402 and/or the communication unit 409 . When a computer program is loaded into RAM 403 and executed by computing unit 401, one or more steps of the driver's behavior monitoring method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the driver's behavior monitoring method by any other suitable means (eg, by means of firmware).

Various implementations of the systems and techniques described herein above can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that The processor, which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage device, at least one input device, and at least one output device, and transmit data and instructions to the storage device, the at least one input device, and the at least one output device an output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, performs the functions/functions specified in the flowcharts and/or block diagrams. Action is implemented. The program code may execute entirely on the machine, partly on the machine, partly on the machine and partly on a remote machine as a stand-alone software package or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user's computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a distributed system server, or a server combined with blockchain.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions of the present disclosure can be achieved, no limitation is imposed herein.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure should be included within the protection scope of the present disclosure.

Claims (11)

1. A method of monitoring driver behavior, the method comprising:

acquiring an image of a driver in a vehicle;

determining a first image of the driver-related body frame area in the image;

determining a second image of the head area of the driver and a third image of the hand area according to the first image;

determining the body motion of the driver according to the first image, determining the angle of the driver relative to the forward direction of the vehicle according to the second image, and determining the hand motion of the driver according to the third image;

identifying behavior of the driver based on at least one of the body motion, the angle, and the hand motion.

2. The method of claim 1, wherein the identifying the driver's behavior as a function of at least one of the body motion, the driver's angle relative to the vehicle heading, and the hand motion comprises at least one of:

according to the body action, recognizing the sleeping behavior of the driver;

identifying smoking behavior of the driver according to the hand action;

according to the angle, recognizing the attention behavior of the driver;

the method further comprises the following steps:

and in response to the sleeping behavior being recognized, acquiring the confidence of the sleeping behavior according to at least one of the distance and the overlapping area between the head region and the hand region.

3. The method according to claim 1 or 2, characterized in that the method further comprises:

and acquiring the confidence that the driver has the behavior according to at least one of the distance and the overlapping area between the head area and the hand area.

4. The method according to any one of claims 1-3, further comprising:

identifying identity information of the driver according to at least one of the first image and the second image;

and marking the recognized behavior of the driver according to the identity information of the driver.

5. A driver's behavior monitoring device, characterized in that the device comprises:

the acquisition module is used for acquiring an image of a driver in the vehicle;

a first determination module for determining a first image of the driver-related body frame region in the image;

the second determining module is used for determining a second image of the head area of the driver and a third image of the hand area according to the first image;

a third determination module, configured to determine a body motion of the driver according to the first image, determine an angle of the driver with respect to the vehicle forward direction according to the second image, and determine a hand motion of the driver according to the third image;

an identification module to identify a behavior of the driver based on at least one of the body motion, the angle, and the hand motion.

6. The apparatus according to claim 5, wherein the identification module is specifically configured to:

according to the body action, recognizing the sleeping behavior of the driver;

identifying the smoking behavior of the driver according to the hand action;

according to the angle, recognizing the attention behavior of the driver;

the method further comprises the following steps:

and in response to the sleeping behavior being recognized, acquiring the confidence of the sleeping behavior according to at least one of the distance and the overlapping area between the head region and the hand region.

7. The apparatus of claim 5 or 6, wherein the obtaining module is further configured to:

and acquiring the confidence coefficient that the driver has the behavior according to at least one of the distance and the overlapping area between the head area and the hand area.

8. The apparatus according to any one of claims 5-7, further comprising a marking module configured to:

identifying identity information of the driver according to at least one of the first image and the second image;

and marking the recognized behavior of the driver according to the identity information of the driver.

9. An electronic device, comprising: at least one processor; and a memory communicatively coupled to the at least one processor;

wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

10. A non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method according to any one of claims 1-4.

11. A computer program product, characterized in that it comprises a computer program which, when being executed by a processor, carries out the method according to any one of claims 1-4.

Images