CN112307846A

CN112307846A - Analysis method for violation of crew service

Info

Publication number: CN112307846A
Application number: CN201910705893.2A
Authority: CN
Inventors: 胡斌; 苗江伟; 腾树标; 庞龙; 张东昆; 周良超
Original assignee: Beijing Sheenline Group Co Ltd
Current assignee: Beijing Sheenline Technology Co Ltd; Beijing Sheenline Group Co Ltd
Priority date: 2019-08-01
Filing date: 2019-08-01
Publication date: 2021-02-02

Abstract

The present application relates to a method for analyzing flight attendant violations. The method includes: performing human body posture estimation processing on crew video data to obtain a key point feature sequence; the key point feature sequence includes: key point feature data of multiple video frames; according to the key point feature sequence and a preset crew violation standard library, Determine whether there is a flight crew violation, and if so, output the flight crew violation data. Using this method can improve the analysis coverage and analysis efficiency.

Description

Analysis method for violation of crew service

Technical Field

The application relates to the technical field of video monitoring, in particular to a crew violation analysis method.

Background

With the development of rail transit technology, various rail locomotives such as ordinary trains, high-speed rails, motor cars and intercity trains appear; the operation of the locomotive is carried out by a crew member, so whether the working process of the crew member is standardized or not is directly related to the operation safety of the locomotive.

At present, the monitoring of the crew member is generally performed by sampling 6A videos of key parts (such as a cab) in the locomotive and then manually analyzing whether the problem of violation of the crew member exists; the 6A video is a video monitored by a locomotive vehicle-mounted safety protection system (6A system), and is mainly reliable video data obtained by effectively monitoring road conditions, machinery rooms, cabs and the like.

However, the current crew violation monitoring has the problems of insufficient sampling coverage rate, low analysis efficiency and the like, and is difficult to realize all-around monitoring and eliminate potential safety hazards.

Disclosure of Invention

In view of the above, it is necessary to provide a method, an apparatus, a server, a system, and a readable storage medium for analyzing a crew violation, which can improve analysis coverage and analysis efficiency.

In a first aspect, a crew violation analysis method, the method comprising:

carrying out human body posture estimation processing on the crew video data to obtain a key point feature sequence; the key point feature sequence comprises: keypoint feature data for a plurality of video frames;

and judging whether the crew violation exists or not according to the key point feature sequence and a preset crew violation standard library, and if so, outputting crew violation data.

In one embodiment, the performing human posture estimation processing on the crew video data to obtain a key point feature sequence includes:

intercepting the crew video data into a plurality of video clips;

and carrying out human body posture estimation processing on each video clip to obtain a key point feature sequence of each video clip.

In one embodiment, the intercepting the crew video data into a plurality of video clips includes:

acquiring locomotive operation monitoring record data corresponding to the crew service video data; the locomotive operation monitoring record data comprises time information of a service scene;

and intercepting a plurality of video clips with corresponding window lengths from the crew video data according to the time information of the service scene.

In one embodiment, the service scenario includes at least one of an inbound/outbound, a double-value-multiplication section, a single-value-multiplication section, and a crew pickup.

In one embodiment, the performing human body pose estimation processing on each video segment to obtain a keypoint feature sequence of each video segment includes:

carrying out human body posture estimation processing on each video clip to obtain key point characteristic data of a plurality of video frames of each video clip;

and tracking the key point characteristics of the plurality of video frames according to the key point characteristic data of the plurality of video frames of each video clip to obtain a key point characteristic sequence of each video clip.

In one embodiment, the keypoint feature data comprises: the method for obtaining the key point characteristic sequence of each video clip by carrying out human body posture estimation processing on each video clip comprises the following steps:

inputting each video clip into a preset depth regression network model for human body posture estimation processing, and outputting the key point static characteristics of a plurality of video frames in each video clip; the keypoint static features include: the relative position relationship between each key point category and each key point;

calculating to obtain the dynamic characteristics of the key points of a plurality of video frames in each video clip according to the static characteristics of the key points of the plurality of video frames in each video clip and the interval duration of the video frames; the dynamic feature of the key points comprises: the motion speed of each key point and the relative angle change relationship of each key point;

and combining the static characteristics and the dynamic characteristics of the key points of the plurality of video frames to obtain a key point characteristic sequence of each video clip.

In one embodiment, the performing human body pose estimation processing on each video segment to obtain a key point feature sequence of each video segment further includes:

if the time length of the key point feature sequence is longer than the preset time length, performing down-sampling processing on the key point feature sequence on a time sequence to obtain a key point feature sequence after down-sampling processing; the time length of the key point feature sequence after the down-sampling processing is equal to the preset time length;

if the time length of the key point feature sequence is less than the preset time length, performing up-sampling processing on the key point feature sequence on a time sequence to obtain an up-sampled key point feature sequence; and the time length of the key point feature sequence after the up-sampling treatment is equal to the preset time length.

In one embodiment, the method further comprises:

performing image target identification processing on each video clip to obtain object characteristic data;

the step of judging whether the crew violation exists according to the key point feature sequence and a preset crew violation standard library comprises the following steps:

and judging whether the crew violation exists or not according to the key point feature sequence, the object feature data and a preset crew violation standard library.

In one embodiment, the crew violation criteria library comprises preset crew violation determining conditions; the judging whether the crew violation exists according to the key point feature sequence, the object feature data and a preset crew violation standard library comprises the following steps:

inputting the key point feature sequences of the video clips into a preset support vector machine classification model to perform illegal action classification processing, and outputting illegal action types corresponding to the video clips; the violation action type corresponds to the crew violation determination condition;

and judging whether the crew violation exists or not according to the key point feature sequence, the violation action type, the object feature data and the crew violation judgment condition.

In one embodiment, the method further comprises the following steps: acquiring locomotive operation monitoring record data corresponding to the crew service video data;

the judging whether the crew violation exists according to the key point feature sequence, the object feature data and a preset crew violation standard library comprises the following steps:

and judging whether the crew violation exists or not according to the operation monitoring record data, the key point feature sequence, the object feature data and a preset crew violation standard library.

In one embodiment, the crew violation criteria library comprises a preset first type of crew violation determining condition; the first type of crew violation determining condition comprises: the crew member has a crew violation determination condition that uses the electronic device to act.

In one embodiment, the crew presence crew violation determination condition using an electronic device action includes:

whether the object characteristic data of the video clip comprises a human body and electronic equipment or not;

if so, whether the position information of the electronic equipment is matched with the position information of the wrist joint point of the human body;

if so, it is determined that a ride violation exists in the video segment.

In one embodiment, the crew violation criteria library comprises a second predetermined type of crew violation determining condition; the second type of crew violation determining condition comprises at least one of:

the crew member has a crew violation judgment condition for leg-lifting action, a crew violation judgment condition for snooze action, and a crew violation judgment condition for boredom state.

In one embodiment, the method further comprises:

sending the crew violation data to an auditing terminal for auditing;

receiving an audit state sent by the audit terminal aiming at each crew violation item in the crew violation data; the audit state comprises: the audit is passed and the audit is not passed;

and correcting the crew violation data according to the audit state.

In one embodiment, the method further comprises accounting for crew violation data and generating a crew violation report comprising at least one of:

the number of times/frequency of crew violations corresponding to different types of crew violation peaks;

the number/frequency of the service violations of different locomotives and different crew members;

a retrieval index table of a complete one-time crew operation process; the retrieval index table comprises the service violation statistical results corresponding to different retrieval items, and the retrieval items comprise at least one of the following contents: time slot, locomotive, crew.

In a second aspect, a crew violation analysis device comprises:

the attitude feature extraction module is used for estimating the human body attitude of the crew video data to obtain a key point feature sequence; the key point feature sequence comprises: keypoint feature data for a plurality of video frames;

and the violation judging module is used for judging whether the crew violation exists according to the key point feature sequence and a preset crew violation standard library, and outputting crew violation data if the crew violation exists.

In a third aspect, a crew violation analysis server includes a memory and a processor, where the memory stores a computer program, and the processor implements the steps of the crew violation analysis method when executing the computer program.

In a fourth aspect, a crew violation analysis system includes at least one task scheduling server and a plurality of the crew violation analysis servers; the task scheduling server is connected with the plurality of crew violation analysis servers and is used for receiving to-be-processed crew video data and distributing the crew video data to the plurality of crew violation analysis servers for crew violation analysis; and acquiring and summarizing the crew violation data output by the plurality of crew violation analysis servers.

In a fifth aspect, a readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the above-described crew violation analysis method.

According to the method, the device, the server, the system and the readable storage medium for analyzing the crew violation, the server can estimate and process the human posture of the crew video data to obtain the key point characteristic sequence; and then, based on a preset crew violation standard library, judging the key point feature sequence, judging whether the key point feature sequence has a corresponding crew violation, and outputting crew violation data when the crew violation exists, namely, the intelligent analysis of the crew video through the server realizes the crew violation analysis, so that the coverage rate, accuracy and efficiency of the crew violation analysis can be improved, the labor cost is reduced, the driving safety is guaranteed, and a decision basis is provided for the standardized operation of crew members.

Drawings

FIG. 1 is a diagram of an application environment for a method for analysis of a crew violation, according to one embodiment;

FIG. 2 is a flow diagram of a method for analysis of a crew violation, according to one embodiment;

FIG. 3 is a flow diagram of a method for analysis of a crew violation, according to one embodiment;

FIG. 4 is a flow diagram of a method for analysis of a crew violation, according to one embodiment;

FIG. 5 is a flow diagram of a method for analysis of a crew violation, according to one embodiment;

FIG. 6 is a block diagram of the structure of a crew violation analysis device in one embodiment;

FIG. 7 is a block diagram of a crew violation analysis system in one embodiment;

FIG. 8a is a first diagram of a crew violation analysis system in accordance with an embodiment;

FIG. 8b is a diagram of a second example of a crew violation analysis system;

FIG. 8c is a diagram of a third example of a crew violation analysis system;

FIG. 8d is a fourth diagram of a crew violation analysis system in one embodiment;

FIG. 8e is a fifth diagram of a crew violation analysis system in one embodiment;

FIG. 8f is a diagram illustrating a sixth example of a crew violation analysis system;

FIG. 8g is a seventh diagram of a crew violation analysis system in one embodiment;

FIG. 8h is an eighth schematic diagram of a crew violation analysis system in one embodiment;

FIG. 8i is a ninth schematic diagram of a crew violation analysis system in one embodiment;

FIG. 8j is a diagram ten of a crew violation analysis system in one embodiment;

FIG. 8k is an eleventh schematic diagram of a crew violation analysis system in one embodiment;

FIG. 8l is a diagram twelve of a crew violation analysis system in one embodiment;

FIG. 8m is a thirteen schematic diagram of a crew violation analysis system in one embodiment;

FIG. 8n is a fourteenth schematic diagram illustrating a crew violation analysis system in one embodiment;

FIG. 8o is a block diagram fifteen of a crew violation analysis system in one embodiment;

FIG. 8p is a diagram sixteen of a crew violation analysis system in one embodiment;

fig. 8q is a seventeenth schematic diagram of a crew violation analysis system in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

The crew violation analysis method provided by the application can be applied to the application environment shown in fig. 1. For example, the locomotive may be in direct communication with a server, and the server may obtain crew video data and operation monitoring record data sent by the locomotive, and perform crew violation analysis on the crew video data of the locomotive; of course, the locomotive may also indirectly communicate with the server through a third party, which may be, but is not limited to, other servers, communication base stations, satellites, etc.; the video data of the locomotive crew services and the operation monitoring record data can be exported through a storage device or a data line, and then the exported crew services video data is analyzed by the server for the violation of the crew services. The server may be implemented by an independent server or a server cluster composed of a plurality of servers.

In one embodiment, as shown in fig. 2, a method for analyzing a crew violation is provided, which is exemplified by the application of the method to the server in fig. 1, and includes the following steps:

s201, carrying out human body posture estimation processing on the crew video data to obtain a key point feature sequence; the key point feature sequence comprises: keypoint feature data for a plurality of video frames.

In this embodiment, the server may adopt a deep learning model to perform the human body posture estimation processing; the deep learning model may be: DensePose-RCNN (a regional convolutional neural network, i.e., mapping 2D image coordinates onto a 3D human body surface by deep learning, and processing dense coordinates at a speed of multiple frames per second to finally realize accurate positioning and pose estimation of a dynamic person), AlphaPose (an accurate multi-person pose estimation system), and the like, which are not limited by the embodiment.

The above-mentioned key points may include, but are not limited to: for each joint point, head, eye, nose, ear, etc. of the human body, the keypoint feature data may include, but is not limited to: the category of the key point, the position coordinate of the key point, the pixel characteristic of the key point, the image block corresponding to the position coordinate of the key point, and the like.

For the crew video data, performing human posture estimation processing, namely performing human posture estimation processing on all video frames or part of video frames in the crew video data respectively to obtain key point feature data of all video frames or part of video frames; thus, the keypoint signature sequence may include: and orderly arranging the key point characteristic data of the video frames according to the time sequence of the video frames. It should be noted that, the partial video frames may be obtained by sampling from all video frames, and the sampling method is not limited.

And S202, judging whether the crew violation exists or not according to the key point feature sequence and a preset crew violation standard library, and if so, outputting crew violation data.

The crew violation standard library can set the crew violation judgment standards corresponding to different service scenes respectively, can also comprise operation items, normal operation standards and the like, and can also set different violation judgment modes according to a value ride mode, a locomotive model, a traffic route, a locomotive traction mode, a transportation type and the like, and different crew violation judgment conditions corresponding to the different violation judgment modes. Optionally, the crew violation judgment condition corresponds to a preset monitoring area; for example, if the preset monitoring area is a cab, the traffic violation judgment condition is directed to a business process of a driver (a primary driver, a secondary driver, and the like) in the cab; for example, if the preset monitoring area is a rest room for a crew member in a passenger car, the crew member violation judgment condition is directed to the business process of the crew member in the passenger car.

For example, based on the judgment basis of the violation action provided by the key point feature sequence, the server may judge whether the violation action exists, for example, may judge whether the crewmember has a violation action such as dozing, leg raising, and chatting; the crew violation data may include at least one of: violation occurrence time, violation type, train number, locomotive information, driver information, speed information, violation description, violation video, and the like.

According to the analysis method for the violation of the crew service, the server can estimate and process the human posture of the crew service video data to obtain a key point characteristic sequence; and then, based on a preset crew violation standard library, judging the key point feature sequence, judging whether the key point feature sequence has a corresponding crew violation, and outputting crew violation data when the crew violation exists, namely, the intelligent analysis of the crew video through the server realizes the crew violation analysis, so that the coverage rate, accuracy and efficiency of the crew violation analysis can be improved, the labor cost is reduced, the driving safety is guaranteed, and a decision basis is provided for the standardized operation of crew members.

In an embodiment, referring to fig. 3, the embodiment relates to a process of performing human body posture estimation processing on a crew video data segment, and specifically may include:

s301, intercepting the crew video data into a plurality of video clips.

The duration of each video segment may be the same or different. Illustratively, the server may intercept the crew video data into a plurality of video clips of preset duration; for example, the preset time period may be 15 minutes.

And S302, carrying out human body posture estimation processing on each video clip to obtain a key point feature sequence of each video clip.

It can be understood that the crew video data is divided into a plurality of video segments to perform human posture estimation processing, the obtained key point feature sequence is the key point feature sequence of each video segment, correspondingly, the crew violation judgment is performed according to the key point feature sequence of each video segment, compared with the key point feature sequence of the whole crew video data, the time scale is smaller, the crew violation judgment is relatively more accurate and finer, and meanwhile, the requirements of the human posture estimation processing and the crew violation judgment on computing resources are reduced through segmentation processing.

In one embodiment, referring to fig. 4, S301 may include:

s401, obtaining locomotive operation monitoring record data corresponding to the crew service video data; the locomotive operation monitoring record data comprises time information of a service scene;

s402, intercepting a plurality of video clips with corresponding window lengths from the crew video data according to the time information of the service scene.

The locomotive operation monitoring recording data can be from a train operation monitoring recording device (LKJ), namely a monitoring device (LKJ), and the device can acquire and record various locomotive operation state information related to safe operation while realizing speed safety control; the locomotive operation monitoring record data comprises at least one of the following: value of the locomotive rides crew information, operational location information of the locomotive, a business scenario of the locomotive (alternatively referred to as a crew node). Wherein the traffic scenarios of the locomotives can be distinguished by at least one of the following categories: the system comprises the following steps of entering and exiting and running, double-value multiplying sections and single-value multiplying sections, passenger transportation, freight transportation and the like, and can also be divided into the following steps according to business time intervals: the method comprises the following service scenes of train receiving, in-section operation, out-section (station receiving), starting station, interval operation, station entering (station exiting), intermediate station stopping, final station, in-section (station handing over), train handing over and the like.

It will be appreciated that the locomotive operation monitoring record data and the crew video data correspond to each other in time; for example, the server may obtain the locomotive operation monitoring record data, and obtain the crew video data from the pickup time to the delivery time from the crew video database of the locomotive according to the pickup time and the delivery time in the locomotive operation monitoring record data. Of course, the server may also perform the analysis of the violation of the crew service by taking a preset time period as a unit, for example, acquiring the daily locomotive operation monitoring record data and the crew service video data, and performing the analysis of the violation of the crew service; for example, the predetermined time period may be daily, but it will be appreciated that the locomotive generates a significant amount of crew video data daily, but the server may still perform a comprehensive analysis of the crew video data.

The locomotive operation monitoring record data can provide service scene support when the illegal judgment of the crew service is carried out, such as: the method comprises the following steps that two persons are required to be carried out in a double-value multiplication section business scene, and hand ratio is required in an in-out business scene; the locomotive operation monitoring record data can also provide data support when the crew violation judgment is carried out, for example, the identity of a crew member who takes the ride can be known, so that the identity of the crew member corresponding to the crew violation can be determined, and the like.

Illustratively, a time sliding window mechanism can be adopted, and a plurality of video clips with different window lengths are intercepted in the crew service video data according to the time information of the service scene; the window length corresponds to each service scene in the operation monitoring record data. The server can acquire the starting time and the ending time of each service scene in the operation monitoring record data, and segments the crew service video data according to the starting time and the ending time of each service scene to obtain crew service video segments corresponding to each service scene; acquiring window lengths corresponding to the service scenes based on the corresponding relation between the preset service scenes and the window lengths; and respectively intercepting a plurality of video segments with the window length from the crew video segment corresponding to the service scene by adopting the window length corresponding to the service scene and the preset sliding window interval time aiming at the crew video segment corresponding to each service scene. The window sliding interval time may also correspond to each service scene in the operation monitoring record data, and may be smaller than, equal to, or longer than the window length. Of course, in the subsequent analysis of the crew violation, the server may perform the crew violation analysis by using the crew violation determining condition corresponding to the service scene for a plurality of video segments corresponding to different service scenes.

Of course, the server may also intercept a plurality of video segments with the window length from the crew video segment corresponding to the service scene by directly using the window length corresponding to each service scene and the preset sliding window interval time according to the start time and the end time of each service scene in the operation monitoring record data.

It is understood that the crew violation criteria library may include crew violation determination conditions corresponding to different business scenarios, where the locomotive operation monitoring record data provides the business scenario.

For example, in an inbound or outbound service scenario, a hand ratio is required to be performed, so in order to accurately identify whether a hand ratio action exists, the window length may correspond to the time of the hand ratio action, for example, 2 minutes, and the sliding window interval time may be 5 seconds; in an operation scene, for illegal actions such as dozing of a crew, cellphone-taking and the like, the window length may correspond to the time of the illegal action, for example, 1 minute, because the operation time is relatively long, the occurrence of crew violation events is relatively dispersed, and the window sliding interval time may be 1 minute; in an operation scene, for violations such as vehicle door opening, the window length may be 1 minute, and the window sliding interval may be 5 minutes. Of course, in the same service scene, a plurality of window lengths can be used to obtain video clips.

For example, for a certain locomotive, for a one-time direct service from the A place to the B place, the time corresponding to the locomotive running monitoring record data is 11: 30-14: 30, and then the video data of the crew service corresponding to the locomotive from 11: 30-14: 30 can be obtained; in brief, if 11: 30-11: 35 is an outbound service scene, 11: 35-14: 20 is an operating scene, and 14: 20-14: 30 is an inbound service scene, 37 video segments can be obtained by intercepting the train service video data of 11: 30-11: 35 with the window length of 2 minutes and the window sliding interval time of 5 seconds; intercepting 165 video segments from the crew video data with the window length of 1 minute and the sliding window interval time of 1 minute in 11: 35-14: 20; the method comprises the steps that 1 minute of window length and 5 minutes of window sliding interval time are obtained from the crew service video data of 11: 35-14: 20, and then 33 video clips are obtained through interception; the window length of the crew video data of 14: 20-14: 30 is 2 minutes, the sliding window interval time is 5 seconds, and 97 video segments are obtained by interception, namely 332 video segments are obtained in total.

It can be understood that different service scenes may correspond to different crew violation judgment conditions and corresponding violation actions, and therefore in this embodiment, the correlation between the keypoint feature sequence obtained by performing human posture estimation processing on the video clip with the window length of the corresponding service scene and the violation action is stronger, and when the keypoint feature sequence corresponds to the violation action, the violation judgment performed on the keypoint feature sequence by using the corresponding crew violation judgment condition is more accurate.

In an embodiment, the S302 may include: carrying out human body posture estimation processing on each video clip to obtain key point characteristic data of a plurality of video frames of each video clip; and tracking the key point characteristics of the video frames according to the key point characteristic data of the video frames of each video clip to obtain a key point characteristic sequence of each video clip.

Illustratively, the tracking algorithm may be a kalman filter algorithm, a particle filter algorithm, a mean shift algorithm (mean shift algorithm), and the like, and the tracking processing of the keypoints is implemented by matching the characteristics of the keypoints among the upper and lower frames. For example, the estimation value of the current video frame with higher confidence can be obtained by using the estimation value of the previous video frame and the observation value of the current video frame; the observed value of the current video frame may be the key point feature data of the current video frame, and the estimated value of the current video frame may be the key point tracking feature data of the current video frame. It will be appreciated that for the first video frame in a video segment, the estimate may be equal to the observation. For example, in this embodiment, the human body pose estimation processing may be performed on a part of video frames in each video segment to obtain the key point feature data of the part of video frames in each video segment, then, the tracking processing is performed on other video frames according to the key point feature data of the part of video frames to obtain the key point feature data of each video frame in each video segment, and the key point feature sequence of each video segment is obtained by stitching. For example, the partial video frame may be the first N video frames, N being a positive integer.

It can be understood that various complex situations may exist in the locomotive running process, which may cause many situations such as the movement of the crew member being blocked or missing, and may cause the missing of the key point feature data of the individual video frames; in this embodiment, the key point tracking feature sequence obtained based on the tracking of the key points can obtain complete key point tracking data, and the missing of the key point tracking feature data of individual video frames is avoided, so that the accuracy of the analysis of the crew violation can be further improved.

Also, in one embodiment, as illustrated with reference to FIG. 5, the keypoint feature data may include: the key point static feature data and the key point dynamic feature data, S302 may include:

s501, inputting each video clip into a preset depth regression network model for human body posture estimation processing, and outputting the key point static characteristics of a plurality of video frames in each video clip; the key point static features include: the relative position relationship between each key point category and each key point.

S502, calculating to obtain the dynamic characteristics of the key points of a plurality of video frames in each video clip according to the static characteristics of the key points of the plurality of video frames in each video clip and the interval duration of the video frames; the dynamic characteristics of the key points comprise: the moving speed of each key point and the relative angle change relationship of each key point.

And S503, combining the key point static characteristics and the key point dynamic characteristics of the plurality of video frames to obtain a key point characteristic sequence of each video clip.

The server can also adopt a depth regression network model to carry out human posture estimation processing, and the basic idea is to directly utilize a convolutional neural network to realize nonlinear mapping from an image to be processed to a human key point; the deep regression network model may be formed by cascading a plurality of convolutional neural networks, taking the image to be processed as an input of a first convolutional neural network, the output of the first convolutional neural network being a first position coordinate (initial position coordinate) of the key point, then acquiring an image block with a preset size corresponding to the first position coordinate of the key point from the image to be processed, and the image block is used as the input of a second convolution neural network, the output of the second convolution neural network is the second position coordinate of the key point, then obtaining an image block with a preset size corresponding to the second position coordinate of the key point from the image to be processed, taking the image block as the input of a third convolutional neural network, and by analogy, the final position coordinates of the key points output by the last neural network are obtained from the first neural network to the last neural network and are used as the key point feature data of the image to be processed. For different convolutional neural networks, the preset sizes of the image blocks may be the same or different. It can be understood that, in the deep regression network model formed by the cascaded convolutional neural networks, because the input of each convolutional neural network is more accurate, the more accurate positions of the key points can be obtained, and thus the types of the key points of each video frame in each video segment and the relative position relationship of each key point can be obtained. Wherein, the category of the key point may include: the relative position relationship of each key point can be actually expressed as the position of each key point relative to a world coordinate system or the relative distance and the relative angle between every two key points.

It can be understood that the server may calculate, for any two adjacent video frames, a variation of the positions of the corresponding key points in the two adjacent video frames, and calculate a change speed of the positions of the corresponding key points according to the variation and a time interval between the two adjacent video frames. For example, for a video frame ranked as 6 in a video clip, the amount of change in the position information of each key point in the video frame relative to its neighboring video frame (which may be an adjacent video frame, such as a video frame ranked as 7, or may be separated by multiple video frames, such as a video frame ranked as 10) may be calculated, and in combination with the time interval between the video frame and its neighboring video frame, the motion speed of each key point may be calculated as the dynamic feature of the key point of the video frame between the video frame and its neighboring video frame (such as a video frame ranked as 8).

In a word, the dynamic feature of the key point of each video frame can be obtained based on the static feature of the key point of each video frame, so that the feature sequence of the key point can be characterized by two angles, namely static and dynamic, and a more accurate analysis result can be obtained when the illegal crew service judgment is carried out based on richer multi-angle features.

Optionally, the step S302 may further include a process of normalizing the key point feature sequence, which may include: if the time length of the key point feature sequence is longer than the preset time length, performing down-sampling processing on the key point feature sequence on a time sequence to obtain a key point feature sequence after the down-sampling processing; the time length of the key point feature sequence after the down-sampling processing is equal to the preset time length; if the time length of the key point feature sequence is less than the preset time length, performing up-sampling processing on the key point feature sequence on a time sequence to obtain the key point feature sequence after the up-sampling processing; and the time length of the key point feature sequence subjected to the upsampling treatment is equal to the preset time length.

Because the time lengths of the video clips are inconsistent and the time lengths of the key point feature sequences are inconsistent, time length normalization needs to be carried out on the key point feature sequences; the time length normalization can solve the influence of the speed of action execution on illegal action recognition, and improve the accuracy of illegal action recognition. The specific method is to perform interpolation operation on the key point feature sequence in time sequence, and the key point feature sequence can be shortened and lengthened by respectively adopting the interpolation operation of down sampling and up sampling to obtain the key point feature sequence with standard length. When the time length of the key point feature sequence is longer than the standard time length, shortening the key point feature sequence by down-sampling interpolation; when the time length of the key point characteristic sequence is shorter than the standard time, the key point characteristic sequence is extended through an up-sampling interpolation value; the preset duration may be a standard duration. The up-sampling and the down-sampling are both to resample the key point characteristic sequence with time attribute, the number of the key point characteristics in the key point characteristic sequence obtained by resampling is compared with the original number of the key point characteristics in the key point characteristic sequence which is not sampled, the key point characteristic sequence which is more than the original number is called up-sampling, and the key point characteristic sequence which is less than the original number is called down-sampling.

In one embodiment, the above-mentioned crew violation analysis method may further include: performing image target identification processing on each video clip to obtain object characteristic data; accordingly, S202 may include: and judging whether the crew violation exists according to the key point feature sequence, the object feature data and a preset crew violation standard library so as to judge whether the crew violation exists more comprehensively and more accurately.

In this embodiment, the server may perform image target recognition processing on each video segment by using a deep convolutional network model to obtain object feature data of each video segment; the deep convolutional network model can be a classification network, for example, the front end is a feature extraction sub-network, the rear end is a classification sub-network, the classes of various objects in the object to be processed can be identified, and the features of the object can be extracted; for each video clip, the object type, the object characteristics and the like of each video frame of each video clip can be subjected to statistical processing to obtain the object characteristic data of each video clip; object feature data is feature data associated with an offending object and may include, but is not limited to: the object type, the position coordinates of the object, the pixel characteristics of the object, the image block corresponding to the position coordinates of the object, the number of objects, and the like.

For example, performing image target recognition processing on each video segment to obtain object feature data may include: inputting each video clip into a preset deep neural network model for image target recognition processing to obtain object characteristic data of each video clip; further, the method can also comprise the following steps: performing image target Recognition processing on each video segment by adopting an Optical Character Recognition algorithm (OCR) to obtain Character characteristic data, and adding the Character characteristic data into the object characteristic data to obtain new object characteristic data of each video segment; the character feature data includes at least one of: crew value data, train number data, locomotive speed data. Since various character-type video data may exist in the crew video data, for example, the current video time, the train number data, the crew value multiplier data, the locomotive speed data, and the like, the character-type video data in the video clip can be recognized by the OCR algorithm, the crew value multiplier data, the train number data, and the like can be recognized as auxiliary data for the analysis of the crew violation, and the locomotive speed data at different times can be recognized. Of course, the human body posture estimation processing and the image target recognition processing can be performed by using other machine learning models.

The object feature data may be used as a basis for determining whether or not there is an offending object (an object having an offending feature related to a vehicle occupancy violation, for example, a vehicle door satisfying a preset offending feature). Based on the judgment basis of the illegal object provided by the object characteristic data, the server can judge whether the illegal object exists or not; the server can perform joint judgment according to the illegal action and the illegal object in different service scenes based on the service violation judgment standards corresponding to different service scenes to judge whether the service violation exists. For example, for any traffic scenario, for a passenger traffic type (transportation type), if the number of objects in a human category in the cab is greater than two, then a ride violation of "passenger cab is greater than two" is determined to exist.

According to the analysis method for the duty violation, the server can acquire the key point feature sequence and the object feature data of the video clip aiming at each video clip. It can be understood that, in this embodiment, the key point feature sequences and the object feature data corresponding to the same video clip have temporal correspondence, so that the analysis of the crew violation can be more accurate by performing joint determination on the crew violation based on the key point feature sequences and the object feature data corresponding to the same video clip.

Specifically, the crew violation criterion library may include a crew violation judgment condition, and then judging whether a crew violation exists according to the key point feature sequence, the object feature data, and a preset crew violation criterion library may include: inputting the key point feature sequences of the video clips into a preset support vector machine classification model to classify the illegal actions, and outputting the illegal action types corresponding to the video clips; the violation action type corresponds to a preset violation judgment condition of the crew; and judging whether the crew violation exists or not according to the key point feature sequence, the violation action type, the object feature data and the crew violation judgment condition.

A Support Vector Machine (SVM) classification model, which is a generalized linear classifier for performing multi-element classification on data according to a supervised learning mode, wherein a decision boundary of the SVM classification model is a maximum edge distance hyperplane for solving learning samples; of course, the learning sample may be a plurality of key point feature sequences labeled with violating action types. Optionally, the keypoint feature sequence of each video segment may be input into the SVM, a probability value or a score that each keypoint feature sequence belongs to each violation action type is output, and the violation action type with the maximum probability value or score may be used as the violation action type corresponding to the video segment. Compared with a neural network model, the SVM is more suitable for classifying input dynamic features, and the neural network model is more suitable for classifying static features.

In one embodiment, the above-mentioned crew violation analysis method may further include: acquiring locomotive operation monitoring record data corresponding to the crew service video data; and judging whether the crew violation exists according to the operation monitoring record data, the key point characteristic sequence, the object characteristic data and a preset crew violation standard library.

As can be seen from the above description, in this embodiment, the server may perform the illegal crew service judgment related to the service scene in combination with the locomotive operation monitoring record data, the key point feature sequence, the object feature data and the preset illegal crew service standard library, and perform the illegal crew service judgment in combination with the key point feature sequence and the object feature data, thereby improving the accuracy of the illegal crew service judgment.

Alternatively, the above-mentioned crew violation determination conditions may be distinguished according to the type of data required for violation determination: a first type of crew violation determining condition, a second type of crew violation determining condition, a third type of crew violation determining condition, and a fourth type of crew violation determining condition; the first type of the passenger service violation judgment condition needs to be used for carrying out passenger service violation judgment by combining the key point feature sequence and the object feature data, the second type of the passenger service violation judgment condition needs to be used for carrying out passenger service violation judgment by combining the key point feature sequence, the third type of the passenger service violation judgment condition needs to be used for carrying out passenger service violation judgment by combining the object feature data, and the fourth type of the passenger service violation judgment condition needs to be used for carrying out passenger service violation judgment by combining the key point feature sequence and the locomotive operation monitoring record data.

The first type of crew violation determination condition may include: the crew member has a crew violation determination condition that uses the electronic device to act. Illustratively, the crew presence crew violation determination condition for using the electronic device action includes: whether the object characteristic data of the video clip comprises a human body and electronic equipment or not; if so, whether the position information of the electronic equipment is matched with the position information of the wrist joint point of the human body; if so, a ride violation is determined to exist in the video segment. The electronic device may be an electronic entertainment device such as a mobile phone and a game machine, or other electronic devices that affect the operation of the locomotive. Taking an electronic device as an example of a mobile phone, if the object feature data of the video clip includes: the method comprises the steps that a human body and a mobile phone are used, and the position information of the mobile phone is matched with the position information of a wrist joint point of the human body, so that the fact that the crew violation exists in a video clip is determined. The position information of the mobile phone is derived from the object feature data, the position information of the human wrist joint point is derived from the key point feature sequence, and the matching of the position information of the mobile phone and the position information of the human wrist joint point can ensure that the difference value of the position information of the mobile phone and the position information of the human wrist joint point is smaller than a preset threshold value, so that the accurate identification of the mobile phone duty violation can be realized.

Wherein the second type of crew violation determining condition comprises at least one of: the crew member has a crew violation judgment condition for leg-lifting action, a crew violation judgment condition for snooze action, and a crew violation judgment condition for boredom state. For example, in this embodiment, the server may further perform the following crew violation determination operation: if the violation action type is a leg lifting type of the crew, determining that the crew violation exists; when the included angle between the knee-hip straight line and the knee-ankle straight line of the human body is larger than a preset first angle threshold value, and the included angle between the knee-hip straight line and the preset horizontal direction of the human body is larger than a preset second angle threshold value, the illegal action type is a type that a crew lifts legs; the knee-hip straight line is a straight line formed by the knee joint points and the hip joint points, and the knee-ankle straight line is a straight line formed by the knee joint points and the ankle joint points; if the violation action type is the doze type of the crew member, determining that the crew member violation exists; when the included angle between the head-neck straight line and the neck-shoulder straight line of the human body is smaller than a preset third angle threshold, the illegal action type is a type that the crew dozes off; the neck-neck straight line is a straight line formed by the head key point and the neck joint point, and the neck-shoulder straight line is a straight line formed by the neck joint point and the shoulder joint point; if the violation action type is a crewman chatting type, determining that a crewman violation exists; when the position relation between the eyes and the ears of the human body is different from the preset position relation, the illegal action type is a crewmember chatting type; the preset position relation is as follows: the two eyes are positioned at one side of the two ears, or the two eyes are positioned between the two ears and are related to the position of the camera in the cab.

Wherein the third type of crew violation determining condition comprises at least one of: the method comprises the following steps that a crew service violation judgment condition of an off-seat action exists when the locomotive runs, a crew service violation judgment condition that the number of human bodies in a passenger cab is larger than a preset number, a crew service violation judgment condition that the locomotive is in a door opening state when the locomotive runs, and a crew service violation judgment condition that a camera is in a shielding state. Illustratively, the object characteristic data includes a speed of the locomotive and a position of the primary driver, and if the object characteristic data meets a condition for determining that a crew violation condition exists when the crew is away from the seat when the locomotive runs, that is, the speed of the locomotive is greater than a preset speed threshold, and a distance from the position of the primary driver to a preset primary driving area is greater than a preset distance threshold, it is determined that the crew violation exists. The object characteristic data comprises the number of human bodies in the cab, and if the object characteristic data meets the condition that the number of human bodies in the passenger cab is larger than the preset number of the illegal riding services, namely the number of human bodies in the cab is larger than the preset number, the illegal riding services are determined to exist. The object feature data includes: if the object characteristic data meets the condition of judging the violation of the ride service when the locomotive is in a door-opening state during running, namely the locomotive speed is greater than a preset speed threshold value, and the average value of the pixel values of the image frame area where the door is located is greater than a first preset pixel value or less than a second preset pixel value, determining that the violation of the ride service exists; the first predetermined pixel value represents the daytime environment outside the vehicle door, such as the gray value 245, and the second predetermined pixel value represents the nighttime environment outside the vehicle door, such as the gray value 10. The object feature data includes: and determining that the crew violation exists if the number of the edge features in the image frame meets the crew violation judgment condition that the camera is in a shielding state, namely the number of the edge features in the image frame is less than the preset number of the edge features.

Wherein the fourth type of crew violation determining condition comprises at least one of: the method comprises the following steps of determining the traffic violation condition that the number of human bodies in a double-value cab in a section is abnormal, determining the traffic violation condition that a crew does not act manually when an locomotive is in and out of a station, determining the traffic violation condition that the crew does not confirm a pantograph when the locomotive is parked, determining the traffic violation condition that the crew does not confirm a shunting signal when shunting, and determining the traffic violation condition that the crew does not stare at and control in a key operation link. For example, if the service scene is determined to be a binary riding section service scene through the locomotive operation monitoring recorded data, the object characteristic data includes the number of human bodies in the cab, and the object characteristic data meets the riding violation judgment condition that the number of human bodies in the binary riding section cab is abnormal, if the number of human bodies in the cab is not equal to two persons, the riding violation is determined. If the service scene is determined to be an in-out service scene through the locomotive operation monitoring record data, the object characteristic data comprises: the action type (which can be detected through image target recognition processing) of the crew member determines that the crew member violation exists if the crew member violation judgment condition that the crew member does not perform manual operation when the locomotive is at an entrance or exit station is met, namely the action type does not include the manual operation type. If the service scene is determined to be a parking scene through the locomotive operation monitoring record data, the object characteristic data comprises the head position of the primary driver and the position of a cab window area (which can be detected through image target identification processing), and if the condition for judging the duty violation of the pantograph when the crew stops is met, namely the head of the primary driver is not in the cab window area, the head of the primary driver does not extend out of the window, the primary driver does not confirm the pantograph, and the presence of the duty violation is determined. If the service scene is determined to be a shunting scene through the locomotive operation monitoring record data, the object characteristic data comprise the head position of the primary driver, the position of the secondary driver, the position of a cab window area and the position of an area outside a cab door, and the object characteristic data meet the illegal cab judgment condition that the shunting signal is not confirmed by the crew during shunting, namely the head of the primary driver is not in the cab window area and the secondary driver is not in the area outside the cab door, the situation means that the head of the primary driver does not extend out of the window and the secondary driver does not go out of the cab door, and the illegal cab shunting signal is determined to exist if the confirmation of the shunting signal is not executed. If the service scene is determined to be a key operation scene through the locomotive operation monitoring record data, the object characteristic data comprises the position of the primary driver and the position of the secondary driver (which can be detected through image target identification processing), and if the condition that the crew violation judgment condition that the crew does not stare at the key operation link is met, namely the distance between the primary driver and the secondary driver is greater than the preset staring distance threshold value, the secondary driver does not stand around the primary driver, and the existence of the crew violation is determined.

In one embodiment, the server may also post-process the crew violation data; the post-processing includes at least one of: displaying the illegal crew service item points (corresponding to various illegal action types, illegal object types and the like), recording, storing, comparing and analyzing, retrieving all illegal crew service data in the complete one-time crew service operation process, generating a illegal crew service report, auditing, and the like.

For example, the server may send the crew violation data to the audit terminal for auditing, receive the audit state sent by the audit terminal, and correct the crew violation data according to the audit state; the audit status may include: the audit is passed and the audit is not passed; for example, the corresponding violation type may be marked for the video segment that passes the audit, and the normal type may be marked for the video segment that fails the audit. Therefore, for the audited crew violation data, a video segment corresponding to the crew violation data can be called, and the video segment is used as a video sample to train the support vector machine classification model and the neural network classification model.

In one embodiment, the server may also count the crew violation data, generating a crew violation report, the crew violation report including at least one of: the number of times/frequency of crew violations corresponding to different types of crew violation peaks; the number/frequency of the service violations of different locomotives and different crew members; a retrieval index table of a complete one-time crew operation process; the retrieval index table comprises the service violation statistical results corresponding to different retrieval items, and the retrieval items comprise at least one of the following contents: time slot, locomotive, crew. For example, the server may perform comparative analysis on the crew violation data, for example, take time periods, locomotives, crew members, and the like as statistical granularity, perform statistics on the corresponding crew violation data, and perform statistics on the number/frequency of crew violations corresponding to different types of crew violation terms; the number/frequency of the violation times of the crew service of different time periods, different locomotives, different crew members and the like can be sequenced; counting all the crew violation data in the complete primary crew operation process, and generating a retrieval index table which takes time periods, locomotives, crew members and the like as retrieval items, for example, the work number of the crew member can be taken as the retrieval index, and all the crew violation data related to the crew member in the retrieval index table can be found; in addition, a crew violation report can be generated for the crew violation data, which can include but is not limited to the above statistics, analysis results of a essay, and the like; of course, the crew violation data, the statistical result, the analysis result of the comparison text, the crew violation report, and the like may also be recorded and stored, and the number/frequency of the crew violations corresponding to different types of crew violation nodes obtained through statistics may be sent to a preset display terminal for displaying.

In addition, it should be noted that the technical features of the above-mentioned method for performing the illegal crew service judgment, the segmentation processing into video segments, the video segment division according to the service scene, the tracking processing, the dynamic feature data of the key point, the normalization processing of the duration of the feature sequence of the key point, the illegal crew service judgment according to the feature data of the object and the like in combination with the locomotive operation monitoring record data may be combined with each other, and are not described herein again.

It should be noted that, in the image target recognition processing in this embodiment, the object features in the image may be extracted by using a deep neural network, and then the detection and classification of the object may be implemented by using an anchor frame mechanism, classification, and frame regression.

(1) Regional candidate network

Each convolutional layer feature (feature map) in the Network can also be used to predict candidate regions related to categories, but if a simple Network with a specially extracted candidate Region added in front is not elegant enough, the candidate Region extraction and Fast-RCNN (a target detection Network) part are finally merged to obtain a target detection Region candidate Network (RPN).

The target detection area candidate network takes an image (with any size) as an input, outputs a set of rectangular target suggestion boxes, each box has a target score, and a full convolution network can be adopted to build a model for the process. Since the final goal is to share the computation with the Fast R-CNN target detection network, it is assumed that the two networks share a series of convolutional layers; specifically, models of Zeiler and Fergus (ZF models) with 5 sharable convolutional layers and models of simony and Zisserman (VGG models) with 13 sharable convolutional layers may be used.

To generate the region candidate box, a small net is slid over the convolutional feature map output by the last shared convolutional layer, this net being fully connected to the nxn spatial window of the input convolutional feature map. Each sliding window maps to a low-dimensional vector (256-d dimensions for ZF and 512-d dimensions for VGG, one sliding window for each feature map for one value). This vector is output to two siblings of fully connected layers — bounding box regression layer (reg) and bounding box classification layer (cls). Where n is 3, where the effective field of view of the image is large (ZF is 171 pixels, VGG is 228 pixels); since the small network is in the form of a sliding window, the fully connected layers are shared by all spatial positions (meaning that the n × n layer parameters used to calculate the inner product for all positions are the same); this structure is implemented as an n × n convolutional layer followed by two 1 × 1 convolutional layers of the same rank (corresponding to reg and cls, respectively), with the ReLU applied to the output of the n × n convolutional layers.

(2) Anchor point mechanism

At the position of each sliding window, k region candidates are predicted simultaneously, so the reg layer has 4k outputs, i.e. coordinate encoding of k boxes. The cls layer outputs 2k scores, the estimated probability of being a target/non-target for each suggestion box (for simplicity, cls layers implemented with a class two softmax layer, k scores may also be generated with logistic regression). The k proposed boxes are parameterized by the corresponding k boxes called anchors. Each anchor is centered at the current sliding window center and corresponds to one scale and aspect ratio, using 3 scales and 3 aspect ratios, so that at each sliding position there are k-9 anchors. For a convolution feature map of size W × H (typical value about 2400), there are a total of W × H × k anchors. The method employed by the system has an important property, namely translation invariance, both for the anchor and for the function that computes the anchor's corresponding suggestion box.

(3) Problem of multiple scales

The problem in multi-scale detection is very important. In earlier solutions, an image pyramid was useful, the image being scaled to different sizes and the computational features of each size. This method is easy to consider and effective, but the weakness of this method is time consuming; another approach is to use a filter pyramid. The anchor-based method creates a series of anchors, so that the anchors can be used for classifying and degrading bounding boxes with multiple scales and aspect ratios, and images with one size and features with one size can be used; the calculation is performed based on the feature map, so the speed is very high.

An image is a mapping of a realistic 3D object in a two-dimensional plane, where depth information is lost. Even the same object may have a very different appearance. For example, from different perspectives of a photograph of an automobile, many different target contours may be obtained. In general, however, several representative profiles may be summarized, such as different categories of face, back, front, back, left, and right profiles. For each different category, the appearance of the outline contains unique information. However, the original anchor is set by hand with three scales and aspect ratios and does not change after detecting different objects; the structure can be further enhanced, the information extraction of fine-grained objects is enhanced, and the detection precision is improved.

(4) Similarity measure

The present application aims to select a representative bounding box by clustering the training data so that the fitting anchor is more adaptive. In order to realize the method, firstly, the similarity of the bounding boxes is defined, and the position information of the bounding boxes is ignored; two different rectangles can be compared with a ratio r of width to length and an area size s. By the two parameters r and s representing the rectangle, the aspect ratio of the scale and the shape of the object information can be more noticed, which is more practical in a fine-grained image; for example, in a fine-grained dataset of birds, the parameter r for most objects is less than 1, i.e. the width of the rectangle is shorter than the height; in the ship fine-grained image, the parameter r of most objects is greater than 1, namely the width of a rectangle is greater than the height; and s is a parameter that provides scaling information when determining the aspect ratio of the rectangle.

(5) Size normalization

The present application may use a pair of numbers (s, r) to represent a rectangular box. However, due to the different values of the dimensions, the different parameters have their distribution, requiring the parameters to be normalized; s is the area of the rectangular box; the object rectangular box pixel area may be from hundreds to hundreds of thousands, but r is typically between 0 and 20. To equally consider both factors, the parameters must be normalized to the same range. Otherwise, the change of r will be masked by the change of s during the calculation process; the size normalization was performed as follows:

where ms is the average value of s and mr is the average value of r. I S i is the variance of S and i R i is the variance of R. With both equations, the average of s and r is changed to zero and the ranges of both equations are normalized. After transformation, the properties of each rectangle are expressed using (| S |, | R |). During the analysis, it was found that several rectangular boxes with larger width and height may be larger than a few hundred times smaller, since the unit of s is the square of the pixel; in experiments, some points are orders of magnitude larger than most; this is because some special images, rather than they being marked as erroneous, are relatively large or the object is almost entirely the entire image, for example.

For these outliers, the present application uses the principle of normal distribution, and these points can be ignored; then, the (| S |, | R |) can be updated and then the bounding box is adapted. The application can adopt anchors of some adaptive objects; the classification of thousands of bounding box objects is the basic operation of cluster analysis, and the criteria is to make the individual distance as small as possible and the distance between different individual distances as large as possible, including but not limited to the K-means method, the CLARANS method, the BIRCH method, and other near 100 clustering methods. The present application may employ a modified version of the distance-based k-means method, referred to as k-means + +, to cluster rectangles.

(6) K-means + + clustering

The purpose of k-means is to find the cluster center that minimizes the class variance, which is the sum of the squared distances from each data point to its cluster center (closest to its center). Although the exact solution to the k-means problem of finding arbitrary inputs is NP-hard (so-called non-deterministic), there are standard methods, the k-means algorithm, that can broadly find an approximate solution. However, the k-means algorithm has a theoretical disadvantage, and the approximation it finds may be arbitrary for the objective function, as compared to optimal clustering. The k-means + + algorithm solves these obstacles by specifying a process to initialize the cluster center before performing a standard k-means optimization iteration, using k-means + + initialization, which is guaranteed to find a solution for o (logk) that competes with the optimal k-means solution; distributing k initial cluster centers is a good method: the first cluster center is randomly selected from the data points being clustered, and then each subsequent cluster center is selected from the remaining data points with a probability proportional to the squared distance of its nearest existing cluster center.

In the application, due to the adoption of the sliding window, the result of the final detection of the window has a large amount of overlapping and redundancy, so that the overlapped detection windows can be deleted by adopting a non-maximum suppression method, and the accurate position of the occurrence of the crew violation can be found.

Specifically, the deep learning network adopted in the present application is a convolutional neural network, and the design thereof often uses convolutional layers and pooling layers which are alternately arranged, and a fully-connected layer generally follows the last pooling layer, which is used for integrating global information of input data, taking a LeNet convolutional neural network as an example.

The convolution layer contains convolution kernels and convolution operation, and a plurality of feature maps are obtained by performing convolution operation on each position of input data through a plurality of convolution kernels. For image data I of a single channel, a two-dimensional data image input to a convolutional neural network is converted into three-dimensional data in a feature map (feature maps) output for each layer in the network, and the size of the three-dimensional data can be represented as w × h × d, w is the width of the feature map, h is the height of the feature map, and d is the number of feature map channels. First, assuming that the size of input image data I is W × H, W and H are spatial sizes, respectively, a convolution kernel K^lIs C × D, where C is the number of convolution kernels and D is the size of the convolution kernel. Characteristic diagram FM corresponding to the convolution kernel^lThe value at position (i, j) is calculated as:

let the ith channel of the l-1 layer characteristic diagram be

The jth convolution kernel of the ith characteristic diagram of the ith layer of the CNN acting on the ith layer-1 is

Then the characteristics of the jth channel of the ith layer output profile can be expressed as:

wherein, the operation of convolution is carried out,

is the bias term.

For an operation on the entire convolutional layer, the convolutional kernel performs sliding convolution on the input data, samples are taken at intervals of s, and this operation is called stride (stride) and is denoted as s. In addition, in order to output a feature map of a fixed size after convolution, the size of the input layer feature map needs to be changed, and it is common practice to fill the edge with 0 or 1 so that the center of the convolution kernel can slide to each position. This is called padding (pad) and is denoted as p. For example, assuming that p is 1 and s is 1, the convolution kernel can be:

detailed procedure of convolution operation of convolutional layer: the convolutional layer firstly fills the input characteristic graph at the periphery, then moves a convolutional kernel according to the step length, then carries out convolution operation, and finally outputs a convolution result.

The pooling layer is usually located after the convolutional layer, and its input is the feature map output from the previous layer, and the main function of the pooling layer is to reduce the network parameters while preventing the network from being overfit. When input data has a small amount of translation, the output result can not be changed by the pooling operation in the convolutional neural network, and the convolutional neural network has certain translation invariance due to the operation. In the present phase, there are three most pooling modes, namely maximum pooling, minimum pooling and average pooling, which are all the overall statistical characteristics of a certain location within the neighborhood range of the location, for example, the maximum pooling gives the maximum value within the neighborhood range.

Taking the maximum pooling as an example, a feature map is output on each channel of the input layer. The local area acted by the pooling operation is of size D × D and span s, then the operation expression of the maximum pooling is:

FM(c，i，j)＝max({I(c，si+m，sj+n)|1≤m，n≤D})

the convolution layer and the pooling layer are continuously alternated, so that the CNN can extract the high-level semantic features of the image layer by layer from bottom to top. After the alternation of the multiple layers is finished, the fully-connected layer integrates the extracted features of all the previous steps so as to classify the violation. And the last layer is a classification layer, the number of the nodes is the number of the illegal behaviors plus 1, wherein 1 is a normal behavior without violation. In the using process, the audited non-violation videos serving as normal categories can be added into a convolutional neural network for training, and the recognition accuracy rate of the crew violation is improved.

It should be understood that although the various steps in the flow charts of fig. 2-5 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2-5 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternating with other steps or at least some of the sub-steps or stages of other steps.

In one embodiment, as shown in fig. 6, there is provided a crew violation analysis device, comprising: an attitude feature extraction module 61 and a violation judgment module 62, wherein: the attitude feature extraction module 61 is used for estimating the human body attitude of the crew video data to obtain a key point feature sequence; the key point feature sequence comprises: keypoint feature data for a plurality of video frames; and the violation judging module 62 is configured to judge whether a crew violation exists according to the key point feature sequence and a preset crew violation standard library, and if so, output crew violation data.

In one embodiment, the pose feature extraction module 61 includes: an intercepting unit for intercepting the crew video data into a plurality of video clips; and the attitude feature extraction unit is used for estimating the human body attitude of each video clip to obtain the key point feature sequence of each video clip.

In one embodiment, the intercepting unit is specifically configured to obtain locomotive operation monitoring record data corresponding to the crew video data; the locomotive operation monitoring record data comprises time information of a service scene; and intercepting a plurality of video clips with corresponding window lengths in the crew video data according to the time information of the service scene.

In one embodiment, the traffic scenario includes at least one of inbound and outbound, double-value-by-section, single-value-by-section, and crew pickup.

In one embodiment, the gesture feature extraction unit is specifically configured to perform human gesture estimation processing on each video clip to obtain key point feature data of a plurality of video frames of each video clip; and tracking the key point characteristics of the video frames according to the key point characteristic data of the video frames of each video clip to obtain a key point characteristic sequence of each video clip.

In one embodiment, the keypoint feature data comprises: the gesture feature extraction unit is specifically used for inputting each video clip into a preset depth regression network model for human gesture estimation processing and outputting the key point static features of a plurality of video frames in each video clip; the key point static features include: the relative position relationship between each key point category and each key point; calculating to obtain the dynamic characteristics of the key points of the video frames in each video clip according to the static characteristics of the key points of the video frames in each video clip and the interval duration of the video frames; the dynamic characteristics of the key points comprise: the motion speed of each key point and the relative angle change relationship of each key point; and combining the static characteristics and the dynamic characteristics of the key points of the plurality of video frames to obtain the key point characteristic sequence of each video clip.

In one embodiment, the pose feature extraction unit is further configured to perform downsampling on the key point feature sequence in a time sequence if the duration of the key point feature sequence is greater than a preset duration, so as to obtain a downsampled key point feature sequence; the time length of the key point feature sequence after the down-sampling processing is equal to the preset time length; if the time length of the key point feature sequence is less than the preset time length, performing up-sampling processing on the key point feature sequence on a time sequence to obtain the key point feature sequence after the up-sampling processing; and the time length of the key point feature sequence subjected to the upsampling treatment is equal to the preset time length.

In one embodiment, the method further comprises the following steps: the object identification module is used for carrying out image target identification processing on each video clip to obtain object characteristic data; the violation determination module 62 may include: and the violation judging unit is used for judging whether the crew violation exists according to the key point feature sequence, the object feature data and a preset crew violation standard library.

In one embodiment, the crew violation criteria library includes preset crew violation determining conditions; the violation judging unit is specifically configured to input the key point feature sequences of the video segments into a preset support vector machine classification model to perform violation action classification processing, and output a violation action type corresponding to each video segment; the violation action type corresponds to the crew violation judgment condition; and judging whether the crew violation exists or not according to the key point feature sequence, the violation action type, the object feature data and the crew violation judgment condition.

In one embodiment, the method further comprises the following steps: the operation data acquisition module is used for acquiring locomotive operation monitoring record data corresponding to the crew service video data; the violation judging unit is used for judging whether the service violation exists or not according to the operation monitoring record data, the key point feature sequence, the object feature data and a preset service violation standard library.

In one embodiment, the crew violation criteria library includes a preset first type of crew violation determining condition; the first type of crew violation determining condition includes: the crew member has a crew violation determination condition that uses the electronic device to act.

In one embodiment, the presence of a crew violation determination condition using an electronic device action by a crew member comprises: whether the object characteristic data of the video clip comprises a human body and electronic equipment or not; if so, whether the position information of the electronic equipment is matched with the position information of the wrist joint point of the human body; if so, a ride violation is determined to exist in the video segment.

In one embodiment, the crew violation criteria library includes a second predetermined type of crew violation determining condition; a second type of crew violation determining condition comprising at least one of: the crew member has a crew violation judgment condition for leg-lifting action, a crew violation judgment condition for snooze action, and a crew violation judgment condition for boredom state.

In one embodiment, the method further comprises the following steps: the auditing module is used for sending the crew violation data to the auditing terminal for auditing; receiving an audit state sent by an audit terminal aiming at each crew violation item in the crew violation data; the audit state includes: the audit is passed and the audit is not passed; and correcting the crew violation data according to the audit state.

In one embodiment, the method further comprises the following steps: a statistics report module, configured to perform statistics on the crew violation data and generate a crew violation report, where the crew violation report includes at least one of the following: the number of times/frequency of crew violations corresponding to different types of crew violation peaks; the number/frequency of the service violations of different locomotives and different crew members; a retrieval index table of a complete one-time crew operation process; the retrieval index table comprises the service violation statistical results corresponding to different retrieval items, and the retrieval items comprise at least one of the following contents: time slot, locomotive, crew.

For specific limitations of the crew violation analysis device, reference may be made to the above limitations of the crew violation analysis method, which are not described herein again. The various modules in the above-described crew violation analysis device may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent of a processor in the server, and can also be stored in a memory in the server in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a crew violation analysis server is provided, comprising a memory and a processor, the memory storing a computer program, the processor when executing the computer program further implementing the steps of: carrying out human body posture estimation processing on the crew video data to obtain a key point feature sequence; the key point feature sequence comprises: keypoint feature data for a plurality of video frames; and judging whether the crew violation exists or not according to the key point feature sequence and a preset crew violation standard library, and if so, outputting crew violation data.

In one embodiment, referring to fig. 7, a crew violation analysis system is provided that includes at least one task scheduling server 71 and a plurality of crew violation analysis servers 72; the task scheduling server is connected with the multiple crew violation analysis servers and is used for receiving the crew video data to be processed and distributing the crew video data to the multiple crew violation analysis servers for crew violation analysis; and acquiring the crew violation data output by the multiple crew violation analysis servers, and summarizing.

The task scheduling server and the crew violation analysis server may adopt a GPU (Graphics Processing Unit) server, and may perform parallel Processing. The crew violation analysis system can be deployed on a plurality of special GPU servers, a uniform coding standard is formulated, a cluster technology (cluster) is adopted, the cluster technology is realized through a software architecture and a hardware architecture, and the crew violation analysis system can comprise a plurality of video file calculation processing servers and is specifically configured according to the running number of locomotives; the system can comprise 1 task scheduling server and a plurality of crew violation analysis servers; the task scheduling server can adopt double E5 series CPUs (central processing units) to improve the task scheduling processing speed; and the task scheduling server can be deployed with task scheduling software which is used for distributing a large amount of videos to be processed to different crew violation analysis servers for analysis and summarizing and analyzing the obtained result data.

In one embodiment, the task scheduling server is further configured to receive a crew violation auditing request sent by an auditing terminal; returning the crew violation data corresponding to the crew violation data request to the auditing terminal; and receiving the audit data sent by the audit terminal, and revising the crew violation data according to the audit data.

In one embodiment, the task scheduling server is further configured to receive a service violation reading request sent by the reading terminal; returning the illegal crew service data corresponding to the illegal crew service reading request to the reading terminal; the crew violation data comprises at least one of: violation occurrence time, violation type, train number, locomotive information, driver information, speed information, auditor information, audit state, violation description and violation video. The browsing terminal can call violation information and violation videos through a client or a Web end.

Those skilled in the art will appreciate that the configuration shown in fig. 7 is a block diagram of only a portion of the configuration associated with the present application and does not constitute a limitation on the devices to which the present application may be applied, and that a particular device may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a readable storage medium is provided, having stored thereon a computer program which, when executed by a processor, further performs the steps of: carrying out human body posture estimation processing on the crew video data to obtain a key point feature sequence; the key point feature sequence comprises: keypoint feature data for a plurality of video frames; and judging whether the crew violation exists or not according to the key point feature sequence and a preset crew violation standard library, and if so, outputting crew violation data.

In addition, the following illustrates the crew violation analysis system of the present embodiment, and a schematic diagram of various crew violations; it should be noted that the present embodiment may be applied to monitoring other crew service scenarios besides trains or other types of crew service violations. Fig. 8a is a schematic diagram of a plurality of GPU servers employed by the crew violation analysis system; as shown in fig. 8b, the crew violation analysis system can monitor the 6A video file in the dump path in real time, automatically analyze the newly generated 6A video file for crew violation, and generate violation information and violation video; related personnel only need to call violation information and violation videos through the client or the Web; the figure shows key point characteristics of a certain video frame, such as a connecting line schematic diagram of a joint point corresponding to a primary driver (crew member) in a rectangular box (detection box) in the figure; as shown in fig. 8c, the crew violation analysis system can show the current day violation conditions of each workshop and each violation item in a graphical manner, and perform comparative analysis and proportion analysis on the violation behaviors of the crew; the trend analysis of violation items of each workshop is realized according to conditions such as day, time period, month, year and the like, and the violation details of each workshop and each violation item can be directly called; as shown in fig. 8d, the crew violation analysis system can search, identify and count all violation behaviors in a complete one-time crew operation process of crew taking, intra-segment operation, departure (standing and taking), starting station, interval operation, arrival (leaving), intermediate station stop, terminal station, arrival (standing and handing over) and vehicle handing over, and can call a corresponding violation video by clicking violation information details; the crew violation analysis system can call corresponding violation videos through violation information, can directly inquire all violation videos in the crew value-taking process of each time, and provides functions of accurate inquiry, audit, hidden list, displayed list, pause playing, continuous playing, video replay, video stop, previous page, next page, full video screen and the like for the videos; fig. 8e-8m are schematic diagrams of different types of crew violations, respectively, including: the passenger cab is larger than two persons (as shown in fig. 8e), the crew lifts legs (as shown in fig. 8f), leaves during operation (as shown in fig. 8g), plays a mobile phone and electronic equipment (as shown in fig. 8h), twists about head and chats, realizes intermittent watching (as shown in fig. 8i), opens a door during operation (as shown in fig. 8j), shields a camera (as shown in fig. 8k), sleeps (as shown in fig. 8l), does not have a hand ratio between an entry station and an exit station (as shown in fig. 8m), does not carry out two-person co-riding in a double-value riding section (as shown in fig. 8n), and the like; as shown in fig. 8o and 8p, the crew violation analysis system may count the number of violations in each workshop on the condition of date, time period, month, and type of crew violation (violation item), may select each quantity of violations in each workshop to call the corresponding violation details, call the violation video file according to the violation details, and compare and analyze the violation conditions in each workshop with the graph; as shown in fig. 8q, the crew violation analysis system can record and store all processed video files, facilitate the user to query at any time, and provide backup operations such as export and printing.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features. The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A method for analysis of a crew violation, the method comprising:

2. The method according to claim 1, wherein the performing human pose estimation on the crew video data to obtain a key point feature sequence comprises:

intercepting the crew video data into a plurality of video clips;

3. The method of claim 2, wherein said truncating the crew video data into a plurality of video segments comprises:

4. The method of claim 3, wherein the traffic scenario includes at least one of inbound and outbound, double value block, single value block, crew pickup.

5. The method according to claim 2, wherein the performing the human body pose estimation process on each video segment to obtain the key point feature sequence of each video segment comprises:

6. The method of claim 2, wherein the keypoint feature data comprises: the method for obtaining the key point characteristic sequence of each video clip by carrying out human body posture estimation processing on each video clip comprises the following steps:

7. The method according to claim 2, wherein the performing human body pose estimation processing on each video segment to obtain a key point feature sequence of each video segment further comprises:

8. The method according to any one of claims 2-7, further comprising:

9. The method according to claim 8, wherein the crew violation criteria library comprises preset crew violation determining conditions; the judging whether the crew violation exists according to the key point feature sequence, the object feature data and a preset crew violation standard library comprises the following steps:

10. The method of claim 8, further comprising: acquiring locomotive operation monitoring record data corresponding to the crew service video data;

11. The method according to claim 8, wherein the library of crew violation criteria comprises a preset first type of crew violation determining condition; the first type of crew violation determining condition comprises: the crew member has a crew violation determination condition that uses the electronic device to act.

12. The method according to claim 11, wherein the presence of a crew violation determination condition for an action using an electronic device by the crew comprises:

if so, it is determined that a ride violation exists in the video segment.

13. The method according to any one of claims 1-7, wherein the library of crew violation criteria comprises a preset second type of crew violation determining condition; the second type of crew violation determining condition comprises at least one of:

14. The method of claim 1, further comprising:

sending the crew violation data to an auditing terminal for auditing;

and correcting the crew violation data according to the audit state.

15. The method according to claim 1, further comprising accounting for crew violation data, generating a crew violation report, the crew violation report comprising at least one of: