CN117462146A - Human brain abnormal discharge detection method, device, storage medium and electronic equipment - Google Patents

Abstract

The disclosed embodiments relate to a human brain abnormal discharge detection method, device, storage medium and electronic equipment, and relate to the fields of artificial intelligence and multi-modal technology. The method includes: obtaining the biomedical characteristic data of the tested object; obtaining the video monitoring data of the tested object; detecting the action information of the tested object according to the video monitoring data; extracting user information from the video monitoring data. For an image sequence of interest that characterizes the action of the subject under test; using a pre-trained abnormal discharge detection model to process the biomedical feature data, the action information, and the image sequence of interest to obtain the Abnormal discharge detection results of the object under test. The present disclosure can realize automated detection of abnormal human brain discharge and improve detection accuracy.

Description

Human brain abnormal discharge detection method, device, storage medium and electronic equipment

Technical field

Embodiments of the present disclosure relate to the fields of artificial intelligence and multi-modal technologies. More specifically, embodiments of the present disclosure relate to a human brain abnormal discharge detection method, a human brain abnormal discharge detection device, a computer-readable storage medium and an electronic device.

Background technique

This section is intended to provide background or context for the embodiments of the disclosure set forth in the claims, and the description herein is not an admission of prior art by inclusion in this section.

Abnormal discharges in the human brain are caused by abnormal activity of brain cells and can cause neurological diseases such as epilepsy. Detecting abnormal discharges in the human brain through examination methods such as electroencephalography can help doctors identify patients' diseases. For example, the detection of epileptiform discharges in electroencephalogram is one of the important criteria for the diagnosis of epilepsy.

The detection of abnormal human brain discharge involves detection data such as electroencephalogram. The work of data detection and analysis is heavy and consumes a lot of manpower and time costs. Therefore, the industry's demand for automated detection of abnormal human brain discharges is increasing.

Contents of the invention

However, the current accuracy of abnormal human brain discharge detection needs to be improved.

In related technologies, artificial intelligence technology is used to detect abnormal discharges in the human brain, such as training an intelligent network for epileptiform discharge recognition based on deep learning to detect epileptiform discharges in the human brain. Relevant literature reports that its accuracy rate only reaches 70%, requiring professional doctors and technicians to post-process the artificial intelligence reading results.

For this reason, there is a great need for an improved method for detecting abnormal human brain discharges, which can realize automated detection of abnormal human brain discharges and improve detection accuracy.

In this context, embodiments of the present disclosure are expected to provide a human brain abnormal discharge detection method, a human brain abnormal discharge detection device, a computer-readable storage medium, and an electronic device.

According to a first aspect of the present disclosure, a method for detecting abnormal human brain discharge is provided, including: obtaining biomedical characteristic data of a subject; obtaining video monitoring data of the subject; detecting the abnormal discharge according to the video monitoring data Action information of the subject under test; extracting an image sequence of interest for characterizing the action of the subject under test from the video monitoring data; using a pre-trained abnormal discharge detection model to analyze the biomedical feature data, the The action information and the image sequence of interest are processed to obtain the abnormal discharge detection result of the subject under test.

In one embodiment, obtaining the biomedical characteristic data of the subject includes: obtaining the biomedical monitoring data of the subject collected by biomedical monitoring equipment; and preprocessing the biomedical monitoring data. ; Obtain the biomedical characteristic data based on the preprocessed biomedical monitoring data.

In one embodiment, the biomedical monitoring data includes: multi-channel EEG monitoring data collected from multiple parts of the scalp of the subject; the obtained data are obtained based on the preprocessed biomedical monitoring data. The biomedical characteristic data includes: calculating the potential difference between each channel and the reference electrode from the preprocessed multi-channel EEG monitoring data to obtain initial characteristic data of the EEG signal; extracting the said EEG signal according to the initial characteristic data Biomedical characterization data.

In one embodiment, the biomedical feature data includes EEG signal waveform features; and extracting the biomedical feature data based on the EEG initial feature data includes: using a pre-trained waveform feature extraction model to The initial characteristic data of the EEG signal is processed to extract the waveform characteristics of the EEG signal.

In one embodiment, the biomedical feature data includes time-frequency features of the EEG signal; and extracting the biomedical feature data based on the initial feature data of the EEG signal includes: analyzing the initial feature data of the EEG signal. The data is subjected to time-frequency transformation to obtain the EEG signal time-frequency data corresponding to the EEG signal initial characteristic data; the EEG signal time-frequency characteristics are extracted according to the EEG signal time-frequency data.

In one embodiment, the preprocessing of the biomedical monitoring data includes at least one of the following processes: resampling, filtering, removing noise data, and numerical standardization.

In one implementation, the video monitoring data includes face video monitoring data; and detecting the action information of the subject under test based on the video monitoring data includes: detecting people from the face video monitoring data. Face key point data; obtain facial action information of the subject under test based on the face key point data.

In one implementation, the image sequence of interest includes a face image sequence; and extracting the image sequence of interest from the video monitoring data for characterizing the movement of the subject under test includes: according to The face key point data is used to cut out face area images from multiple frames of the face video monitoring data to obtain a face image sequence.

In one embodiment, obtaining the facial action information of the subject according to the facial key point data includes: determining the facial key points where movement occurs and their movement based on the facial key point data. Displacement information is used to obtain facial movement information of the subject under test.

In one embodiment, the video monitoring data includes body video monitoring data; and detecting the action information of the subject according to the video monitoring data includes: detecting body key points from the body video monitoring data. Data; obtain the body movement information of the subject according to the body key point data.

In one embodiment, the image sequence of interest includes a body image sequence; and extracting the image sequence of interest from the video monitoring data for characterizing the movement of the subject under test includes: according to the Using the body key point data, body region images are cut out from multiple frames of the body monitoring video data to obtain a body image sequence.

In one embodiment, obtaining the body movement information of the subject according to the body key point data includes: determining the body key points where movement occurs and their displacement information based on the body key point data, to obtain The body movement information of the subject under test.

In one embodiment, the abnormal discharge detection model includes a feature processing layer, an attention layer, and a classification layer; the pre-trained abnormal discharge detection model is used to analyze the biomedical feature data, the action information, and the Processing the image sequence of interest to obtain the abnormal discharge detection result of the subject, including: inputting the biomedical feature data, the action information, and the image sequence of interest into the abnormal discharge detection model; using the The feature processing layer extracts action feature data from the action information, extracts image feature data from the image sequence of interest, and fuses the biomedical feature data, the action feature data, and the image feature data to obtain Fusion features; use the attention layer to characterize the fusion features to obtain embedded features; use the classification layer to map the embedded features to the output space to obtain abnormal discharge detection results of the tested object.

In one embodiment, before using the pre-trained abnormal discharge detection model to process the biomedical feature data, the action information, and the image sequence of interest, the method further includes: At least one of motion information and the image sequence of interest is time aligned with the biomedical feature data.

In one embodiment, time-aligning at least one of the action information and the image sequence of interest with the biomedical feature data includes: detecting a or multiple biomedical characteristic data of suspected abnormal discharge and the corresponding one or more first time points; detect one or more action data of suspected abnormal discharge and the corresponding one or more second time points from the action information point; match the biomedical characteristic data of the suspected abnormal discharge with the action data of the suspected abnormal discharge, and determine the corresponding relationship between the first time point and the second time point according to the matching results; based on the The corresponding relationship between the first time point and the second time point is determined to determine a time calibration parameter, and the action information and the biomedical feature data are time aligned using the time calibration parameter.

In one embodiment, matching the biomedical characteristic data of the suspected abnormal discharge with the action data of the suspected abnormal discharge includes: determining whether the biomedical characteristic data of the suspected abnormal discharge is consistent with other biological data. The first relative value between the medical feature data; determining the second relative value between the action data of the suspected abnormal discharge and other action data in the action information; by comparing the first relative value with the third Two relative values are used to obtain a matching result between the biomedical characteristic data of the suspected abnormal discharge and the action data of the suspected abnormal discharge.

According to a second aspect of the present disclosure, a human brain abnormal discharge detection device is provided, including: a first acquisition module configured to acquire biomedical characteristic data of a subject; a second acquisition module configured to acquire the subject video monitoring data of the measured object; an action information detection module configured to detect the action information of the measured object according to the video monitoring data; an image sequence extraction module configured to extract from the video monitoring data for An image sequence of interest that characterizes the movement of the subject under test; a model processing module configured to use a pre-trained abnormal discharge detection model to perform processing on the biomedical feature data, the action information, and the image sequence of interest. Process to obtain the abnormal discharge detection result of the object under test.

In one embodiment, obtaining the biomedical characteristic data of the subject includes: obtaining the biomedical monitoring data of the subject collected by biomedical monitoring equipment; and preprocessing the biomedical monitoring data. ; Obtain the biomedical characteristic data based on the preprocessed biomedical monitoring data.

In one embodiment, the biomedical monitoring data includes: multi-channel EEG monitoring data collected from multiple parts of the scalp of the subject; the obtained data are obtained based on the preprocessed biomedical monitoring data. The biomedical characteristic data includes: calculating the potential difference between each channel and the reference electrode from the preprocessed multi-channel EEG monitoring data to obtain initial characteristic data of the EEG signal; extracting the said EEG signal according to the initial characteristic data Biomedical characterization data.

In one embodiment, the biomedical feature data includes EEG signal waveform features; and extracting the biomedical feature data based on the EEG initial feature data includes: using a pre-trained waveform feature extraction model to The initial characteristic data of the EEG signal is processed to extract the waveform characteristics of the EEG signal.

In one embodiment, the biomedical feature data includes time-frequency features of the EEG signal; and extracting the biomedical feature data based on the initial feature data of the EEG signal includes: analyzing the initial feature data of the EEG signal. The data is subjected to time-frequency transformation to obtain the EEG signal time-frequency data corresponding to the EEG signal initial characteristic data; the EEG signal time-frequency characteristics are extracted according to the EEG signal time-frequency data.

In one embodiment, the preprocessing of the biomedical monitoring data includes at least one of the following processes: resampling, filtering, removing noise data, and numerical standardization.

In one implementation, the video monitoring data includes face video monitoring data; and detecting the action information of the subject under test based on the video monitoring data includes: detecting people from the face video monitoring data. Face key point data; obtain facial action information of the subject under test based on the face key point data.

In one implementation, the image sequence of interest includes a face image sequence; and extracting the image sequence of interest from the video monitoring data for characterizing the movement of the subject under test includes: according to The face key point data is used to cut out face area images from multiple frames of the face video monitoring data to obtain a face image sequence.

In one embodiment, obtaining the facial action information of the subject according to the facial key point data includes: determining the facial key points where movement occurs and their movement based on the facial key point data. Displacement information is used to obtain facial movement information of the subject under test.

In one embodiment, the video monitoring data includes body video monitoring data; and detecting the action information of the subject according to the video monitoring data includes: detecting body key points from the body video monitoring data. Data; obtain the body movement information of the subject according to the body key point data.

In one embodiment, the image sequence of interest includes a body image sequence; and extracting the image sequence of interest from the video monitoring data for characterizing the movement of the subject under test includes: according to the Using the body key point data, body region images are cut out from multiple frames of the body monitoring video data to obtain a body image sequence.

In one embodiment, obtaining the body movement information of the subject according to the body key point data includes: determining the body key points where movement occurs and their displacement information based on the body key point data, to obtain The body movement information of the subject under test.

In one embodiment, the abnormal discharge detection model includes a feature processing layer, an attention layer, and a classification layer; the pre-trained abnormal discharge detection model is used to analyze the biomedical feature data, the action information, and the Processing the image sequence of interest to obtain the abnormal discharge detection result of the subject, including: inputting the biomedical feature data, the action information, and the image sequence of interest into the abnormal discharge detection model; using the The feature processing layer extracts action feature data from the action information, extracts image feature data from the image sequence of interest, and fuses the biomedical feature data, the action feature data, and the image feature data to obtain Fusion features; use the attention layer to characterize the fusion features to obtain embedded features; use the classification layer to map the embedded features to the output space to obtain abnormal discharge detection results of the tested object.

In one embodiment, the model processing module is further configured to: process the biomedical feature data, the action information, and the image sequence of interest using the pre-trained abnormal discharge detection model. Previously, at least one of the action information and the image sequence of interest is time-aligned with the biomedical feature data.

In one embodiment, time-aligning at least one of the action information and the image sequence of interest with the biomedical feature data includes: detecting a or multiple biomedical characteristic data of suspected abnormal discharge and the corresponding one or more first time points; detect one or more action data of suspected abnormal discharge and the corresponding one or more second time points from the action information point; match the biomedical characteristic data of the suspected abnormal discharge with the action data of the suspected abnormal discharge, and determine the corresponding relationship between the first time point and the second time point according to the matching results; based on the The corresponding relationship between the first time point and the second time point is determined to determine a time calibration parameter, and the action information and the biomedical feature data are time aligned using the time calibration parameter.

In one embodiment, matching the biomedical characteristic data of the suspected abnormal discharge with the action data of the suspected abnormal discharge includes: determining whether the biomedical characteristic data of the suspected abnormal discharge is consistent with other biological data. The first relative value between the medical feature data; determining the second relative value between the action data of the suspected abnormal discharge and other action data in the action information; by comparing the first relative value with the third Two relative values are used to obtain a matching result between the biomedical characteristic data of the suspected abnormal discharge and the action data of the suspected abnormal discharge.

According to a third aspect of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the human brain abnormal discharge detection method of the first aspect and its possible implementation are implemented. Way.

According to a fourth aspect of the present disclosure, an electronic device is provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the operation via executing the executable instructions. Implement the above-mentioned first aspect of the human brain abnormal discharge detection method and its possible implementation.

In the solution of the present disclosure, on the one hand, biomedical feature data and video monitoring data are obtained, and the video monitoring data is processed to obtain the action information of the subject under test and the image sequence of interest. By combining the biomedical feature data, action information, and sensory information, The three types of data of interest image sequence can be used to detect abnormal human brain discharges, which can improve the accuracy of detection results. Different types of data can make up for each other's lack of information. For example, action information or interest image sequence can make up for the lack of information that cannot be reflected in biomedical feature data. The information ensures the stability of the test results, reduces misjudgments, and does not require manual reading and other human processing throughout the process, realizing automated detection of abnormal human brain discharges. On the other hand, through the processing of video monitoring data, two different types of data, namely action information and image sequences of interest, are obtained, which enables full mining of video monitoring data and improves the richness and completeness of the data.

Description of the drawings

FIG. 1A shows a schematic diagram of a system architecture in this exemplary embodiment.

FIG. 1B shows a schematic diagram of another system architecture in this exemplary embodiment.

Figure 2 shows a flow chart of a method for detecting abnormal human brain discharge in this exemplary embodiment.

FIG. 3 shows a flow chart for obtaining biomedical feature data in this exemplary embodiment.

Figure 4 shows a schematic diagram of multi-channel EEG signals in this exemplary embodiment.

Figure 5 shows a schematic diagram of feature extraction from multi-channel EEG monitoring data in this exemplary embodiment.

Figure 6 shows a schematic diagram of facial key points and body key points in this exemplary embodiment.

FIG. 7 shows a schematic diagram of processing video monitoring data in this exemplary embodiment.

FIG. 8 shows a flow chart for obtaining abnormal discharge detection results using an abnormal discharge detection model in this exemplary embodiment.

Figure 9 shows a schematic diagram of obtaining a training data set in this exemplary embodiment.

FIG. 10 shows a schematic diagram of training an abnormal discharge detection model in this exemplary embodiment.

FIG. 11 shows a schematic diagram of abnormal discharge detection of the human brain in this exemplary embodiment.

Figure 12 shows a schematic structural diagram of a human brain abnormal discharge detection device in this exemplary embodiment.

Figure 13 shows a schematic structural diagram of an electronic device in this exemplary embodiment.

In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

Detailed ways

The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Embodiments of the present disclosure may be implemented as a system, apparatus, equipment, method or computer program product. Therefore, the present disclosure can be implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

The principles and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure.

Summary of the invention

The inventor found that the current accuracy of abnormal human brain discharge detection needs to be improved. Specifically, in related technologies, artificial intelligence technology is used to detect abnormal discharges in the human brain, for example, an intelligent network for epileptiform discharge recognition is trained based on deep learning to detect epileptiform discharges in the human brain. Relevant literature reports that its accuracy rate only reaches 70%, requiring professional doctors and technicians to post-process the artificial intelligence reading results.

In view of the above, the present disclosure provides a human brain abnormal discharge detection method, a human brain abnormal discharge detection device, a computer-readable storage medium and an electronic device. On the one hand, the biomedical characteristic data and video monitoring data are obtained, and the video monitoring data is processed Process to obtain the action information and image sequence of interest of the subject under test. By combining the three types of data, biomedical feature data, action information, and image sequence of interest, to detect abnormal human brain discharges, the accuracy of the detection results can be improved. Different types of Data can make up for each other's missing information. For example, action information or image sequences of interest can make up for information that cannot be reflected in biomedical feature data, ensuring the stability of detection results, reducing misjudgments, and no manual reading is required throughout the process. Through artificial processing, the automatic detection of abnormal discharges in the human brain was realized. On the other hand, through the processing of video monitoring data, two different types of data, namely action information and image sequences of interest, are obtained, which enables full mining of video monitoring data and improves the richness and completeness of the data.

After introducing the basic principles of the present disclosure, various non-limiting implementations of the present disclosure are described in detail below.

Overview of application scenarios

It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principles of the present disclosure, and the implementation of the present disclosure is not subject to any limitation in this regard. Rather, embodiments of the present disclosure may be applied to any applicable scenario.

The embodiments of the present disclosure can be applied to scenarios related to the detection and auxiliary diagnosis and treatment of suspected patients. For example, a user is suspected of suffering from epilepsy. During the treatment in the hospital, the doctor needs to know the user's brain electrical status, especially whether there is any abnormal discharge in the brain. It can be detected by the human brain abnormal discharge detection method of the present disclosure. The following is a detailed description of the application scenarios combined with the system architecture.

Figure 1A shows a schematic diagram of a system architecture for abnormal human brain discharge detection. The system architecture includes a tested object 101, a biomedical monitoring device 102, a video collection device 103, and a processing device 104. When it is necessary to detect abnormal human brain discharge on the subject 101, a biomedical monitoring device 102 and a video collection device 103 can be set up for the subject 101 to collect corresponding data and process the data through the processing device 104.

The biomedical monitoring device 102 may include detection electrodes 1021, 1022, 1023 and a main device 1024. The detection electrodes 1021, 1022, 1023 may contact different parts of the subject 101 to collect electrical signals, and the main device 1024 summarizes and processes the electrical signals. Obtain biomedical monitoring data or biomedical characteristic data. For example, the biomedical monitoring device 102 can be an EEG monitoring device (such as an electroencephalograph), and the detection electrodes 1021, 1022, and 1023 can be fixed to various positions on the head of the subject 101 to collect multi-channel EEG. Signal. Of course, the biomedical monitoring device 102 shown in FIG. 1A is only exemplary, and may also include various other components such as helmets, hats, bedding, and packaged portable detection electrodes, which are not limited by this disclosure.

The video collection device 103 is used to collect video monitoring data of the object 101 under test. For example, it can be a surveillance camera, a mobile phone with a shooting function, etc. It can be set at a position where the camera faces the object 101 under test, and continuously monitors and shoots to obtain video monitoring. data. In one implementation, the video collection device 103 may include a dual-channel camera that captures the face and body of the subject to collect face video monitoring data and body video monitoring data respectively.

The biomedical monitoring device 102 and the video collection device 103 can communicate with the processing device 104, for example, through a wired or wireless communication link, so that the biomedical monitoring device 102 and the video collection device 103 send the collected data. to processing device 104. The processing device 104 can execute the human brain abnormal discharge detection method in this exemplary embodiment, process the acquired multi-modal data, and obtain the abnormal discharge detection result of the subject under test. In one embodiment, the processing device 104 may include a display, which may display abnormal discharge detection results, and may also display one or more of biomedical monitoring data and video monitoring data.

Any two or more of the biomedical monitoring device 102, the video collection device 103, and the processing device 104 can be integrated into the same device. Illustratively, the biomedical monitoring device 102 and the processing device 104 may be integrated in the same device, or more specifically, the main device 1024 of the biomedical monitoring device 102 may serve as the processing device 104 . Or, the video collection device 103 can be integrated in the biomedical monitoring device 102 or the processing device 104. For example, the biomedical monitoring device 102 can include a head-mounted device, and detection electrodes 1021, 1022, and 1023 can be provided in the head-mounted device. When the subject 101 wears the head-mounted device, and the head contacts the detection electrodes 1021, 1022, 1023, a fixed bracket can also be provided on the head-mounted device, and the other end of the fixed bracket is fixedly placed with the video collection device 103 (such as The mobile phone is fixedly placed by a clamp), and the video collection device 103 faces the face and body of the subject 101 to capture video monitoring data.

Figure 1B shows a schematic diagram of another system architecture for abnormal human brain discharge detection. The system architecture includes a tested object 101, a biomedical monitoring device 102, a video collection device 103, a data transceiver device 105, and a server 106. The biomedical monitoring device 102 and the video collection device 103 can be communicatively connected with the data transceiver device 105, and the data transceiver device 105 is communicatively connected with the server 106. As a result, the data transceiving device 105 obtains the biomedical monitoring data or biomedical feature data, face video data, and body key point data, and then sends them to the server 106 . The server 106 may include any form of data processing server such as a cloud server or a distributed server. After acquiring the multi-modal data sent by the data transceiving device 105, the server 106 obtains the abnormal discharge detection result of the subject under test by executing the human brain abnormal discharge detection method in this exemplary embodiment. In one implementation, the server 106 can return the abnormal discharge detection result to the data transceiver device 105 for display on the data transceiver device 105 or the biomedical monitoring device 102 .

The system architecture shown in Figure 1B is suitable for portable scenarios. For example, the subject 101 can use the portable biomedical monitoring device 102 and video collection device 103 in any place such as home or office. The data is sent to the service by the data transceiver device 105. Terminal 106 to realize abnormal discharge detection of human brain. In this way, users can learn the abnormal brain discharge detection results without leaving home. When encountering problems such as suspected epilepsy, the obtained abnormal brain discharge detection results can be sent to doctors to help doctors judge the condition.

In one implementation, the system architecture shown in FIG. 1A or FIG. 1B may also include a motion capture device. The motion capture device may include one or more sensors bound to key parts of the body of the subject 101 to sense. The movement of each part collects body key point data of the subject 101 under test. For example, the sensor can be bound to the arm joints, fingers and palm of the subject 101 (for example, the sensor is arranged on the fingers and palm of the motion capture glove. When the subject 101 wears the motion capture glove, the sensor is located on the subject 101 Fingers and palms), the motion capture device can collect real-time position data of the arm joints, fingers, and palms of the subject 101 to obtain body key point data. Of course, the present disclosure does not limit the number of sensors and the bound body parts. In addition to the above-mentioned arm joints, fingers, and palms, sensors of the motion capture device can be bound to other key parts of the subject 101 according to specific needs. Collect corresponding body key point data.

Example methods

Exemplary embodiments of the present disclosure provide a human brain abnormal discharge detection method. Referring to FIG. 2 , the method may include steps S210 to S250. Each step in Figure 2 is explained in detail below.

Referring to Figure 2, in step S210, the biomedical characteristic data of the subject is obtained.

Among them, the test subjects refer to people who need to detect abnormal discharges in the human brain, such as possible epilepsy patients. Biomedical signals are signals generated by human physiological processes and can reflect a person's physiological state or physical signs. Biomedical feature data is feature data extracted from biomedical signals. Biomedical signals include but are not limited to any one or more of the following signals: electrocardiographic signals, electroencephalographic signals, electromyographic signals, electroocular signals, gastric electrical signals and other electrophysiological signals, and may also include body temperature, blood pressure, pulse, respiration and other non-electrophysiological signals. The following is an exemplary description of biomedical signals and their acquisition process.

The electrocardiogram signal can be an electrocardiogram collected by a multi-lead electrocardiogram machine. Electrocardiogram (ECG) refers to the sequential excitement of the pacemaker, atria, and ventricles of the heart in each cardiac cycle, accompanied by the electrocardiogram bioelectricity. Changes, various forms of patterns of potential changes are elicited from the body surface through an electrocardiograph. Electrocardiogram is an objective indicator of the occurrence, propagation and recovery process of cardiac excitement.

The electroencephalogram signal can be an electroencephalogram collected using a multi-channel electroencephalogram machine. Electroencephalogram (EEG) uses precise instruments to amplify and record the spontaneous biological potential of the cerebral cortex from the scalp. The pattern obtained is the spontaneous and rhythmic electrical activity of brain cell groups recorded through electrodes. This kind of electrical activity is a plan view of the relationship between potential and time recorded with potential as the vertical axis and time as the horizontal axis. The frequency (period), amplitude and phase of brain waves constitute the basic characteristics of electroencephalography.

Electromyography (EMG) is the superposition of action potentials of motor units in many muscle fibers in time and space, and can be obtained by sticking an electromyographic sensor on the skin.

Electrooculogram (EOG) is a bioelectric signal caused by the potential difference between the cornea and retina of the eye. It is extremely convenient to collect and can be completed with a small number of electrodes.

Electrogastrogram (EGG) is an electrical signal generated by the contraction of stomach muscles, which can be collected using electrodes on the surface of the abdominal skin of the human body.

This disclosure does not limit the specific methods of extracting feature data from biomedical signals, which may include but are not limited to: preprocessing, data statistics, feature data extraction through neural networks, etc.

In one embodiment, referring to Figure 3, the above-mentioned acquisition of biomedical characteristic data of the subject may include the following steps S310 to S330:

Step S310: Obtain the biomedical monitoring data of the subject collected by the biomedical monitoring equipment.

Among them, the biomedical monitoring data can be original biomedical signals. For example, the biomedical monitoring equipment can be an electrocardiograph or an electroencephalograph, and the biomedical monitoring data can be electrocardiogram signals or electrocardiograms collected by an electrocardiograph, electroencephalogram signals or electroencephalograms collected by an electroencephalograph, etc. . Electrocardiogram and electroencephalogram are essentially graphs drawn from electrocardiogram signals and electroencephalogram signals at different times. It can be considered that electrocardiogram signals are equivalent to electrocardiograms, and electroencephalogram signals are equivalent to electroencephalograms. Figure 4 shows a schematic diagram of a multi-channel EEG signal. The original EEG monitoring data is drawn into a graph to obtain the waveform shown in Figure 4, that is, the EEG. The electroencephalograph can have 23 electrodes, and each electrode collects signals from one channel. Of course, this disclosure does not limit the number of electrodes of the electroencephalograph, and can be increased or decreased according to specific circumstances.

Step S320: Preprocess the biomedical monitoring data.

By way of example, preprocessing may include but is not limited to one or more of the following processing methods:

① Resampling. Resampling can change the sampling rate of a signal and convert a non-uniformly sampled signal into a uniformly sampled signal. For example, the EEG monitoring data can be resampled at 500Hz.

②Filtering. For example, the EEG monitoring data, ECG monitoring data, and EMG monitoring data include a total of 29 channels, of which the EEG monitoring data includes 23 channels. It can perform 50Hz notch filtering on all 29 channels to reduce the interference of AC signals, and perform bandpass filtering (such as using the 0.1~70Hz frequency band) on 23 EEG channels to reduce signal interference in non-EGG signal frequency bands.

③ Eliminate noise data. Noisy data may be caused by factors such as poor contact (such as poor contact between biomedical monitoring equipment and parts of the subject being tested, poor contact of equipment cables, etc.). Removing noise data can improve data quality and accuracy of test results. For example, channel data with more noise can be eliminated.

④Numerical standardization processing. Numerical standardization processing can map different modalities and different types of data into the same appropriate numerical range for unified processing. For example, different types of biomedical monitoring data such as EEG monitoring data, ECG monitoring data, and EMG monitoring data can be dimensioned first, and then normalized to a suitable value range. For example, by multiplying the corresponding The coefficients are numerically mapped to complete numerical standardization.

Step S330: Obtain biomedical feature data based on the preprocessed biomedical monitoring data.

The preprocessed biomedical monitoring data can be used as biomedical feature data, or further feature extraction processing can be performed on the preprocessed biomedical monitoring data. For example, the preprocessed biomedical monitoring data can be extracted through a pretrained feature extraction model. Monitoring data are processed to obtain biomedical characteristic data.

In one embodiment, the biomedical monitoring data may include multi-channel EEG monitoring data collected from multiple locations on the subject's scalp. The electroencephalograph may have multiple electrodes (such as the above-mentioned detection electrodes 1021, 1022, and 1023, often referred to as movable electrodes), and each electrode can collect one channel of EEG monitoring data. One or more electrodes can be placed on each part of the subject's scalp, so that a total of multiple electrodes are used to collect multi-channel EEG monitoring data. Correspondingly, the above-mentioned obtaining of biomedical characteristic data based on preprocessed biomedical monitoring data may include the following steps:

Calculate the potential difference between each channel and the reference electrode from the preprocessed multi-channel EEG monitoring data to obtain the initial characteristic data of the EEG signal;

Extract biomedical feature data based on the initial feature data of the EEG signal.

Among them, the reference electrode is introduced to ensure the EEG signal with potential difference, which can reduce the impact of noise. The selection of reference electrodes includes but is not limited to the following methods: using unipolar leads, setting movable electrodes and irrelevant electrodes on the subject under test. If the irrelevant electrodes can be set at positions such as the earlobe, it is equivalent to using the irrelevant electrodes as the reference electrode, and then collecting The multi-channel EEG monitoring data includes the potential difference of each channel relative to irrelevant electrodes, which is preprocessed to obtain the initial characteristic data of the EEG signal. Using the bipolar lead method, no irrelevant electrodes are set on the subject, and each movable electrode uses other movable electrodes as reference electrodes. The collected multi-channel EEG monitoring data includes the potential difference of each channel relative to other movable electrodes. , and obtain the initial characteristic data of the EEG signal after preprocessing. Average reference, set the movable electrode and the ground electrode on the subject, then the collected multi-channel EEG monitoring data includes the potential difference of each channel relative to the ground terminal. One or more of the electrodes of the multi-channel EEG monitoring data can be used as an average reference. As a reference electrode, the preprocessed multi-channel EEG monitoring data is used to calculate the potential difference between each channel and the reference electrode, and the initial characteristic data of the EEG signal is obtained. For example, c electrodes of the electroencephalograph are placed on the scalp of the subject, and any one of the c electrodes can be selected as a reference electrode, or multiple electrodes can be used as reference electrodes, and the preprocessed The average value of the multi-channel EEG monitoring data is calculated as a reference value, and the difference between the preprocessed multi-channel EEG monitoring data of each channel and the reference value is calculated to obtain the initial characteristic data of the EEG signal. In one embodiment, different electrodes can be selected as reference electrodes, and the potential difference of each channel can be calculated under different reference electrodes, and then the average value can be calculated as the initial characteristic data of the EEG signal.

After obtaining the initial feature data of the EEG signal, the initial feature data of the EEG signal can be used as the biomedical feature data obtained in step S210, or the initial feature data of the EEG signal can be further processed, such as further extracting effective information therein, Obtain biomedical characteristic data.

In one embodiment, the biomedical characteristic data may include EEG signal waveform characteristics. The above-mentioned extraction of biomedical feature data based on the initial feature data of the EEG signal may include the following steps:

The pre-trained waveform feature extraction model is used to process the initial feature data of the EEG signal to extract the EEG signal waveform features.

Among them, the waveform feature extraction model can be a model with a structure such as a pre-trained Transformer, which can extract timing features and other features in the initial feature data of the electroencephalogram signal. For example, the initial feature data of the EEG signal can be input into the waveform feature extraction model, and the waveform feature extraction model performs embedding and other processing on the initial feature data of the EEG signal to obtain the EEG signal waveform features.

In one embodiment, the biomedical feature data may include EEG signal image features. Image features can be extracted from the EEG signal map that has been preprocessed and/or calculated based on the reference electrode potential difference. For example, the EEG signal map can be input into a pre-trained convolutional neural network and other models to obtain the EEG signal image features.

In one implementation, the biomedical characteristic data may include time-frequency characteristics of the brain electrical signal. The above-mentioned extraction of biomedical feature data based on the initial feature data of the EEG signal may include the following steps:

Perform time-frequency transformation on the initial characteristic data of the EEG signal to obtain the time-frequency data of the EEG signal corresponding to the initial characteristic data of the EEG signal;

Extract the time-frequency characteristics of the EEG signal based on the EEG signal time-frequency data.

Among them, the initial characteristic data of the EEG signal is usually data in the time domain, expressing the changes of the electrical signal over time, and the information is relatively simple. Through time-frequency transformation, the initial characteristic data of the EEG signal can be converted into the joint time-frequency domain, and the time-frequency data of the EEG signal corresponding to the initial characteristic data of the EEG signal can be obtained, which can provide joint distribution information of the signal in the time domain and frequency domain. . Time-frequency transformation methods include but are not limited to short-time Fourier transform, wavelet transform, etc. This disclosure does not limit the specific method used. Furthermore, the time-frequency characteristics of the EEG signal can be extracted from the EEG signal time-frequency data. For example, image features can be extracted from the time-frequency diagram, or statistics and feature extraction can be performed on the EEG signal time-frequency data to obtain the EEG signal. time-frequency characteristics.

Figure 5 shows a schematic diagram of feature data extraction from multi-channel EEG monitoring data. By way of example, the biomedical monitoring data includes multi-channel EEG monitoring data. The multi-channel EEG monitoring data can be preprocessed first, and then the potential difference between each channel and the reference electrode can be calculated using the preprocessed multi-channel EEG monitoring data to obtain the initial characteristic data of the EEG signal. Next, on the one hand, the initial feature data of the EEG signal is input into a pre-trained waveform feature extraction model, such as Transformer, etc., to obtain the EEG signal waveform features. On the other hand, perform short-time Fourier transform on the initial feature data of the EEG signal to obtain the EEG signal time-frequency data, and then input the EEG signal time-frequency data into a pre-trained time-frequency feature extraction model, such as EfficientNetv2, to obtain Time-frequency characteristics of EEG signals.

In addition to the above-mentioned EEG signal waveform characteristics and EEG signal time-frequency characteristics, other aspects of EEG characteristic data can also be extracted, such as EEG statistical characteristic data.

In addition, the method of extracting EEG feature data can be used to extract biomedical feature data from biomedical monitoring data such as ECG, EMG, EOG, and GET.

By way of example, the biomedical monitoring data may include multi-channel ECG monitoring data. The preprocessed multi-channel ECG monitoring data can be used to calculate the potential difference between each channel and the reference electrode to obtain the initial feature data of the ECG signal; biomedical feature data can be extracted based on the initial feature data of the ECG signal. Among them, the biomedical characteristic data includes ECG signal waveform characteristics. The pre-trained waveform feature extraction model can be used to process the initial feature data of the ECG signal to extract the ECG signal waveform features. Biomedical characteristic data includes time-frequency characteristics of ECG signals. The ECG signal initial characteristic data can be time-frequency transformed to obtain the ECG signal time-frequency data corresponding to the ECG signal characteristic data; the ECG signal time-frequency characteristics can be extracted based on the ECG signal time-frequency data.

By way of example, the biomedical monitoring data may include multi-channel electromyographic monitoring data. The preprocessed multi-channel electromyographic monitoring data can be used to calculate the potential difference between each channel and the reference electrode to obtain the initial characteristic data of the electromyographic signal; biomedical characteristic data can be extracted based on the initial characteristic data of the electromyographic signal. Among them, the biomedical characteristic data includes myoelectric signal waveform characteristics. The pre-trained waveform feature extraction model can be used to process the initial feature data of the electromyographic signal to extract the electromyographic signal waveform features. Biomedical characteristic data includes time-frequency characteristics of electromyographic signals. The time-frequency transformation of the initial characteristic data of the electromyographic signal can be performed to obtain the time-frequency data of the electromyographic signal corresponding to the characteristic data of the electromyographic signal; the time-frequency characteristics of the electromyographic signal can be extracted based on the time-frequency data of the electromyographic signal.

In one implementation, after obtaining the original biomedical monitoring data, the biomedical monitoring data can be fragmented within a certain length of time (such as 4 seconds). For example, biomedical monitoring data segments in units of 4 seconds can be obtained. The biomedical monitoring data of each segment are preprocessed respectively, and features are extracted from the preprocessed biomedical monitoring data to obtain the biomedical feature data of each segment. Subsequently, on a segment-by-segment basis, it is detected whether abnormal discharge occurs in each segment of the subject under test.

Continuing to refer to Figure 2, in step S220, the video monitoring data of the object under test is obtained.

Video monitoring data can be captured by video collection equipment, and can be videos of the objects under test taken over a period of time. In one implementation, the video monitoring data may include face video monitoring data and body video monitoring data, which may be collected separately by a dual-channel camera of the video collection device, or may be captured simultaneously by a single-channel camera of the video collection device. Video monitoring data is obtained from the face and body of the subject, and then the face video monitoring data and body video monitoring data are separated from the video monitoring data through screen cropping and other methods.

In this exemplary embodiment, after obtaining the video monitoring data of the subject under test, two aspects of processing can be performed. On the one hand, it is to detect the action information of the subject under test, and on the other hand, it is to extract the image sequence of interest. These two aspects of processing are respectively Executed in steps S230 and S240.

Continuing to refer to FIG. 2 , in step S230 , action information of the object under test is detected based on the video monitoring data.

Among them, the action information is used to characterize the parts where the subject moves, or what actions the subject made. The dynamic changes of the body parts of the subject can be detected from the video monitoring data, and the action information of the subject can be identified. Alternatively, the position information or position change information of the key points of the object under test can be used as the action information of the object under test. For example, the action information of the subject under test may include: key points that move in the face and body of the subject under test and displacement information of the key points.

Continuing to refer to FIG. 2 , in step S240 , an image sequence of interest used to characterize the motion of the subject is extracted from the video monitoring data.

The image sequence of interest may be an image sequence of a Region of Interest (ROI) in the video monitoring data, or it may be an image sequence of frames of interest. For example, the user's face and body can be identified from video monitoring data, and multi-frame face region images and multi-frame body region images can be cropped out to form an image sequence of interest. Alternatively, the frame of interest in which the subject under test has dynamic changes can be first identified from the video monitoring data, and then the body part area in which the subject under test has dynamic changes, that is, the area of interest, can be identified from the frame of interest. The image of the area of interest is cropped out of the frame to form an image sequence of interest.

In one implementation, the video monitoring data includes face video monitoring data. For example, it can be video data collected by a camera in the video collection device that is specially used to capture faces, or it can be people cropped from the video monitoring data. Video data of face area. Correspondingly, the above-mentioned detection of motion information of the subject under test based on video monitoring data may include the following steps:

Detect facial key point data from facial video monitoring data;

Obtain facial action information of the subject based on facial key point data.

Figure 6 shows a schematic diagram of face key points and body key points. It is possible to number 33 key points on the face and body, which are recorded as key points 0 to 32 respectively. Among them, key points 0 to 10 are face key points, and key points 11 to 32 are body key points. Of course, this disclosure does not limit the number of key points and can be increased or decreased according to specific circumstances.

Facial key point data can include the locations of facial key points at different times. Facial key point data can be detected through a pre-trained face detection model or using a face detection algorithm. For example, one or more frames of images can be intercepted from the face video monitoring data, and the images can be analyzed through target detection and key point detection algorithms to detect the locations of key points on the face, such as by detecting eyes, nose, The positions of facial parts such as mouth and ears are used to determine the positions of facial key points, thereby obtaining facial key point data.

The facial key point data represents the static position of the facial key points in one or more frames. The position change information of the facial key points can be analyzed from it and used as facial action information of the subject under test, or for further identification. It can identify the actions performed by the human face, such as blinking, shaking the head, frowning, opening the mouth, etc., and use the recognition results as facial action information of the subject.

In one implementation, the above-mentioned method of obtaining facial action information of the subject under test based on facial key point data may include the following steps:

Based on the face key point data, the key points of the face in motion and their displacement information are determined, and the facial movement information of the subject is obtained.

Among them, the face key point data can represent the positions of the face key points at different times, and based on this, the face key points that have moved within a period of time can be determined, and the identification of these face key points can be recorded (for example, it can be in Figure 6 Key point numbers) and their displacement information to form facial action information of the subject.

In one implementation, the positional relationship between different face key points can be determined based on the face key point data to generate the face action data to be recognized; the face action data to be recognized is combined with the data of multiple standard face actions. The facial action data is matched, and the facial action information corresponding to the facial key point data is determined based on the matching results. Among them, the facial action data to be recognized can be positional relationship sequence data of facial key points. For example, the positional relationship between facial key points can be represented as a vector. The different dimensions of the vector represent the distance, orientation, etc. between different key points. According to the facial key point data of each frame, each frame can be generated correspondingly. The position relationship data of facial key points is in vector form. The position relationship data of the face key points in a single frame is used as the face action data to be recognized, and is matched with the face action data of the preset standard face action. The face action data of the standard face action can also be the face key point. By calculating the similarity between the vectors of the point position data, the face action data of the standard facial action that matches the face action data to be recognized is obtained. If the similarity between the two reaches the similarity threshold, the face action data to be identified is determined. The facial action data corresponds to the standard facial action, thereby obtaining the facial action information corresponding to the facial key point data. Or, the position relationship data of facial key points in different frames is formed into a sequence as the facial action data to be recognized. The facial action data of standard facial actions can also be a sequence. Through matching between sequences, the key points of the face are identified. Facial action information corresponding to point data.

In one embodiment, the sequence of images of interest includes a sequence of human face images. The above-mentioned extraction of the image sequence of interest from the video monitoring data used to characterize the movement of the subject under test may include the following steps:

According to the face key point data, the face area image is cropped from multiple frames of the face video monitoring data to obtain the face part image sequence.

Among them, the face area image is an image containing the entire face. Face video monitoring data may include content other than the face, such as images of the environment around the subject being tested. After obtaining the face key point data, the position of the face area can be determined, and the face area image can be cropped from multiple frames of the face video monitoring data. The face area images of different frames form a face image sequence.

By cropping the face area image to obtain the face image sequence, the information irrelevant to the face in the face video monitoring data can be eliminated, and the cropping process is simple, which is beneficial to reducing the amount of calculation and improving the accuracy of subsequent detection results.

In one implementation, after obtaining the face video monitoring data, the face video monitoring data can be processed within a certain length of time (such as 4 seconds, which time length can be the same as the time length of biomedical monitoring data fragmentation). Segmentation, such as obtaining face video monitoring data segments in units of 4 seconds. In the face video monitoring data of each segment, determine one or more key frames, such as the first frame, the midpoint frame, the last frame, etc., detect the key points of the face in the key frames, and obtain the face The displacement information of key points forms the facial action information of the subject under test. The location of the face area is determined based on the detected face key points, and the face area image is intercepted from the face video monitoring data to form a face image sequence. Subsequently, facial action information, facial image sequence and other information are input into the abnormal discharge detection model in units of segments to detect whether abnormal discharge occurs in the subject under test in each segment.

In one embodiment, the image sequence of interest includes a sequence of facial part images. The above-mentioned extraction of the image sequence of interest from the video monitoring data used to characterize the movement of the subject under test may include the following steps:

According to the face key point data, the face part area images are cut out from multiple frames of the face video monitoring data, and the face part image sequence is obtained.

The face part area image is an image containing one or more types of face parts. Face video monitoring data may include the entire face, and may also include image content other than the face, such as images of the environment around the subject under test. After obtaining the face key point data, the positions of the face parts, such as the eyes, nose, mouth, ears, etc., can be determined, and the area containing all the face parts can be determined (for example, it can be the entire facial area, not Including hair, etc.), you can also determine the area of each key part of the face (such as the eye area, nose area, mouth area, ear area), and crop out the face part area image. The face part area images of different frames form a face part image sequence.

By cropping the face part area image to obtain the face part image sequence, the information irrelevant to the face or facial movements in the face video monitoring data can be eliminated, which is beneficial to reducing the amount of calculation and improving the accuracy of subsequent detection results. sex.

In one implementation, the above-mentioned cropping of face part region images from multiple frames of face video monitoring data based on face key point data may include the following steps:

Based on the face key point data, the facial parts that are moving are determined, and regional images of the moving facial parts are cropped from multiple frames of the face video monitoring data.

Since the facial key point data represents the static position of the facial key points in one or more frames, combined with the facial key point data of different frames, it is possible to determine which facial parts have moved (i.e., there is a dynamic position). changes), thereby intercepting the regional image of the moving facial part from the facial video monitoring data, thereby obtaining the above facial part regional image. For example, based on the face key point data, it can be determined that the key points of the eyes and nose have dynamic changes (usually position changes), but the key points of other parts such as the mouth and ears have not changed dynamically, indicating that the eyes and nose parts have moved. According to the key point position information of the eyes and nose in the face key point data, the bounding box of the eyes and nose parts can be generated from the face video monitoring data (it can be a bounding box that contains both eyes and nose, or it can be a bounding box that contains The bounding box of the eye and the bounding box containing the ear), and crop out the image within the bounding box, or the bounding box can be appropriately enlarged (such as multiplying the width and height of the bounding box by a proportional factor greater than 1, which can be 1.1, 1.2, etc., which can be determined based on experience or specific needs), intercept the image within the enlarged bounding box, and obtain the face region image.

It should be understood that the above-mentioned cropping process can be performed on each frame in the face video monitoring data. For example, based on the facial key point data of each frame, the facial parts that move in each frame are determined, and then the corresponding parts are cropped. Face region image. Alternatively, the above-mentioned cropping process can be performed only on some frames in the face video monitoring data without processing each frame, thereby reducing the number and processing volume of face part area images. For example, after obtaining face video monitoring data, key frame images can be extracted, such as extracting one frame every certain number of frames, or detecting the difference between adjacent frames. When the difference reaches a predetermined value, the The next frame among adjacent frames is extracted as a key frame image. Perform facial key point detection on the key frame image to obtain facial key point data, determine the moving facial parts in the key frame image based on the facial key point data, and crop out the facial part region image. Or, for the face video monitoring data within a period of time (usually the unit detection duration, for example, the duration can be determined based on the number of images that the abnormal discharge detection model can process), based on the facial key point data of each frame, determine what happened The moving facial part is then cropped for each frame or key frame to obtain the facial part regional image.

Since facial movements are mainly reflected in the moving parts of the face, extracting the regional images of the moving parts of the face from the face video monitoring data can focus subsequent processing on the moving parts of the face. Position. Especially in the case of abnormal discharge of the human brain, facial movements are usually relatively subtle expressions. By detecting the regional images of the moving parts of the face, it is easier to detect the detailed information and further improve the subsequent detection results. accuracy and reduce the amount of calculation.

In one implementation, the video monitoring data includes body video monitoring data. For example, it can be video data collected by a camera in the video collection device that specifically shoots the body, or it can be a body area cropped from the video monitoring data ( (excluding human faces) video data. Correspondingly, the above-mentioned detection of motion information of the subject under test based on video monitoring data may include the following steps:

Detect body key point data from body video monitoring data;

The body movement information of the subject is obtained based on the body key point data.

Among them, the key points of the body can be seen in Figure 6. Body key point data may include the positions of body key points at different times. Body key point data can be detected through a pre-trained body detection model or using a limb detection algorithm. For example, one or more frames of images can be intercepted from body video monitoring data, and the images can be analyzed through target detection and key point detection algorithms to detect the positions of key points of the body, such as by detecting the neck, chest, and limbs. Wait for the position of the body parts to determine the position of the key points of the body, thereby obtaining the key point data of the body.

The body key point data represents the static position of the body key points at the moment of one or more frames. The position change information of the body key points can be analyzed from it and used as the body action information of the subject, or the body's actions can be further identified. Which actions are recognized, such as raising hands, clapping, shaking and other actions, and using the recognition results as the body action information of the subject.

In one embodiment, obtaining the body movement information of the subject based on body key point data may include the following steps:

According to the key point data of the body, the key points of the body in motion and their displacement information are determined, and the body movement information of the subject is obtained.

Among them, the body key point data can represent the positions of the body key points at different times. Based on this, the body key points that have moved within a period of time can be determined, and the identification of these body key points can be recorded (for example, it can be the key point number in Figure 6). and its displacement information to form body movement information of the subject under test.

In one implementation, the positional relationship between different body key points can be determined based on body key point data to generate body action data to be identified; the body action data to be identified is matched with body action data of multiple standard body actions. , determine the body action information corresponding to the body key point data based on the matching results. Among them, the body action data to be recognized can be positional relationship sequence data of key points of the body. For example, the positional relationship between body key points can be represented as a vector. The different dimensions of the vector represent the distance, orientation, etc. between different key points. According to the body key point data of each frame, the body key points of each frame can be generated correspondingly. Point position relationship data, in vector form. The body key point position relationship data of a single frame is used as the body action data to be recognized, and is matched with the body action data of the preset standard body action. The body action data of the standard body action can also be a vector of body key point position data, through Calculate the similarity between vectors to obtain the body action data of the standard body action that matches the body action data to be identified. If the similarity between the two reaches the similarity threshold, it is determined that the body action data to be identified corresponds to the standard body action, as follows This obtains the body action information corresponding to the body key point data. Alternatively, the body key point position relationship data of different frames can be formed into a sequence as the body action data to be identified. The body action data of standard body actions can also be a sequence. Through matching between sequences, the body corresponding to the body key point data can be identified. action information.

In one embodiment, the sequence of images of interest includes a sequence of body images. The above-mentioned extraction of the image sequence of interest from the video monitoring data used to characterize the movement of the subject under test may include the following steps:

According to the body key point data, body region images are cropped from multiple frames of body video monitoring data to obtain a body part image sequence.

Among them, the body region image is an image containing the entire body. Body video monitoring data may include images other than the body, such as images of the environment around the subject under test. After obtaining the body key point data, the position of the body region can be determined, and the body region image can be cropped from multiple frames of body video monitoring data. The body region images of different frames form a body image sequence.

By cropping the body region image to obtain the body image sequence, information irrelevant to the body in the body video monitoring data can be eliminated, and the cropping process is simple, which is beneficial to reducing the amount of calculation and improving the accuracy of subsequent detection results.

In one implementation, after acquiring the body video monitoring data, the body video monitoring data can be fragmented with a certain length of time (such as 4 seconds, which time length can be the same as the length of time for fragmenting biomedical monitoring data). , such as obtaining body video monitoring data segments in units of 4 seconds. In the body video monitoring data of each segment, determine one or more key frames, such as the first frame, the midpoint frame, the last frame, etc., detect the body key points in the key frames, and obtain the body key points. Displacement information, thus forming body movement information of the subject under test. The position of the body region is determined based on the detected body key points, and the body region image is intercepted from the body video monitoring data to form a body image sequence. Subsequently, body action information, body image sequences and other information are input into the abnormal discharge detection model on a segment basis to detect whether abnormal discharge occurs in the subject under test in each segment.

In one embodiment, the sequence of images of interest includes a sequence of body part images. The above-mentioned extraction of the image sequence of interest from the video monitoring data used to characterize the movement of the subject under test may include the following steps:

According to the body key point data, body part area images are cropped from multiple frames of body monitoring video data to obtain a body part image sequence.

The body part region image is an image containing one or more body parts. Body video monitoring data may include the entire body, and may also include image content outside the body, such as images of the environment around the subject under test. After obtaining the key point data of the body, the position of the body parts, such as the neck, chest, left arm, left hand, right hand, right arm, left leg, left foot, right leg, right foot, etc., can be determined. The areas of all body parts can also be determined for each key body part (such as the neck area, left arm area, and left hand area), and the body part area image can be cropped. The body part region images of different frames form a body part image sequence.

By cropping body part region images to obtain body part image sequences, information irrelevant to the body or body movements in the body video monitoring data can be eliminated, which is beneficial to reducing the amount of calculation and improving the accuracy of subsequent detection results.

In one implementation, the above-mentioned cropping of body part region images from multiple frames of body monitoring video data based on body key point data may include the following steps:

Based on the body key point data, the body parts where movement occurs are determined, and regional images of the body parts where movement occurs are cropped from multiple frames of body monitoring video data.

Since the body key point data represents the static position of the body key point at one or more frames, combining the body key point data of different frames can determine which body parts have moved (that is, there is a dynamic change in position). Therefore, The regional image of the body part that is in motion is intercepted from the body video monitoring data to obtain the above-mentioned body part regional image. For example, based on the body key point data, it can be determined that the key points of the eyes and nose have undergone dynamic changes (usually position changes), while the key points of other parts such as the mouth and ears have not changed dynamically. This means that the eyes and nose have moved, and the key points of the eyes and nose have moved. According to the key point position information of the eyes and nose in the body key point data, the bounding box of the eyes and nose is generated from the body video monitoring data (it can be a bounding box that contains both eyes and nose, or it can be a bounding box that contains the eyes. frame and a bounding box containing ears), and crop out the image within the bounding box, or you can appropriately enlarge the bounding box (such as multiplying the width and height of the bounding box by a proportional factor greater than 1, which can be 1.1, 1.2 etc., can be determined based on experience or specific needs), intercept the image within the enlarged bounding box, and obtain the body part area image.

It should be understood that the above-mentioned cropping process can be performed on each frame in the body video monitoring data. For example, based on the body key point data of each frame, the body parts that move in each frame are determined, and then the corresponding body part area images are cropped. . Alternatively, the above-mentioned cropping process can be performed only on some frames in the body video monitoring data without processing each frame, thereby reducing the number and processing volume of body part region images. For example, after obtaining the body video monitoring data, key frame images can be extracted, such as extracting one frame every certain number of frames, or detecting the difference between adjacent frames. When the difference reaches a predetermined value, the corresponding The next frame among adjacent frames is extracted as a key frame image. Perform body key point detection on the key frame image to obtain body key point data, determine the body parts that are moving in the key frame image based on the body key point data, and crop out the body part region image. Or, for the body video monitoring data within a period of time (usually the unit detection duration, for example, the duration can be determined based on the number of images that the abnormal discharge detection model can process), based on the body key point data of each frame, determine where the movement occurs body part, and then crop the regional image of the moving body part for each frame or key frame to obtain the body part regional image.

Since body movements are mainly reflected in the moving body parts, extracting the regional images of the moving body parts from the body video monitoring data can focus subsequent processing on the moving body parts. Especially in the case of abnormal discharge of the human brain, the body movements are usually subtle. By detecting the regional images of the moving body parts, it is easier to detect the detailed information, further improve the accuracy of subsequent detection results, and reduce the amount of calculation.

Figure 7 shows a schematic diagram of processing video monitoring data. After obtaining the video monitoring data, the face video monitoring data and body video monitoring data are separated. Input the face video monitoring data into the pre-trained face detection model to obtain the face key point data; obtain the face action information based on the face key point data; crop the person from the face video monitoring data based on the face key point data face image sequence. Input the body video monitoring data into the pre-trained body detection model to obtain body key point data; obtain body action information based on the body key point data; crop the body image sequence from the body video monitoring data based on the body key point data.

Continuing to refer to Figure 2, in step S250, the pre-trained abnormal discharge detection model is used to process the biomedical feature data, action information, and the image sequence of interest to obtain the abnormal discharge detection result of the subject under test.

Biomedical feature data, action information, and image sequences of interest are data in different modalities. The abnormal discharge detection model can comprehensively process the three types of data to obtain the final abnormal discharge detection result. Different types of data can make up for each other's missing information. For example, when the subject makes some actions such as blinking or speaking, it may affect biomedical feature data such as EEG, causing misjudgment. Through action information or image sequences of interest, Compensating for information that cannot be reflected in biomedical feature data can reduce misjudgments.

In one implementation, the abnormal discharge detection model includes a feature processing layer, an attention layer, and a classification layer. The feature processing layer, attention layer, and classification layer are the three main parts of the abnormal discharge detection model, and each part can include one or more intermediate layers.

Referring to Figure 8, the above-mentioned process of using the pre-trained abnormal discharge detection model to process biomedical feature data, action information, and image sequences of interest to obtain abnormal discharge detection results of the subject under test may include the following steps S810 to S840:

Step S810, input biomedical feature data, action information, and image sequences of interest into the abnormal discharge detection model;

Step S820, use the feature processing layer to extract action feature data from action information, extract image feature data from the image sequence of interest, and fuse biomedical feature data, action feature data, and image feature data to obtain fusion features;

Step S830, use the attention layer to characterize the fused features to obtain embedded features;

Step S840: Use the classification layer to map the embedded features to the output space to obtain the abnormal discharge detection result of the object under test.

Among them, the feature processing layer can directly use action information as action feature data. For example, both action information and action feature data can be action recognition results, or the action information includes position information of facial key points and/or body key points. Feature processing The layer can further process the action information through full connection, attention, etc., and extract action feature data. The feature processing layer can perform convolution and other processing on the image sequence of interest to extract image feature data. For example, the feature processing layer may include neural networks with structures such as LSTM (Long Short-Term Memory), GRU (Gated Recurrent Unit), and CNN (Convolutional Neural Network). The unit can process the image sequence of interest and extract image feature data. The feature processing layer can also use MLP (multi-layer perceptron), splicing and other methods for feature fusion. For example, the feature processing layer may include one or more fully connected layers and splicing layers. Biomedical feature data, action feature data, and image feature data are aligned in feature dimensions through the fully connected layer, and then the splicing operation is performed through the splicing layer. Get fused features.

In the attention layer, fusion features can be selected, such as re-characterizing the fusion features through attention weights to obtain embedded features. Embedded features can be dense features, each dimension of which can incorporate information from different modalities.

The classification layer can include a fully connected layer, etc., which can be gradually mapped to the input space by performing a fully connected operation on the embedded features. Abnormal discharges can also be obtained through activation functions such as sigmoid (S-shaped function) and softmax (normalized exponential function). The probability value of detection can be used as the final abnormal discharge detection result, or the probability value can be used to determine whether there is abnormal discharge, thereby obtaining the final abnormal discharge detection result.

Figure 9 shows a schematic diagram of obtaining the training data set. For example, to obtain biomedical monitoring sample data, for example, EEG signals can be collected through an EEG machine to obtain multi-channel EEG monitoring sample data. Perform preprocessing, and then mark abnormal discharges. The continuous data can be cut according to a fixed duration, such as cutting into 4-second segments, classified into positive and negative samples, and the time period of each sample on the original continuous data is recorded. Extract fragments containing abnormal discharge annotations as positive samples, fragments without abnormal discharge annotations as negative samples, and discard samples marked as ictal or suspicious (doctors are not sure whether they are abnormal discharges). Further extract the medical feature training data. The marked result of abnormal discharge is used as marked data. Obtain video monitoring sample data in the same time period as the biomedical monitoring sample data, which can include video monitoring sample data of faces and bodies. Analyze faces and limbs on video monitoring sample data, detect facial key points and body key points, and then obtain action sample information and image sample sequences of interest. Biomedical feature training data, action sample information, image sample sequences of interest and corresponding annotation data in the same time period form a set of supervised data, and a large amount of supervised data forms a training data set.

Figure 10 shows a schematic diagram of training an abnormal discharge detection model. Input the biomedical feature training data, action sample information, and image sample sequence of interest in the training data set into the abnormal discharge detection model to be trained, and obtain the corresponding abnormal discharge detection sample data. Calculate the loss function based on the abnormal discharge detection sample data and annotation data. value, and update the parameters of the abnormal discharge detection model according to the loss function value. The update process is iteratively executed until the abnormal discharge detection model reaches the predetermined training completion conditions. For example, the accuracy on the verification set or test set reaches the standard, and the trained abnormal discharge detection model is obtained.

In one embodiment, before using the pre-trained abnormal discharge detection model to process biomedical feature data, action information, and image sequences of interest, the human brain abnormal discharge detection method may also include the following steps:

Time-aligning at least one of the action information and the image sequence of interest with the biomedical feature data.

Among them, biomedical feature data, action information, and image sequences of interest come from different devices, and different devices may have time differences, resulting in time asymchronization between biomedical feature data, action information, and image sequences of interest, affecting people. Abnormal brain discharge test results. For example, the time information of biomedical feature data comes from the time of biomedical monitoring equipment, and the time information of action information and image sequences of interest comes from the time of video acquisition equipment. Therefore, it can be considered that the time information of the action information and the image sequence of interest is consistent, and at least one of the action information and the image sequence of interest will be time aligned with the biomedical feature data, thereby achieving time consistency of the three types of data.

The following takes the time of aligning action information and biomedical feature data as an example to illustrate.

In one embodiment, the above-mentioned time alignment of at least one of the action information and the image sequence of interest with the biomedical feature data may include the following steps:

Detect one or more biomedical feature data suspected of abnormal discharge and the corresponding one or more first time points from the biomedical feature data;

Detect one or more action data suspected of abnormal discharge and the corresponding one or more second time points from the action information;

Match the biomedical characteristic data of the suspected abnormal discharge with the action data of the suspected abnormal discharge, and determine the correspondence between the first time point and the second time point based on the matching results;

Based on the correspondence between the first time point and the second time point, the time calibration parameter is determined, and the time calibration parameter is used to time align the action information and the biomedical feature data.

Among them, suspected abnormal discharge may occur at one or more time points, and these times are the above-mentioned first time point or second time point. Alternatively, the suspected abnormal discharge may also occur in one or more time periods, and the starting time point and end time point of these time periods can be used as the first time point or the second time point. For example, if a time period in which the signal value is abnormally increased is detected in the biomedical feature data, the biomedical feature data in this time period can be used as the biomedical feature data of suspected abnormal discharge, and the starting time point of this time point and The end time point is taken as the first time point. Based on a similar method, the action data and the second time point of the suspected abnormal discharge can be determined.

The biomedical characteristic data of the suspected abnormal discharge and the action data of the suspected abnormal discharge are matched, and then the time difference between the first time point and the second time point with the corresponding relationship is calculated to obtain the time calibration parameters. For example, the biomedical characteristic data of the suspected abnormal discharge at each first time point can be obtained, and the first time point and the nearest second time point can be matched into the same set of time calibration data.

In one embodiment, matching the biomedical characteristic data of suspected abnormal discharge with the action data of suspected abnormal discharge may include the following steps:

Determine the first relative value between the biomedical characteristic data of the suspected abnormal discharge and other biomedical characteristic data;

Determine the second relative value between the action data suspected of abnormal discharge and other action data in the action information;

By comparing the first relative value and the second relative value, a matching result between the biomedical characteristic data of suspected abnormal discharge and the action data of suspected abnormal discharge is obtained.

The first relative value represents the relative difference between the biomedical characteristic data of the suspected abnormal discharge and the biomedical characteristic data in the normal state, and may be in the form of a percentage, for example. The second relative value represents the relative difference between the action data suspected of abnormal discharge and the action data under normal conditions. The closest first relative value and second relative value can be matched together to obtain a matching result between the biomedical characteristic data of suspected abnormal discharge and the action data of suspected abnormal discharge, thereby matching the corresponding first time point and second relative value. Two time points form a set of time calibration data. This can improve the accuracy of matching.

Calculate the time difference between the first time point and the second time point in each set of time calibration data to obtain the time calibration parameters. The time calibration parameters of each group can be averaged to obtain the final time calibration parameters. For example, the first time point may be used as a reference, and the time calibration parameter includes the time difference between the second time point and the first time point. According to the time difference between the second time point and the first time point, the time stamp of the action information is adjusted, and the time stamp of the image sequence of interest can also be adjusted. This achieves time alignment of biomedical feature data, action information, and image sequences of interest.

Figure 11 shows a schematic diagram of abnormal discharge detection in the human brain. The EEG machine collects multi-channel EEG monitoring data, and after preprocessing and feature extraction, biomedical feature data is obtained; the video acquisition equipment collects face video monitoring data and body video monitoring data, and the face video monitoring data is processed Face key points are detected, and after further processing, facial action information and face image sequences are obtained. Body video monitoring data are subjected to body key point detection, and after further processing, body action information and body image sequences are obtained. Input biomedical feature data, face action information, face image sequence, body action information, and body image sequence into the trained abnormal discharge detection model, and output the abnormal discharge detection results.

Exemplary device

Exemplary embodiments of the present disclosure also provide a human brain abnormal discharge detection device. Referring to Figure 12, the human brain abnormal discharge detection device 1200 may include the following program modules: a first acquisition module 1210, configured to acquire the biomedical characteristic data of the subject; a second acquisition module 1220, configured to acquire the biomedical characteristic data of the subject; Video monitoring data of the object; the action information detection module 1230 is configured to detect the action information of the object under test according to the video monitoring data; the image sequence extraction module 1240 is configured to extract actions for the object under test from the video monitoring data Characterized image sequence of interest; the model processing module 1250 is configured to use a pre-trained abnormal discharge detection model to process biomedical feature data, action information, and image sequences of interest to obtain abnormal discharge detection results of the subject.

In one embodiment, obtaining the biomedical characteristic data of the subject includes: obtaining the biomedical monitoring data of the subject collected by the biomedical monitoring equipment; preprocessing the biomedical monitoring data; and based on the preprocessed Biomedical monitoring data yields biomedical characteristic data.

In one embodiment, the biomedical monitoring data includes: multi-channel EEG monitoring data collected from multiple parts of the subject's scalp; biomedical feature data is obtained based on the preprocessed biomedical monitoring data, including: Calculate the potential difference between each channel and the reference electrode from the preprocessed multi-channel EEG monitoring data to obtain the initial feature data of the EEG signal; extract biomedical feature data based on the initial feature data of the EEG signal.

In one embodiment, the biomedical feature data includes EEG signal waveform features; extracting biomedical feature data based on the EEG signal initial feature data includes: using a pre-trained waveform feature extraction model to process the EEG signal initial feature data , to extract EEG signal waveform features.

In one embodiment, the biomedical feature data includes time-frequency features of the EEG signal; extracting the biomedical feature data based on the initial feature data of the EEG signal includes: performing time-frequency transformation on the initial feature data of the EEG signal to obtain the EEG signal. The time-frequency data of the EEG signal corresponding to the initial feature data; the time-frequency characteristics of the EEG signal are extracted based on the EEG signal time-frequency data.

In one embodiment, preprocessing the biomedical monitoring data includes at least one of the following processes: resampling, filtering, removing noise data, and numerical standardization.

In one implementation, the video monitoring data includes face video monitoring data; detecting the action information of the subject under test based on the video monitoring data includes: detecting face key point data from the face video monitoring data; based on the face key points The data obtains facial movement information of the subject.

In one embodiment, the image sequence of interest includes a face image sequence; extracting the image sequence of interest for characterizing the actions of the subject from the video monitoring data includes: extracting the image sequence from the face according to the face key point data. Face area images are cropped from multiple frames of face video monitoring data to obtain a face image sequence.

In one implementation, obtaining the facial action information of the subject under test based on the facial key point data includes: determining the facial key points in motion and their displacement information based on the facial key point data, and obtaining the facial movement information of the subject under test. Facial action information.

In one embodiment, the video monitoring data includes body video monitoring data; detecting the action information of the subject under test based on the video monitoring data includes: detecting body key point data from the body video monitoring data; obtaining the subject under test based on the body key point data. The object's body movement information.

In one embodiment, the image sequence of interest includes a body image sequence; extracting the image sequence of interest for characterizing the movement of the subject from the video monitoring data includes: based on the body key point data, extracting the image sequence of interest from the body monitoring video Body region images are cropped from multiple frames of data to obtain a body image sequence.

In one implementation, obtaining the body movement information of the subject under test based on the body key point data includes: determining the body key points where movement occurs and their displacement information based on the body key point data, and obtaining the body movement information of the subject under test.

In one implementation, the abnormal discharge detection model includes a feature processing layer, an attention layer, and a classification layer; the pre-trained abnormal discharge detection model is used to process biomedical feature data, action information, and image sequences of interest to obtain the measured The abnormal discharge detection results of objects include: inputting biomedical feature data, action information, and image sequences of interest into the abnormal discharge detection model; using the feature processing layer to extract action feature data from action information, and extract image features from the image sequence of interest data, and fuse biomedical feature data, action feature data, and image feature data to obtain fused features; use the attention layer to characterize the fused features to obtain embedded features; use the classification layer to map the embedded features to the output space to obtain the test object Abnormal discharge detection results.

In one embodiment, the model processing module 1250 is also configured to: before using the pre-trained abnormal discharge detection model to process the biomedical feature data, action information, and the image sequence of interest, combine the action information and the image of interest. At least one of the sequences is time aligned with the biomedical characteristic data.

In one embodiment, time-aligning at least one of the action information and the image sequence of interest with the biomedical feature data includes: detecting one or more biomedical features suspected of abnormal discharge from the biomedical feature data. data and one or more corresponding first time points; detect one or more action data of suspected abnormal discharge and one or more corresponding second time points from the action information; compare the biomedical characteristic data of suspected abnormal discharge with Match the action data of suspected abnormal discharge, determine the correspondence between the first time point and the second time point based on the matching results; determine the time calibration parameters based on the correspondence between the first time point and the second time point, and Time alignment parameters are used to time-align motion information with biomedical feature data.

In one embodiment, matching the biomedical feature data of the suspected abnormal discharge with the action data of the suspected abnormal discharge includes: determining a first relative value between the biomedical feature data of the suspected abnormal discharge and other biomedical feature data. ; Determine the second relative value between the action data of the suspected abnormal discharge and other action data in the action information; by comparing the first relative value and the second relative value, obtain the biomedical characteristic data of the suspected abnormal discharge and the data of the suspected abnormal discharge Matching results between action data.

In addition, other specific details of the implementation of the present disclosure have been described in detail in the implementation of the above method and will not be described again here.

Example storage media

Exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above method of the present disclosure is implemented. The above method can be implemented by a program product, such as a portable compact disk read-only memory (CD-ROM) and containing program code, and can be run on a device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

The Program Product may take form in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: electrical connection with one or more conductors, portable disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium that can send, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RE, etc., or any suitable combination of the above.

Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., as well as conventional procedural programming. Language - such as "C" or a similar programming language. The program code may execute entirely on the user's computing device, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In situations involving remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., provided by an Internet service). (business comes via Internet connection).

Example electronic device

Exemplary embodiments of the present disclosure also provide an electronic device, which may be any device in FIG. 1A or FIG. 1B. The electronic device includes a processor and a memory, and the memory is used to store executable instructions of the processor. The processor is configured to perform the above-described methods of the present disclosure via executing executable instructions.

An electronic device according to an exemplary embodiment of the present disclosure will be described with reference to FIG. 13 . The electronic device 1300 shown in FIG. 13 is only an example and should not bring any limitations to the functions and usage scope of the embodiments of the present disclosure.

As shown in Figure 13, electronic device 1300 is embodied in the form of a general computing device. The components of the electronic device 1300 may include, but are not limited to: at least one processing unit 1310, at least one storage unit 1320, and a bus 1330 connecting different system components (including the storage unit 1320 and the processing unit 1310).

Wherein, the storage unit stores program code, and the program code can be executed by the processing unit 1310, so that the processing unit 1310 performs the steps according to various exemplary embodiments of the present disclosure described in the "Example Method" section of this specification. For example, the processing unit 1310 may perform the method steps shown in FIG. 2 and the like.

The storage unit 1320 may include a volatile storage unit, such as a random access storage unit (RAM) 1321 and/or a cache storage unit 1322, and may further include a read-only storage unit (ROM) 1323.

Storage unit 1320 may also include a program/utility 1324 having a set of (at least one) program modules 1325 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, Each of these examples, or some combination, may include the implementation of a network environment.

Bus 1330 may include a data bus, an address bus, and a control bus.

Electronic device 1300 may also communicate with one or more external devices 1400 (eg, keyboard, pointing device, Bluetooth device, etc.), which communication may occur through an input/output (I/O) interface 1340. Electronic device 1300 may also communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN), and/or a public network, such as the Internet) through a network adapter 1350. As shown, network adapter 1350 communicates with other modules of electronic device 1300 via bus 1330. It should be understood that, although not shown in the figures, other hardware and/or software modules may be used in conjunction with electronic device 1300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives And data backup storage system, etc.

It should be noted that although several modules or sub-modules of the device are mentioned in the above detailed description, this division is only exemplary and not mandatory. Indeed, according to embodiments of the present disclosure, the features and functions of two or more units/modules described above may be embodied in one unit/module. Conversely, the features and functions of one unit/module described above may be further divided to be embodied by multiple units/modules.

Furthermore, although the operations of the disclosed methods are depicted in a particular order in the drawings, this does not require or imply that the operations must be performed in that particular order, or that all illustrated operations must be performed to achieve desired results. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step for execution, and/or one step may be broken down into multiple steps for execution.

Although the spirit and principles of the present disclosure have been described with reference to a number of specific embodiments, it should be understood that the disclosure is not limited to the specific embodiments disclosed, nor does the delineation of aspects mean that features in these aspects cannot be combined. Benefit, this division is only for convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims (10)

1. A method for detecting abnormal human brain discharge, which is characterized by including: Obtain biomedical characteristic data of the subject; Obtain video monitoring data of the subject under test; Detect the action information of the subject under test based on the video monitoring data; Extract an image sequence of interest for characterizing the movement of the subject from the video monitoring data; The biomedical feature data, the action information, and the image sequence of interest are processed using a pre-trained abnormal discharge detection model to obtain the abnormal discharge detection result of the tested object. 2. The method according to claim 1, characterized in that said obtaining the biomedical characteristic data of the subject includes: Obtain the biomedical monitoring data of the subject collected by the biomedical monitoring equipment; Preprocess the biomedical monitoring data; The biomedical characteristic data is obtained according to the preprocessed biomedical monitoring data. 3. The method according to claim 2, wherein the biomedical monitoring data includes: multi-channel EEG monitoring data collected from multiple parts of the scalp of the subject; The biomedical characteristic data is obtained from the subsequent biomedical monitoring data, including: Calculate the potential difference between each channel and the reference electrode from the preprocessed multi-channel EEG monitoring data to obtain the initial characteristic data of the EEG signal; The biomedical feature data is extracted according to the initial feature data of the EEG signal. 4. The method according to claim 3, wherein the biomedical feature data includes EEG signal waveform features; and extracting the biomedical feature data according to the EEG signal initial feature data includes: The initial feature data of the EEG signal is processed using a pre-trained waveform feature extraction model to extract the EEG signal waveform features. 5. The method according to claim 3, wherein the biomedical feature data includes time-frequency features of the EEG signal; and extracting the biomedical feature data according to the initial feature data of the EEG signal includes: Perform time-frequency transformation on the initial characteristic data of the EEG signal to obtain EEG signal time-frequency data corresponding to the initial characteristic data of the EEG signal; The time-frequency characteristics of the EEG signal are extracted according to the EEG signal time-frequency data. 6. The method according to claim 2, wherein the preprocessing of the biomedical monitoring data includes at least one of the following processes: resampling, filtering, eliminating noise data, and numerical standardization. 7. The method according to claim 1, wherein the video monitoring data includes face video monitoring data; and detecting the action information of the subject under test according to the video monitoring data includes: Detect facial key point data from the facial video monitoring data; The facial action information of the subject is obtained according to the facial key point data. 8. A human brain abnormal discharge detection device, characterized in that it includes: The first acquisition module is configured to acquire biomedical characteristic data of the subject; The second acquisition module is configured to acquire the video monitoring data of the object under test; An action information detection module configured to detect action information of the subject under test based on the video monitoring data; An image sequence extraction module configured to extract an image sequence of interest for characterizing the action of the subject from the video monitoring data; The model processing module is configured to use a pre-trained abnormal discharge detection model to process the biomedical feature data, the action information, and the image sequence of interest to obtain the abnormal discharge detection result of the tested object. 9. A computer-readable storage medium with a computer program stored thereon, characterized in that when the computer program is executed by a processor, the method of any one of claims 1 to 7 is implemented. 10. An electronic device, characterized in that it includes: processor; and memory for storing executable instructions for the processor; wherein the processor is configured to perform the method of any one of claims 1 to 7 via execution of the executable instructions.