KR102835913B1 - Method, and apparatus for providing user interface for labeling of medical artificial intelligence - Google Patents

Abstract

According to one embodiment of the present disclosure, a method, a program and a device for providing a user interface for labeling medical artificial intelligence are disclosed. The method may include the steps of providing a first user interface listing labelable electrocardiogram features on an electrocardiogram signal; and the steps of providing a second user interface for visually displaying the electrocardiogram signal.

Description

METHOD, AND APPARATUS FOR PROVIDING USER INTERFACE FOR LABELING OF MEDICAL ARTIFICIAL INTELLIGENCE

The present disclosure relates to user interface technology, and more particularly, to a user interface that provides a labeling function for learning of artificial intelligence used in analysis of medical data.

Electrocardiogram signals can be typically divided into P waves, QRS waves, and T waves. When trying to assess health status or disease using an electrocardiogram, the electrocardiogram signal is not analyzed as a whole, but is divided into P waves, QRS waves, and T waves and then analyzed. For example, the PR interval, which means the start of the P wave to the start of the QRS wave, is normally no longer than 0.2 seconds, the QRS wave is within 0.1 seconds, and the QT interval, which means the interval from the start of the Q wave to the end of the T wave, is approximately 0.4 seconds. Signal waveforms and intervals that affect the assessment of health status or disease in this way are generally called electrocardiogram features.

Accurately distinguishing P waves, QRS waves, and T waves in an ECG signal is an important factor in increasing the accuracy of the analysis. When analyzing an ECG signal using AI, accuracy can be further improved by inputting information on the exact P waves, QRS waves, and T waves. In other words, if AI can accurately identify ECG characteristics when analyzing an ECG signal, it can make judgments about health status and the presence of diseases very accurately.

However, in order for AI to accurately identify ECG features on its own, it needs to have the right answer to how the ECG features are distinguished in the ECG signal. This answer is given through labeling by experts in the ECG interpretation field. For example, an expert can find the onset, peak, and offset of the ECG features, P wave, QRS wave, and T wave, in the ECG signal and give the AI the right answer that it can recognize. Since this task is burdensome and time-consuming, it is necessary to build an environment where it can be performed as efficiently as possible.

Republic of Korea Patent Publication No. 10-2251291 (May 6, 2021)

The present disclosure aims to provide a method that provides a user interface that enables an expert to efficiently perform labeling of electrocardiogram signals with simple access and operation.

However, the problems to be solved in the present disclosure are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood based on the description below.

According to one embodiment of the present disclosure for achieving the above-described task, a method for providing a user interface for labeling of medical artificial intelligence is disclosed. The method may include the steps of providing a first user interface listing labelable electrocardiogram features on an electrocardiogram signal; and the steps of providing a second user interface for visually displaying the electrocardiogram signal.

Alternatively, the first user interface may include a first control graphic that dynamically changes a visual representation and state in response to a first command to select an electrocardiogram feature to be labeled from among the labelable electrocardiogram features on the electrocardiogram signal.

Alternatively, the second user interface may include a second control graphic displayed on the electrocardiogram signal in response to a second command to label the electrocardiogram feature selected in accordance with the first command on the electrocardiogram signal.

Alternatively, when a plurality of second control graphics exist on the ECG signal, a second control graphic existing in a waveform related to an ECG characteristic corresponding to a first control graphic activated in response to the first command among the plurality of second control graphics may be activated on the ECG signal.

Alternatively, the second user interface may include a third control graphic displayed on the electrocardiogram signal based on labels for predicted electrocardiogram features by inputting the electrocardiogram signal, on which the second control graphic is not displayed, or the electrocardiogram signal with the second control graphic displayed, into a pre-trained artificial intelligence model.

Alternatively, the third control graphic may specify a location of the predicted electrocardiogram feature on the electrocardiogram signal to guide the input of the second command, or may express error information of the electrocardiogram feature specified through the second control graphic based on the predicted label.

Alternatively, the second user interface may include a fourth control graphic representing location information of a point on the electrocardiogram signal at which the second command is input or a guide line to assist in labeling on the electrocardiogram signal.

Alternatively, the second user interface may dynamically change the visual representation of the electrocardiogram signal in response to a third command selecting either a state in which noise present in the electrocardiogram signal is removed or a state in which noise is not removed.

Alternatively, the method may further comprise the step of providing a third user interface that lists electrocardiogram features defined by user commands without labeling the electrocardiogram signal.

Alternatively, the third user interface may include a fifth control graphic that dynamically changes its visual representation and state in response to a fourth command that determines the presence or absence of electrocardiogram features listed in the third user interface.

Alternatively, the fifth control graphic may be a graphic for determining the presence or absence of electrocardiogram features listed in the third user interface according to the fourth command.

Alternatively, the method may further include the step of providing a fourth user interface that displays the labeling status performed by the user.

Alternatively, the fourth user interface may include a sixth control graphic representing whether at least one electrocardiogram feature per lead is labeled.

According to one embodiment of the present disclosure for achieving the above-described task, a computer program stored in a computer-readable storage medium is disclosed. When the computer program is executed on one or more processors, it performs operations for providing a user interface for labeling of medical artificial intelligence, wherein the operations may include an operation for providing a first user interface for listing labelable electrocardiogram features on an electrocardiogram signal; and an operation for providing a second user interface for visually displaying the electrocardiogram signal.

According to one embodiment of the present disclosure for achieving the above-described task, a computing device providing a user interface for labeling of medical artificial intelligence is disclosed. The device may include a processor including at least one core; a memory including program codes executable by the processor; and an input/output unit providing a first user interface listing labelable electrocardiogram features on an electrocardiogram signal and a second user interface visually displaying the electrocardiogram signal.

The present disclosure provides a computing environment in which labeling work can be performed accurately and efficiently through a user interface that allows experts to perform labeling of electrocardiogram signals through simple operations according to features required for electrocardiogram interpretation and to intuitively understand the progress of labeling.

FIG. 1 is a block diagram of a computing device according to one embodiment of the present disclosure.
FIGS. 2 and 3 are conceptual diagrams showing screens implementing a user interface according to one embodiment of the present disclosure.
FIG. 4 is a flowchart illustrating a method for providing a user interface according to one embodiment of the present disclosure.
FIG. 5 is a block diagram showing a hardware environment according to one embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. The embodiments presented in the present disclosure are provided so that those skilled in the art can utilize or implement the contents of the present disclosure. Accordingly, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure may be implemented in various different forms and is not limited to the embodiments below.

Throughout the specification of the present disclosure, the same or similar drawing reference numerals refer to the same or similar components. In addition, in order to clearly describe the present disclosure, drawing reference numerals of parts that are not related to the description of the present disclosure may be omitted in the drawings.

The term "or" as used herein is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified herein or the context makes clear, "X employs either A or B" should be understood to mean either one of the natural inclusive permutations. For example, unless otherwise specified herein or the context makes clear, "X employs A or B" can be interpreted to mean either X employs A, X employs B, or X employs both A and B.

The term "and/or" as used herein should be understood to refer to and include all possible combinations of one or more of the associated concepts listed.

The terms "comprises" and/or "comprising" as used herein should be understood to mean the presence of particular features and/or components. However, it should be understood that the terms "comprises" and/or "comprising" do not exclude the presence or addition of one or more other features, other components, and/or combinations thereof.

Unless otherwise specified in this disclosure or unless the context makes it clear that the singular form is intended to be referred to, the singular should generally be construed to include “one or more.”

The term "Nth (N is a natural number)" used in the present disclosure can be understood as an expression used to mutually distinguish components of the present disclosure according to a predetermined standard such as a functional viewpoint, a structural viewpoint, or convenience of explanation. For example, components performing different functional roles in the present disclosure can be distinguished as a first component or a second component. However, components that are substantially the same within the technical spirit of the present disclosure but should be distinguished for convenience of explanation may also be distinguished as a first component or a second component.

The term "acquisition" as used in this disclosure may be understood to mean not only receiving data via a wired or wireless communication network with an external device or system, but also generating data in an on-device form.

Meanwhile, the term "module" or "unit" used in the present disclosure may be understood as a term referring to an independent functional unit that processes computing resources, such as a computer-related entity, firmware, software or a part thereof, hardware or a part thereof, a combination of software and hardware, etc. At this time, the "module" or "unit" may be a unit composed of a single element, or may be a unit expressed as a combination or set of multiple elements. For example, as a narrow concept, a "module" or "unit" may refer to a hardware element of a computing device or a set thereof, an application program that performs a specific function of software, a processing process implemented through software execution, or a set of instructions for program execution, etc. In addition, as a broad concept, a "module" or "unit" may refer to a computing device itself that constitutes a system, or an application that is executed on a computing device, etc. However, the above-described concept is only an example, and the concept of “module” or “part” may be variously defined within a category understandable to those skilled in the art based on the contents of the present disclosure.

The term "model" used in the present disclosure may be understood as a system implemented using mathematical concepts and language to solve a specific problem, a set of software units to solve a specific problem, or an abstract model regarding a processing process to solve a specific problem. For example, a neural network "model" may refer to the entire system implemented as a neural network that has a problem-solving ability through learning. In this case, the neural network may have a problem-solving ability by optimizing parameters connecting nodes or neurons through learning. A neural network "model" may include a single neural network, or may include a neural network set in which multiple neural networks are combined.

The term "control graphic" used in the present disclosure may be understood as a graphic object that executes a command and produces a visible result when manipulating a user interface. In other words, a "control graphic" may be understood as a basic unit that constitutes a user interface, and as a user interface element that displays content to be provided to a user or enables manipulation by a user.

The explanation of the terms set forth above is intended to aid in understanding the present disclosure. Therefore, if the terms set forth above are not explicitly stated as matters limiting the contents of the present disclosure, it should be noted that they are not used to limit the technical ideas of the contents of the present disclosure.

FIG. 1 is a block diagram of a computing device according to one embodiment of the present disclosure.

The computing device (100) according to one embodiment of the present disclosure may be a hardware device or a part of a hardware device that performs comprehensive processing and calculation of data, or may be a software-based computing environment connected to a communication network. For example, the computing device (100) may be a server that performs an intensive data processing function and shares resources, or may be a client that shares resources through interaction with a server. In addition, the computing device (100) may be a cloud system in which a plurality of servers and clients interact to comprehensively process data. Since the above description is only one example related to the type of the computing device (100), the type of the computing device (100) may be configured in various ways within a category that can be understood by those skilled in the art based on the contents of the present disclosure.

Referring to FIG. 1, a computing device (100) according to one embodiment of the present disclosure may include a processor (110), a memory (120), a network unit (130), and an input/output unit (140). However, FIG. 1 is only an example, and the computing device (100) may include other configurations for implementing a computing environment. In addition, only some of the disclosed configurations may be included in the computing device (100).

The processor (110) according to one embodiment of the present disclosure may be understood as a configuration unit including hardware and/or software for performing computing operations. For example, the processor (110) may process commands generated as a result of user interaction through a user interface. In addition, the processor (110) may read a computer program to perform data processing for machine learning. The processor (110) may process computational processes such as processing of input data for machine learning, feature extraction for machine learning, and error calculation based on backpropagation. The processor (110) for performing such data processing and operations may include a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). The type of processor (110) described above is only an example, and thus the type of processor (110) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

The processor (110) can generate a user interface for an electrocardiogram analysis expert to use for labeling. The user interface can include control graphics for outputting information required for performing labeling, receiving input according to user operation, or implementing functions required for labeling. For example, the user interface can include control graphics for visualizing data required for labeling, such as electrocardiogram features, electrocardiogram signals, etc. In addition, the user interface can include control graphics for inputting commands by a user for labeling electrocardiogram signals, and control graphics for guiding the user to input commands. Such control graphics can have visual representations or states changed by user operation.

The memory (120) according to one embodiment of the present disclosure may be understood as a configuration unit including hardware and/or software for storing and managing data processed in the computing device (100). That is, the memory (120) may store any type of data generated or determined by the processor (110) and any type of data received by the network unit (130). For example, the memory (120) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, a RAM (random access memory), a SRAM (static random access memory), a ROM (read-only memory), an EEPROM (electrically erasable programmable read-only memory), a PROM (programmable read-only memory), a magnetic memory, a magnetic disk, and an optical disk. In addition, the memory (120) may also include a database system that controls and manages data in a predetermined system. The type of memory (120) described above is only an example, and thus the type of memory (120) can be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

The memory (120) can manage, by structuring and organizing, data required for the processor (110) to perform operations, combinations of data, and program codes executable by the processor (110). For example, the memory (120) can store medical data received through the network unit (130) described below. The memory (120) can store program codes for storing rules for processing medical data, program codes for causing a neural network model to perform learning by receiving medical data, program codes for causing a neural network model to perform inference by receiving medical data in accordance with the intended use of the computing device (100), and processed data generated as the program codes are executed.

The network unit (130) according to one embodiment of the present disclosure may be understood as a configuration unit that transmits and receives data via any type of known wired and wireless communication system. For example, the network unit (130) may perform data transmission and reception using a wired and wireless communication system such as a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), wireless broadband internet (WiBro), fifth generation mobile communication (5G), ultra wide-band, ZigBee, radio frequency (RF) communication, wireless LAN, wireless fidelity, near field communication (NFC), or Bluetooth. Since the above-described communication systems are only examples, the wired and wireless communication system for data transmission and reception of the network unit (130) may be applied in various ways other than the above-described examples.

The network unit (130) can receive data required for the processor (110) to perform calculations through wired or wireless communication with any system or any client, etc. In addition, the network unit (130) can transmit data generated through the calculation of the processor (110) through wired or wireless communication with any system or any client, etc. For example, the network unit (130) can receive biometric data through communication with a cloud server that performs tasks such as standardization of databases and medical data in a hospital environment, a client such as a smart watch, or a medical computing device, etc. The network unit (130) can transmit output data of a neural network model, intermediate data, processed data, etc. derived from the calculation process of the processor (110), etc. through communication with the aforementioned database, server, client, or computing device, etc.

The input/output unit (140) according to one embodiment of the present disclosure may be understood as a configuration unit including hardware and/or software for implementing a user interface. That is, the input/output unit (140) may visualize and output any type of data generated or determined by the processor (110) and any type of data received by the network unit (130). In addition, the input/output unit (140) may receive a user input that generates a command to be transmitted to any system or any client connected to the processor (110), the computing device (100), or the like via wired or wireless communication. For example, the input/output unit (140) may include a display module that may output visualized information or implement a touch screen, such as a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, etc. In addition, the input/output unit (140) may include an input module capable of recognizing actions such as a user's motion, voice, etc., such as a camera, a microphone, a keyboard, a mouse, etc. The modules described above are only examples, and thus, the modules included in the input/output unit (140) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure, in addition to the examples described above.

The input/output unit (140) implements a user interface, and can output graphics generated by the processor (110) or receive user commands generated by user manipulation and transmit them to the processor (110). For example, the input/output unit (140) can output graphics for receiving user commands regarding specific functions, graphics for providing information, etc., in a specific area of the user interface. In addition, the input/output unit (140) can receive user commands regarding graphics output in a specific area of the user interface. At this time, the user command regarding a specific graphic can be understood as an input signal generated by a user manipulation of a specific graphic. In addition, the user manipulation of a specific graphic can mean actions that a user can perform through the input/output unit (140), such as clicking, double-clicking, hovering, flicking, pinching, and spreading of a specific graphic. The input/output unit (140) can receive user commands and transmit them to the processor (110), thereby enabling operations for providing a user interface based on user control to be performed.

FIGS. 2 and 3 are conceptual diagrams showing screens implementing a user interface according to one embodiment of the present disclosure.

According to one embodiment of the present disclosure, the first user interface (210) may configure an area that lists electrocardiogram features that can be labeled on an electrocardiogram signal. The electrocardiogram features listed in the first user interface (210) may be information necessary for electrocardiogram interpretation, such as an electrocardiogram waveform or interval. For example, as shown in FIG. 2, the electrocardiogram features in the first user interface (210) may be expressed by dividing them into two layers. Electrocardiogram features such as a P wave, a QRS wave, a T wave, an F wave, noise, a pacing spike, etc. may be expressed as an upper layer in the first user interface (210). And, detailed features such as a starting point, a peak, an end point, a Q wave, an R wave, an S wave, etc. in the upper layer may be expressed as a lower layer in the first user interface (210).

The first user interface (210) may include a first control graphic (211) that dynamically changes a visual representation and a state in response to a first command for selecting an electrocardiogram feature to be labeled among electrocardiogram features that can be labeled on an electrocardiogram signal. The first control graphic (211) may be configured for each electrocardiogram feature listed in the first user interface (210). The first control graphic (211) may be a button-type graphic that may accept a user command for selecting a specific electrocardiogram feature. For example, the first control graphic (211) may be included in a lower layer for an upper layer such as a P wave, a QRS wave, a T wave, an F wave, etc. If there is no first command for selecting a specific feature to be labeled among the electrocardiogram features listed in the first user interface (210), the first control graphic (211) may be maintained in an inactive state without a color change. When the first command is input, the first control graphic (211) may change color and switch to an activated state. At this time, the activated state may be understood as a state in which a specific feature selected by a user command from among the electrocardiogram features listed in the first user interface (210) may be labeled on the electrocardiogram signal. Meanwhile, the first command for the first control graphic (211) may be input by a user operation, such as an action of clicking the first control graphic (211), an action of double-clicking, or an action of selecting a shortcut key assigned to the first control graphic (211).

The first user interface (210) can display electrocardiogram features as in the example described above, thereby allowing the user to intuitively understand the electrocardiogram features that are labeling targets and easily determine the labeling target among the electrocardiogram features. In addition, the first user interface (210) can provide an environment in which the user can easily select or replace the labeling target through the first control graphic (211) as in the example described above. Such improvement in usability can prevent unnecessary operations by the user, thereby improving the efficiency of computing resources.

The second user interface (220) can configure an area that visually displays an electrocardiogram signal. When electrocardiogram data is acquired by a user command, the second user interface (220) can upload the electrocardiogram signal included in the electrocardiogram data and express it in a graph form. In the second user interface (220), the electrocardiogram signal can be aligned over time, and the position or size at which it is exposed can be changed in response to a user command. At this time, the user command is generated by a user operation performed within the area where the electrocardiogram signal is expressed, and the user operation can include an action such as a drag or a scroll.

The second user interface (220) may include a second control graphic (221) displayed on the electrocardiogram signal in response to a second command for labeling an electrocardiogram feature selected according to a first command in the first user interface (210) on the electrocardiogram signal. When a specific feature is selected in the first user interface (210), a ready state for labeling the specific feature on the electrocardiogram signal displayed in the second user interface (220) may be activated. At this time, when a second command for selecting a specific point on the electrocardiogram signal displayed in the second user interface (220) is input, the second control graphic (221) may be generated at the point where the second command is input. For example, when the first control graphic (211) is activated by the first command and a cursor for inputting a user command is positioned on the electrocardiogram signal displayed in the second user interface (220), the shape and color of the cursor may be changed. When the shape and color of the cursor are changed, the user can label the feature selected by the first command on the ECG signal through the second command. Specifically, when the cursor, whose shape and color are changed by the user operation, is positioned at point A on the ECG signal and the second command is input, the second control graphic (221) can be displayed at point A on the ECG signal. At this time, the second command can be input by a user operation, such as an action of clicking point A, an action of double-clicking point A, or an action of selecting a shortcut key assigned to the first control graphic (211).

Meanwhile, when a plurality of second control graphics exist on the electrocardiogram signal, a second control graphic existing in a waveform related to an electrocardiogram feature corresponding to a first control graphic that is activated in response to a first command among the plurality of second control graphics may be activated on the electrocardiogram signal. For example, when second control graphics for the P wave, QRS wave, and T wave exist on the electrocardiogram signal, and the first control graphic for the starting point of the P wave is activated by the first command, only the second control graphic for the P wave may be activated, and the second control graphics for the remaining QRS waves and T waves may be deactivated. At this time, the activation of the second control graphic may be understood as a state that a user can visually recognize on the electrocardiogram signal. Conversely, the deactivation of the second control graphic may be understood as a state that exists on the electrocardiogram signal but cannot be visually recognized by the user. That is, when labeling is performed on an electrocardiogram signal for multiple features, the second user interface (220) can selectively visualize the second control graphic (221) depending on the activation state of the first control graphic (211) so that the user can smoothly check the labeling status for each electrocardiogram feature. This function allows the user to smoothly distinguish labels existing on the electrocardiogram signal for each feature and perform labeling, thereby reducing unnecessary user operations and improving efficiency of computing resources.

The second user interface (220) may include a third control graphic displayed on the ECG signal based on a label for an ECG feature predicted by inputting an ECG signal on which the second control graphic (221) is not displayed or an ECG signal on which the second control graphic (221) is displayed into a pre-learned artificial intelligence model. In order to enable a user to perform more accurate labeling, the second user interface (220) may display a third control graphic for suggesting or recommending an estimated label using artificial intelligence on the ECG signal. Specifically, when an ECG signal on which the second control graphic (221) is not displayed is input into the artificial intelligence model and labels for ECG features to be displayed are derived from the ECG signal, the third control graphic may be generated on the ECG signal based on the labels, which are predicted values of the artificial intelligence model. At this time, the label for the ECG feature, which is the predicted value of the artificial intelligence model, may be information on the location of the ECG feature to be displayed on the ECG signal. Accordingly, the third control graphic can express the location of the electrocardiogram feature predicted by the artificial intelligence model on the electrocardiogram signal in order to guide the input of the second command. In other words, the third control graphic can display the predicted value of the artificial intelligence model at a specific point on the electrocardiogram signal in a form corresponding to the second control graphic (221). At this time, the second control graphic (221) and the third control graphic can be visually distinguished from each other based on differences in color, brightness, saturation, etc. In addition, the third control graphic can also express error information of the electrocardiogram feature specified through the second control graphic (221) based on the label predicted by the artificial intelligence model. That is, when there is a label predicted by the artificial intelligence model and the second control graphic (221) by the second command based on a specific electrocardiogram feature, the third control graphic can express error information generated as a result of comparing the location based on the label and the location of the second control graphic (221). Users can perform labeling more accurately and easily through third control graphics that visualize labels predicted by artificial intelligence in this way or express errors in user labels. Improving labeling accuracy through third control graphics can reduce unnecessary work that may occur due to incorrect labeling in the repetitive labeling process, thereby improving computing resource efficiency.

Meanwhile, the artificial intelligence model used to generate the third control graphic may be pre-trained to receive an electrocardiogram signal and extract electrocardiogram features present in the electrocardiogram signal. For example, the artificial intelligence model may use training data including an electrocardiogram signal with labeled electrocardiogram features for training. The artificial intelligence model may receive an electrocardiogram signal included in the training data, extract electrocardiogram features, and generate output data including information on the type and location of the electrocardiogram features. The artificial intelligence model may perform training by adjusting parameters that configure the model through an optimizer based on an error between output data produced using a loss function and labels included in the training data. At this time, the artificial intelligence model may be a neural network model based on a convolutional neural network, a transformer, or the like. The types and learning methods of the artificial intelligence model described above are only examples, and the artificial intelligence model of the present disclosure may be configured in various ways within a category understandable to those skilled in the art based on the contents of the present disclosure.

The second user interface (220) may include a fourth control graphic that represents location information of a point where a second command is input on an electrocardiogram signal or a guide line to assist labeling on an electrocardiogram signal. If a user performs labeling solely based on the shape of an electrocardiogram signal, the probability of an error increases. Therefore, in order to reduce such errors, the fourth control graphic may provide auxiliary information that the user can utilize when performing labeling. For example, when a cursor for inputting a second command is positioned on an electrocardiogram signal, the fourth control graphic may represent a guide line representing information or a visual reference of the corresponding location on the electrocardiogram signal. The user may perform labeling more accurately and easily through the fourth control graphic that provides auxiliary information that assists labeling in this way. The improvement of labeling accuracy through the fourth control graphic may reduce unnecessary work that may occur due to incorrect labeling during a repetitive labeling process, thereby improving computing resource efficiency.

The second user interface (220) can dynamically change the visual representation of the electrocardiogram signal in response to a third command that selects either a state in which noise is removed from the electrocardiogram signal or a state in which noise is not removed. Depending on the situation, the user may need to check the raw signal before noise is removed. For example, among the electrocardiogram features, pacing spikes, which are indicated by a machine that artificially generates electrical signals for heartbeats in the heart, may disappear due to noise removal. Therefore, if labeling is performed based on the signal from which noise is removed, the user may not be able to label the pacing spikes. To solve this problem, the second user interface (220) can selectively visualize the signal from which noise is removed or the signal from which noise is not removed through the third command. At this time, the third command may be input by a user operation, such as an action of clicking an additional control graphic provided in the second user interface or another user interface, or an action of selecting a shortcut key assigned to the second user interface.

Meanwhile, in the ECG signal displayed on the second user interface (220), ECG features may overlap with each other. Therefore, in order to accurately label the point where multiple features overlap, the second user interface (220) may be assigned a shortcut key that performs a function of overlapping ECG features. The user may input a command to the second user interface (220) by selecting a shortcut key that performs the function of overlapping ECG features. Then, the second user interface (220) may display the multiple features in the ECG signal by overlapping them in response to the command. For example, the R wave endpoint and the S wave starting point may overlap and exist in the ECG signal. In this case, when the user selects the shortcut key assigned to the second user interface (220), the second user interface (220) may label the S wave starting point at the same location as the already labeled R wave endpoint in response to the user command.

The third user interface (230) can configure an area that lists electrocardiogram features defined by a user command without labeling the electrocardiogram signal. Among the electrocardiogram features, there are features that can be defined as specific points on the electrocardiogram signal, such as the features listed in the first user interface (210), while there are also features that cannot be defined as specific points on the electrocardiogram signal. For example, features that cannot be defined as specific points on the electrocardiogram signal include notched or slurred R, which is used to identify left bundle branch block (LBBB), and delta waves that mainly appear in a disease called wolff-parkinson-white syndrome (WPW). Since these features are variant forms that appear in specific waveforms, they cannot be defined as specific points on the electrocardiogram signal. Therefore, the third user interface (230) can visualize these features so that they can be separately defined by a user command. And, the third user interface (230) can be categorized and listed by lead so that the electrocardiogram features can be defined by lead. The electrocardiogram features listed by the third user interface (230) may appear in different cases depending on the type of lead, and may not appear in all beats in one lead. Therefore, the electrocardiogram features listed by the third user interface can be displayed by distinguishing them by lead so that the presence or absence of each lead can be labeled.

The third user interface (230) may include a fifth control graphic (231) that dynamically changes a visual representation and state in response to a fourth command that determines the presence or absence of electrocardiogram features listed in the third user interface (230). The fifth control graphic (231) may be a graphic for determining the presence or absence of electrocardiogram features listed in the third user interface (230) in accordance with the fourth command. The fifth control graphic (231) may be configured for each electrocardiogram feature listed in the third user interface (230). The third control graphic (231) may be a button-type graphic that may accept a user command for determining the presence or absence of a specific electrocardiogram feature. For example, the third control graphic (231) may include sub-graphics that enable the presence or absence to be individually determined. That is, the third control graphic (231) may include a sub-graphic defining a state that is not a labeling target (“-”), a sub-graphic defining the presence of an electrocardiogram feature (“O” in FIG. 2), and a sub-graphic indicating that an electrocardiogram feature does not exist (“X” in FIG. 2). Each of the sub-graphics may be maintained in an inactive state without a color change if there is no fourth command. If the fourth command is input for any one of the three sub-graphics, the sub-graphic to which the fourth command is input may have its color changed and may be switched to an active state. Depending on the state defined by the sub-graphic switched to the active state, the presence or absence of an electrocardiogram feature corresponding to the third control graphic (230) may be determined. That is, if the fourth command is input for a sub-graphic defining the presence of a delta wave, the delta wave may be labeled as existing on an electrocardiogram signal. Meanwhile, the fourth command for the third control graphic (231) can be input by a user operation such as an action of clicking the third control graphic (231), an action of double-clicking, or an action of selecting a shortcut key assigned to the third control graphic (231).

The third user interface (230) can provide an environment in which a user can conveniently label various electrocardiogram features that can be used for electrocardiogram analysis in addition to basic electrocardiogram features by listing electrocardiogram features that are not displayed in the first user interface (210). In addition, the third user interface (230) can provide an environment in which a user can easily select or replace a labeling target through a third control graphic (231) such as the example described above. Such usability improvement can prevent unnecessary operations by the user, thereby bringing about efficiency in computing resources.

The fourth user interface (240) may configure an area that displays a labeling situation performed by a user. In other words, the fourth user interface (240) may configure a dashboard area that organizes the labeling situation performed up to the present point so that the user can grasp at a glance. In addition, the fourth user interface (240) may include a sixth control graphic (241) that expresses whether at least one electrocardiogram feature has been labeled for each lead. The sixth control graphic (241) may be a graphic that outputs information by reflecting the type and number of labels existing on the electrocardiogram signal displayed in the second user interface (220). For example, as shown in FIG. 3, the fourth user interface (240) may output a table in which electrocardiogram features and standard 12 leads are each configured in rows and columns. In addition, the sixth control graphic (241) may be displayed for each cell configured in rows and columns. If at least one electrocardiogram feature A is labeled in Lead I, the sixth control graphic existing in the cell where Lead I intersects the electrocardiogram feature A can be displayed in a shape that intuitively indicates its existence, such as an “O.” If at least one electrocardiogram feature A is not labeled in Lead I, the sixth control graphic existing in the cell where Lead I intersects the electrocardiogram feature A can be displayed in a shape that intuitively indicates its non-existence, such as an “X.” That is, the sixth control graphic (241) can dynamically change its visual expression and color to reflect the presence or absence of the label. If the sixth control graphic (241) does not intuitively express the presence or absence in this way but instead shows the number of labeling, the number of bits varies for each electrocardiogram and the total number cannot be determined, which causes a problem in that the user loses the meaning of identifying the labeling status. Therefore, in order to make it easy for the user to intuitively identify the entire labeling status, the sixth control graphic (241) provides information in a way that expresses the presence or absence of the label.

FIG. 4 is a flowchart illustrating a method for providing a user interface according to one embodiment of the present disclosure.

A computing device (100) according to one embodiment of the present disclosure can obtain an electrocardiogram signal according to a user command (S100). For example, when a user command for selecting electrocardiogram data including an electrocardiogram signal to be labeled is input, the computing device (100) can receive electrocardiogram data to be labeled through communication with a medical database or an electrocardiogram signal measuring device. If an electrocardiogram signal is already stored in the memory (120) of the computing device (100), the computing device (100) can also perform a task of transferring the electrocardiogram data stored in the memory (120) to the processor (110).

The computing device (100) can provide a first user interface that lists electrocardiogram features that can be labeled on the electrocardiogram signal (S200). The computing device (100) can provide a second user interface that visually displays the electrocardiogram signal (S300). In addition, the computing device (100) can provide a third user interface that lists electrocardiogram features defined by a user command without labeling the electrocardiogram signal (S400). As shown in FIG. 3, the first user interface, the second user interface, and the third user interface can be combined and provided on a single screen configured through the input/output unit (140) of the computing device (100). Through the user interfaces provided in this manner, the labeling process can proceed as follows.

When an ECG signal is acquired, the user can select any one of a plurality of features exposed through the first user interface. For example, if the user wants to specify the start point, peak, and end point of the P wave, the user can select the "(Q)P" button graphic at the top of the first user interface with the cursor. Then, the "(Q)P" button graphic can be activated by changing its color to blue. The user can also select the "(Q)P" button graphic by pressing the "Q" key on the keyboard, which is a shortcut key to activate the button.

After the button graphic of the first user interface is activated, when the user places the cursor over the electrocardiogram signal displayed on the second user interface, the shape of the cursor may change. For example, when the button graphic of the first user interface is not activated, the cursor may be in the shape of an arrow. However, when the button graphic of the first user interface is activated, the cursor may change from the shape of an arrow to a red cross when positioned over the electrocardiogram signal.

When the button graphic of the first user interface is activated, if the user places the cursor on the electrocardiogram signal, additional information such as the x-axis value or y-axis value of the corresponding position is displayed next to the cursor in the second user interface. In addition, if the user selects a specific point of the electrocardiogram signal using the cursor or performs an action of pressing a shortcut key on the specific point of the electrocardiogram signal, the electrocardiogram feature corresponding to the activated button graphic of the first user interface can be labeled. For example, when labeling the starting point of the P wave, the user can place the cursor on the corresponding position on the electrocardiogram signal and then press the shortcut key "1" assigned to the starting point to store the location information of the starting point of the P wave. In order to label the peak of the P wave, the user can place the cursor on the corresponding position on the electrocardiogram signal and then press the shortcut key "2" assigned to the peak to store the location information of the P wave peak. The designated position can be displayed as a circular dot on the electrocardiogram signal according to each matching color.

Meanwhile, since features listed through the third user interface, such as delta waves, are not displayed on the ECG signal, their presence or absence can be labeled through button graphics of the third user interface. For example, a user can label the presence or absence of an ECG feature for an ECG signal by selecting any one of the button graphics represented as “O” (present), “X” (absent), and “-” (no specified value) of the third user interface.

The computing device (100) may provide a fourth user interface that displays a labeling situation performed by a user (S500). The fourth user interface may not be provided simultaneously like the first user interface, the second user interface, and the third user interface, but may be selectively provided according to a specific user command. For example, when a user selects a button graphic displayed in the middle of a screen where the first user interface, the second user interface, and the third user interface are provided using a cursor, the fourth user interface may be exposed on the screen through a method such as a dropdown or a popup. Through the fourth user interface that is selectively exposed in this way, the user can grasp the labeling status by electrocardiogram feature and lead at a glance.

FIG. 5 is a block diagram showing a hardware environment according to one embodiment of the present disclosure.

Referring to FIG. 5, a system according to an embodiment of the present disclosure may include a client (300) that provides a user interface to a user, and a server (400) that processes information obtained through the user interface or information provided as a user interface. For example, the client (300) may correspond to the computing device (100) of FIG. 1. The client (300) may control graphics implemented in the user interface using at least one of a command or data obtained through interaction with a user. In addition, the client (300) may transmit at least one of a user command or data obtained through the user interface to the server (400). The server (400) may perform processing on information to be provided through the user interface using at least one of a command or data obtained through the user interface. Specifically, the server (400) may generate visualized information to be provided to the user through the user interface using an artificial intelligence model. The server (400) may transmit the processed information to the client (300), and the client (300) may provide the processed information to the user through the user interface.

The various embodiments of the present disclosure described above can be combined with additional embodiments and can be modified within a range that can be understood by those skilled in the art in light of the detailed description set forth above. It should be understood that the embodiments of the present disclosure are illustrative in all respects and not restrictive. For example, each component described as a single component may be implemented in a distributed manner, and likewise, components described as distributed may be implemented in a combined manner. Accordingly, all changes or modifications derived from the meaning, scope, and equivalent concept of the claims of the present disclosure should be interpreted as being included in the scope of the present disclosure.

Claims (15)

A method for providing a user interface for labeling of medical artificial intelligence, performed by a computing device including a processor,
A step of providing a first user interface listing labelable electrocardiogram features on an electrocardiogram signal;
providing a second user interface for visually displaying the electrocardiogram signal; and
Providing a third user interface that lists electrocardiogram features defined by user commands without labeling the electrocardiogram signal;
Including,
method.
In paragraph 1,
The above first user interface is,
Including a first control graphic that dynamically changes the visual representation and state in response to a first command for selecting an electrocardiogram feature to be labeled among the electrocardiogram features that can be labeled on the electrocardiogram signal.
method.
In the second paragraph,
The second user interface is,
Including a second control graphic displayed on the electrocardiogram signal in response to a second command for labeling the electrocardiogram feature selected in accordance with the first command on the electrocardiogram signal;
method.
In the third paragraph,
If there are multiple second control graphics on the above ECG signal,
Among the plurality of second control graphics, a second control graphic present in a waveform related to an electrocardiogram characteristic corresponding to a first control graphic that is activated in response to the first command is activated on the electrocardiogram signal.
method.
In the third paragraph,
The second user interface is,
An electrocardiogram signal in which the second control graphic is not displayed or an electrocardiogram signal in which the second control graphic is displayed is input to a pre-trained artificial intelligence model, and based on a label for a predicted electrocardiogram feature, a third control graphic is displayed on the electrocardiogram signal.
method.
In paragraph 5,
The above third control graphic is,
To guide the input of the second command, the location of the predicted electrocardiogram feature is specified and expressed on the electrocardiogram signal, or
Expressing error information of a specified electrocardiogram feature through the second control graphic, which is calculated based on the predicted label,
method.
In the third paragraph,
The second user interface is,
Including a fourth control graphic representing location information of a point where the second command is input on the electrocardiogram signal or a guide line to assist labeling on the electrocardiogram signal.
method.
In paragraph 1,
The second user interface is,
In response to a third command selecting either a state in which noise present in the electrocardiogram signal is removed or a state in which the noise is not removed, dynamically changing the visual representation of the electrocardiogram signal.
method.
delete In paragraph 1,
The third user interface is,
In response to a fourth command determining the presence or absence of electrocardiogram features listed in the third user interface, a fifth control graphic dynamically changing the visual representation and state,
method.
In Article 10,
The above fifth control graphic is,
A graphic for determining the presence or absence of electrocardiogram features listed in the third user interface according to the fourth command above.
method.
In paragraph 1,
Step of providing a fourth user interface that displays the labeling status performed by the user;
Including more,
method.
In Article 12,
The above fourth user interface is,
Including a sixth control graphic representing whether at least one electrocardiogram feature per lead is labeled;
method.
A computer program stored in a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs operations that provide a user interface for labeling of medical artificial intelligence.
The above actions are,
An operation providing a first user interface listing labelable electrocardiogram features on an electrocardiogram signal;
An operation of providing a second user interface for visually displaying the electrocardiogram signal; and
An operation of providing a third user interface that lists electrocardiogram features defined by user commands without labeling the electrocardiogram signal;
Including,
Computer program.
A computing device providing a user interface for labeling medical artificial intelligence,
A processor comprising at least one core;
a memory containing program codes executable by the processor; and
An input/output unit providing a first user interface listing electrocardiogram features that can be labeled on an electrocardiogram signal, a second user interface visually displaying the electrocardiogram signal, and a third user interface listing electrocardiogram features defined by a user command without being labeled on the electrocardiogram signal;
Including
device.