KR102832752B1 - Method for predicting fractional flow reserve based on vascular oct image and system for the same - Google Patents

Abstract

A device for predicting fractional blood flow reserve based on a blood vessel OCT (Optical Coherence Tomography) image, comprising: a medical image acquisition unit for acquiring a medical image set including a plurality of OCT images for a first location to a second location of a blood vessel; a three-dimensional patch segmentation unit for segmenting the medical image set into a plurality of patches of a fixed size based on a first method; an artificial intelligence model unit including an artificial intelligence model for predicting fractional blood flow reserve for the first location to the second location based on the plurality of patches; and a medical information provision unit for providing the predicted fractional blood flow reserve.

Description

Method for predicting fractional flow reserve based on vascular OCT image and system thereof {METHOD FOR PREDICTING FRACTIONAL FLOW RESERVE BASED ON VASCULAR OCT IMAGE AND SYSTEM FOR THE SAME}

The present invention relates to a method and system for predicting fractional flow reserve based on vascular OCT images. Specifically, the present invention relates to a method for predicting fractional flow reserve of a corresponding blood vessel by analyzing the relationship between a series of OCT images based on an artificial intelligence model including a transformer.

In modern medicine, vascular diseases, especially cardiovascular diseases, are considered one of the major causes of death worldwide. Early diagnosis and treatment of cardiovascular diseases greatly improve the survival rate of patients, and various diagnostic devices and methods are being developed for this purpose. Among them, optical coherence tomography (OCT) can provide detailed images of the inside of the cardiovascular system through high resolution, and plays an important role in evaluating lesions and structures inside the blood vessels.

OCT technology is a device that can image the inside of blood vessels in real time, and boasts a very high resolution compared to existing ultrasound or X-rays. OCT uses laser light to capture cross-sections of tissues, and through this, the thickness of blood vessel walls can be accurately determined. Due to these characteristics, OCT is particularly useful in the diagnosis and treatment of coronary artery disease.

Meanwhile, FFR (Fractional Flow Reserve) is an important diagnostic indicator used to help determine treatment for patients with cardiovascular disease, especially coronary artery disease (CAD). It is an indicator that evaluates how much a specific stenotic (narrowed) area of a coronary artery actually affects the myocardium (heart muscle) supplied through that blood vessel. FFR is derived by measuring the pressure inside the coronary artery. Mainly, a catheter is used to measure the blood pressure in front of and behind the narrowed area of the coronary artery, and the FFR value is calculated through the ratio of the two pressures obtained as a result. The FFR value usually has a value between 0 and 1, and the closer the FFR is to 1, the more the coronary artery is diagnosed as normal.

However, the traditional method of measuring FFR has the disadvantage of increasing the procedure time because it requires a separate drug such as adenosine or nicorandil to temporarily increase blood flow in order to accurately measure FFR, and a separate catheter and pressure measurement guide wire to be inserted to measure for more than a few minutes.

However, the traditional method of measuring fractional flow reserve requires a temporary increase in blood flow using drugs such as adenosine and Nicorandil, which can be fatal for patients who are sensitive to adenosine or experience side effects, and can cause inaccurate evaluations along with criticism that it is drug-dependent. In addition, there is a disadvantage in that the procedure time increases because a separate catheter and pressure measurement guide wire must be inserted to measure more than a few minutes.

Therefore, the present invention proposes a technique for predicting FFR based on OCT images acquired with a single pullback without insertion of a separate pressure sensor induction wire.

A related prior literature is Republic of Korea Patent Publication No. 10-2023-0094295 (June 28, 2023).

The purpose of the present invention is to propose a method and system for predicting fractional blood flow reserve based on vascular OCT images.

In order to solve the above-described problem, a device for predicting fractional blood flow reserve based on a vascular OCT (Optical Coherence Tomography) image (hereinafter, referred to as a device according to embodiments) is disclosed. The device according to embodiments includes a medical image acquisition unit for acquiring a medical image set including a plurality of OCT images for a first location to a second location of a blood vessel; a three-dimensional patch segmentation unit for segmenting the medical image set into a plurality of patches of a fixed size based on a first method; an artificial intelligence model unit including an artificial intelligence model for predicting fractional blood flow reserve for the first location to the second location based on the plurality of patches; and a medical information provision unit for providing the predicted fractional blood flow reserve; wherein the artificial intelligence model unit includes a plurality of transformer models.

A first transformer model according to embodiments includes an embedding unit that converts each of the divided patches into a C-dimensional vector; a first multi-head self-attention module that generates intermediate feature data representing a spatial relationship between the embedded vectors; and a first multi-artificial neural network layer unit that learns weight information for learning the spatial relationship.

Furthermore, the first transformer model according to the embodiments further includes a second multi-head self-attention module that generates final feature data representing the spatial relationship of the segmented patches based on the second method of the medical image set or the intermediate feature data. Here, the second method may be a method of shifting the medical image set or the intermediate feature data and segmenting the shifted result according to the first method.

Furthermore, the first transformer model according to the embodiments may further include a dimension merging unit that merges at least some of the generated final feature data and re-embeds the merged result into a C-dimensional vector.

Additionally, the dimension merging unit according to the embodiments concatenates eight adjacent feature data among the generated final feature data.

Meanwhile, the artificial intelligence model according to the embodiments is a model learned based on learning data, and the learning data includes OCT images corresponding to the first location to the second location of the blood vessel and label information indicating the fractional blood flow reserve value for the first location to the second location of the blood vessel.

Furthermore, the artificial intelligence model according to the embodiments learns the first multiple artificial neural network layer unit and the second multiple artificial neural network layer unit included in the plurality of transformer models by using a loss function derived from the absolute value of the difference between the fractional blood flow reserve value predicted based on the OCT images in the learning data and the correct fractional blood flow reserve value included in the label information.

Meanwhile, the artificial intelligence model unit according to the embodiments includes the first transformer model to the fourth transformer model, and the first multiple artificial neural network layer unit or the second multiple artificial neural network layer unit of the first transformer model includes two artificial neural network layers, the first multiple artificial neural network layer unit or the second multiple artificial neural network layer unit of the second transformer model includes two artificial neural network layers, the first multiple artificial neural network layer unit or the second multiple artificial neural network layer unit of the third transformer model includes six artificial neural network layers, the first multiple artificial neural network layer unit or the second multiple artificial neural network layer unit of the fourth transformer model includes two artificial neural network layers, and the embedding unit converts each of the divided patches into a 96-dimensional vector.

The system according to the embodiments enables effective learning from detailed features to overall contextual information within a medical image set due to this configuration, thereby processing complex image data and enabling high-accuracy prediction.

The system according to the embodiments integrates global context information by considering correlations between patches divided in different ways as well as correlations between predetermined patches in a medical image set by passing patches divided in different ways through a transformer block, and effectively learns relationships between input patches to derive more sophisticated high-dimensional features.

The system according to the embodiments maintains the most suitable computational speed of the transformer modules by normalizing (N, W, H) = (128, 128, 128).

Figure 1 is a system diagram showing an overall system for analyzing and providing vascular medical information (hereinafter referred to as “vascular medical information providing system according to embodiments”).
Figure 2 is a configuration diagram showing a vascular medical image analysis device according to embodiments.
Figure 3 is an example of the configuration of a data preprocessing unit and an artificial intelligence model unit in a system or server according to embodiments.
FIG. 4 is another example of the configuration of a data preprocessing unit and an artificial intelligence model unit in a system or server according to embodiments.
Figure 5 shows a specific structure of a transformer block according to embodiments.
Figure 6 illustrates the structure of medical image data to be input into the artificial intelligence model unit according to embodiments.
FIG. 7 illustrates the most desirable structure of medical image data to be input into an artificial intelligence model unit according to embodiments and a method for generating the most desirable data structure.
FIG. 8 illustrates an example of a screen of an interface or software provided to a user by a vascular medical image analysis device according to embodiments.
FIG. 9 illustrates another example of a screen of an interface or software provided to a user by a vascular medical image analysis device according to embodiments.
FIG. 10 is a flowchart showing a process for predicting fractional blood flow reserve by a vascular medical image analysis device according to embodiments.
FIG. 11 is an example of a configuration diagram of a system or server according to embodiments.

The present invention can have various modifications and various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, but should be understood to include all modifications, equivalents, or substitutes included in the spirit and technical scope of the present invention. In describing each drawing, similar reference numerals are used for similar components.

Although the terms first, second, A, B, etc. may be used to describe various components, the components should not be limited by the terms. The terms are used only to distinguish one component from another. For example, without departing from the scope of the present invention, the first component could be referred to as the second component, and similarly, the second component could also be referred to as the first component. The term "and/or" includes any combination of a plurality of related listed items or any item among a plurality of related listed items.

When it is said that a component is "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. On the other hand, when it is said that a component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprises" or "has" and the like are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms defined in commonly used dictionaries, such as those defined in common dictionaries, should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense unless expressly defined in this application.

Figure 1 is a system diagram showing an overall system for analyzing and providing vascular medical information (hereinafter, “vascular medical information providing system according to embodiments”).

A vascular medical information providing system according to embodiments may include one or more medical devices (101) for photographing the inside or outside of a blood vessel, a catheter (102) linked to the medical device (101), an AI-based medical image analysis system (103), an image management system (104), and a user terminal device (105) providing a user interface, each of which may communicate with each other.

Referring to FIG. 1, one or more medical devices (101) are coupled with a catheter (102) to capture an image (100a) of the inside of a blood vessel of a patient (100). The one or more medical devices (101) may be an OCT image (100a) based on an optical coherence tomography (OCT) technique. The catheter (102) may be an imaging catheter configured to capture an OCT image, and may continuously communicate with one or more medical devices (101) to capture an image of the inside of a blood vessel of a patient (100).

Referring to FIG. 1, one or more medical devices (101) can capture the inside of a patient's cardiovascular system to obtain multiple medical images (100a). For example, one or more medical devices (101) can capture the inside of a patient's cardiovascular system at the end of the catheter (102) in conjunction with the movement of the catheter (102), and can obtain medical images (100a) according to the location of the catheter (102) inside the cardiovascular system.

One or more medical devices (101) transmit/transmit multiple acquired medical images (100a) to an AI-based medical image analysis system (103), and the AI-based medical image analysis system (103) can analyze multiple medical images (100a).

The AI-based medical image analysis system (103) can capture or analyze medical information (or information related to diagnosis/disease) such as the size, shape, presence of sclerosis, biochemical information, tissue characteristics, phenotype, presence of plaque, presence of calcification, and various diagnosis-related indicators (e.g., fractional flow reserve (FFR), etc.) inside the cardiovascular system of a patient (100).

Meanwhile, the cardiovascular medical image analysis system according to the embodiments may include an image management system (104) configured to systematically store or manage medical images (100a) of a patient (100) so that they can be reanalyzed or reused at the request of a patient (100) or medical staff (106). One or more medical devices (101) may transmit captured medical images (100a) of a patient (100) to the image management system (104).

The cardiovascular medical image analysis system according to the embodiments can transmit medical information analyzed by the AI-based medical image analysis system (103) to medical staff (106). Specifically, the cardiovascular medical image analysis system according to the embodiments can transmit the analyzed medical information to an interface or terminal device (105) available to the medical staff (106), and the medical information can be processed in the form of a diagnosis report (105a), etc. and provided to the medical staff (106).

With this configuration, the cardiovascular medical image analysis system according to the embodiments can effectively reduce the burden and effort of medical staff in diagnosis. In addition, with this configuration, the cardiovascular medical image analysis system according to the embodiments can reduce unnecessary efforts of medical staff, minimize possible misdiagnosis, and provide the effect of improving the quality of essential medical care.

Meanwhile, the 'blood vessel' generally described in this specification may refer to various blood vessels existing within the body, and may refer to a blood vessel capable of taking an OCT image (i.e., a blood vessel whose interior can be imaged using an imaging catheter).

Meanwhile, a preferred example of 'blood vessel' described in this specification generally refers to the cardiovascular system. At this time, the 'cardiovascular system' is a blood vessel attached to the heart, and is composed of major blood vessels that help the heart function, and plays a role in supplying blood to the heart and circulating blood throughout the body. Major cardiovascular systems include coronary arteries, aorta, pulmonary arteries, pulmonary veins, and vena cava.

Although the above description is based on the cardiovascular system, it may not be limited to the cardiovascular system and may be applied to various blood vessels that can be imaged with OCT.

Below, starting with Fig. 2, the specific configuration and function of the AI-based medical image analysis system (103) (or its device) of Fig. 1 are described.

Figure 2 is a configuration diagram showing a vascular medical image analysis device according to embodiments.

The device shown in Fig. 2 may be a component of the AI-based medical image analysis system (103) of Fig. 1, and represents the configuration of a vascular medical image analysis device (hereinafter referred to as “device according to embodiments”) (200). Referring to Fig. 2, the device (200) according to embodiments may include at least one of a medical image acquisition unit (201), a data preprocessing unit (202), an artificial intelligence model unit (203), a learning unit (204), a visualization unit (205), and a medical information provision unit (206).

A medical image acquisition unit (201) according to embodiments acquires a set of medical images captured by one or more medical devices. The set of medical images includes a plurality of medical images. Each medical image may be, for example, an OCT image based on an optical coherence tomography (OCT) technique.

The medical image set may mean, for example, OCT images captured for a specific section of a blood vessel. The medical image set may mean OCT images acquired when pulling back from a first point to a second point of the blood vessel. That is, the medical image acquisition unit (201) may acquire a plurality of medical images captured at a certain interval for each location within the blood vessel as a bundle.

Hereinafter, in this specification, the width of each medical image in the medical image set is defined as W, and the height is defined as H. Unlike general computer vision where the dimension is defined as three, such as RGB, OCT medical images are generally composed of only black and white, so they can be composed of only one channel. In addition, the medical image set is defined as including N medical images (OCT images). In this specification, the medical image set is defined as having the size of (N, W, H).

Meanwhile, the medical image set acquired by the medical image acquisition unit (201) may include only OCT images, but may also include data in which the intravascular shooting location may be annotated.

The data preprocessing unit (202) according to the embodiments can preprocess or process a set of medical images acquired by the medical image acquisition unit (201). The data preprocessing unit (202) according to the embodiments can extract features identified in the image(s). For example, if the medical image is an OCT image, the feature extraction unit (202) can extract a catheter region identified in the OCT image, a region representing a shadow created by the catheter, a region covered by the catheter, a lumen region, a bifurcation or branch region.

The data preprocessing unit (202) may include a configuration that performs a rule-based algorithm to extract some features from a medical image, and may include an artificial neural network such as a CNN (Convolutional Neural Network).

In addition, the data preprocessing unit (202) may select OCT images equal to the number of the first value among the plurality of OCT images when the number of OCT images included in the medical image set is greater than the first value (e.g., 128 or 256), and resize the width and height of the selected OCT images by the first value pixels. Specific details related thereto will be described in detail in FIG. 7.

The data preprocessing unit (202) can perform regularization or vector embedding into a vector composed of multiple dimensions in order to input the acquired medical image into the artificial intelligence model (203). The data preprocessing unit (202) can convert the acquired medical image into a compatible or suitable data structure in order to input the acquired medical image into the artificial intelligence model (203).

The data preprocessing unit (202) can, for example, partition the acquired medical image (or medical image bundle) into a plurality of patches, and transmit the divided patches as input data of the artificial intelligence model (203). At this time, a patch refers to data expressed by concatenating pixels in the divided image, and this can be expressed in a vector form. In addition, at this time, the data preprocessing unit (202) can divide the acquired medical image into a plurality of patches according to various preset methods. The method by which the data preprocessing unit (202) divides the medical image (or medical image bundle) into a plurality of patches (or a plurality of 3D vectors) will be described in detail with reference to FIGS. 5 and 6.

Specifically, the data preprocessing unit (202) may partition a medical image according to embodiments into a plurality of detailed image patches so that the image may be input to one or more transformer blocks included in the artificial intelligence model unit (203). In addition, the data preprocessing unit (202) may embed each of the divided patches into a vector so that the image may be input to one or more transformer blocks.

The artificial intelligence model unit (203) can receive medical image(s) acquired by the medical image acquisition unit (201) and/or feature(s) extracted by the feature extraction unit (202) as input and predict the fractional flow reserve (FFR) for the corresponding blood vessel or blood vessel section.

The artificial intelligence model unit (203) receives embedded results, i.e. vectors, for each patch segmented by the data preprocessing unit (202) and predicts FFR values for sections of blood vessels corresponding to the medical image set.

The artificial intelligence model unit (203) may include a plurality of transformer blocks. For example, the artificial intelligence model unit (203) first divides a medical image set of size (N, W, H) into sizes (2, 4, 4) (i.e., height and width are four basic units, and four OCT images). The artificial intelligence model unit (203) obtains N/4 × H/4 × W/4 3D tokens by embedding or normalizing each of the divided patches. Thereafter, the artificial intelligence model unit (203) projects each 3D token into C (C is an arbitrary number) dimensions to generate C-dimensional feature data. Thereafter, the artificial intelligence model unit (203) collects C-dimensional feature data corresponding to each patch, passes the collected data through the first transformer block, and generates relationship information for each patch. As a result of passing through the first transformer block, the artificial intelligence model section (203) derives output feature data indicating the relationship between N/4 × H/4 × W/4 3D tokens.

Meanwhile, each transformer block includes a first multi-head self-attention module and a second multi-head self-attention module, i.e., two multi-head self-attention modules. In addition, each transformer block may include a plurality of neural network layers that can be updated by feedforward and backpropagation, and a layer normalization module, etc. Here, the two multi-head self-attention modules may be connected serially. For example, the first transformer block generates C-dimensional feature data for patches divided into (4, 4, 4)-sized units, passes the data through the first multi-head self-attention module, and generates C-dimensional intermediate output features for each patch. At this time, the patches divided into (4, 4, 4) size units can be patches divided in a general manner (i.e., divided to have a constant width and height based on the upper leftmost point as a reference point) of the image image. Thereafter, the first transformer block can shift the intermediate output features, divide the shifted intermediate output features in a similar manner as above, and pass the patches generated as a result of the division to the second multi-head self-attention module. The first transformer block generates C-dimensional final output features for each patch as a result of passing through the second multi-head self-attention module.

Thereafter, the first transformer block merges adjacent output features among the N/4 × H/4 × W/4 output features. For example, the first transformer block can derive N/8 × H/8 × W/8 output features of 8C dimensions by merging adjacent output features among the N/4 × H/4 × W/4 output features, but the merge may be performed based on other methods and numbers. The first transformer block normalizes and/or embeds each of the N/8 × H/8 × W/8 output features of 8C dimensions into a C dimension. That is, the first transformer block finally derives N/8 × H/8 × W/8 output features of C dimensions.

Thereafter, the artificial intelligence model unit (203) inputs each of the N/8 × H/8 × W/8 output features normalized/embedded in the C dimension as described above into the second transformer block. The second transformer block may also include a first multi-head self-attention module and a second multi-head self-attention module, and are connected in series as described above. The second transformer block receives each of the N/8 × H/8 × W/8 output features normalized/embedded in the C dimension as input, and generates N/8 × H/8 × W/8 final output features in the C dimension (i.e., relationships between each of the N/8 × H/8 × W/8 patches). Thereafter, the second transformer block merges the N/8 × H/8 × W/8 final output features of C dimensions to generate N/16 × H/16 × W/16 8C-dimensional output features, and reduces the dimension of the generated output features to generate N/16 × H/16 × W/16 2C-dimensional output features (preferably having a dimension twice the size of the dimension of the output features of the first transformer).

The artificial intelligence model unit (203) can pass multiple transformer blocks through the above method until one patch is merged into one entire image set. The artificial intelligence model unit (203) can perform feed forward by dividing a medical image set with the above configuration, deriving a relationship between each of the divided patches, and merging the derived feature data to derive a relationship between the merged patches, thereby finally predicting the FFR value.

When learning learning data, the artificial intelligence model unit (203) can perform feed-forward on the learning data in the above manner and perform back-propagation based on the feed-forward final prediction result and label information to update the neural network layers included in the artificial intelligence model unit (203).

Meanwhile, although not shown in FIG. 2, the device (200) according to the embodiments may further include a first classification model that predicts information about the location and/or shape of a stent within a blood vessel based on a plurality of OCT images, a second classification model that predicts information about the location and distribution of an artifact within a blood vessel based on a plurality of OCT images, a third classification model that predicts information about the location and distribution of a specific lesion (such as a plaque) within a blood vessel based on a plurality of OCT images, and a fourth classification model that predicts information about the location and distribution of calcification within a blood vessel based on a plurality of OCT images. All or part of the first to fourth classification models may be artificial intelligence models composed of artificial neural networks capable of performing object detection by performing one or more layers or a pooling operation.

The learning unit (204) learns the artificial intelligence model(s) included in the artificial intelligence model unit (203) and/or the feature extraction unit (202) according to the embodiments. The learning unit (204) learns the artificial intelligence model(s) included in the artificial intelligence model unit (203) and/or the feature extraction unit (202) based on learning data including medical image(s) and correct answers (label information) for the corresponding medical images.

The learning unit (204) can generate learning data based on a medical image (or a group of medical images) to be used as learning data. The learning unit (204) adds label information to the medical image (or a group of medical images) in conjunction with the data preprocessing unit (202), divides the medical image (or a group of medical images) into a plurality of patches according to various preset methods, and trains the artificial intelligence model unit (203) based on the divided and/or preprocessed information.

The learning unit (204) can likewise learn each or all of the first to fourth classification models according to the embodiments.

The visualization unit (205) provides the fractional flow reserve (FFR) derived and predicted by the artificial intelligence model unit (203) to the terminal device of the medical staff or the user, and visualizes it. The medical information provision unit (206) provides the medical staff with information related to diagnosis based on the fractional flow reserve (FFR) predicted by the artificial intelligence model unit (203).

With this configuration, the vascular medical image analysis system according to the embodiments can effectively reduce the burden and effort of medical staff in diagnosis. In addition, with this configuration, the vascular medical image analysis system according to the embodiments can reduce unnecessary efforts of medical staff, minimize possible misdiagnosis, and provide the effect of improving the quality of essential medical care.

In addition, with this configuration, the vascular medical image analysis system according to the embodiments can reduce the effort of medical staff to reach the location of the blood vessel where the stent is to be implanted with a catheter (such as a balloon catheter).

Below, FIGS. 3 to 6 describe in detail the structure and input data of the artificial intelligence model unit (203) according to embodiments.

Figure 3 is an example of the configuration of a data preprocessing unit and an artificial intelligence model unit in a system or server according to embodiments.

Specifically, FIG. 3 shows a configuration of a method in which a system according to embodiments acquires a medical image set (300), i.e., OCT images, a 3D patch segmentation unit (301) segments them into a plurality of patches, and an artificial intelligence model unit (302) receives each of the segmented patches as input to ultimately predict fractional flow reserve (FFR) for a blood vessel section.

Meanwhile, Fig. 3 shows an example in which only one transformer block described in Fig. 2 is implemented in the artificial intelligence model section and one transformer block is repeatedly used.

Referring to FIG. 3, a medical image set (300) refers to medical images acquired by the medical image acquisition unit (200) of FIG. 2, and may refer to OCT images of blood vessels or cardiovascular systems.

Referring to FIG. 3, the 3D patch segmentation unit (301) performs part or all of the operations of the data preprocessing unit (202) of FIG. 2. The 3D patch segmentation unit (301) segments a medical image set into a plurality of patches. Here, the size of the patch can be set by the administrator, but in this specification, it is assumed that the OCT image frame, width, and height are each (4, 4, 4) in size and explained.

Referring to FIG. 3, the artificial intelligence model unit (302) receives the patches divided as described above and performs embedding, transformer block passing, and dimension merging multiple times. The artificial intelligence model unit (302) includes an embedding unit (302a), a transformer block (302b), and a dimension merging unit (302c).

The embedding unit (302a) performs embedding on each of the segmented patches received from the 3D patch segmentation unit (301). Each patch is a patch segmented into units of size (4, 4, 4) in the OCT image, and the embedding unit (302a) projects this into a C-dimensional vector. For example, if a medical image set is segmented into N/4 × H/4 × W/4 patches having a size of (4, 4, 4), the embedding unit (302a) converts each patch into a C-dimensional vector, thereby generating N/4 × H/4 × W/4 C-dimensional embedding vectors. In this process, the embedded vector is converted into a form suitable as an input for an artificial intelligence model, and is ready to be processed in a subsequent transformer block.

The transformer block (302b) receives the C-dimensional embedding vector generated from the embedding unit (302a) as input and derives the relationship between patches. The transformer block generates a new feature vector that considers the interaction and spatial relationship between each patch through a first multi-head self-attention module and a second multi-head self-attention module. For example, the transformer block (302b) performs an attention operation on the input N/4 × H/4 × W/4 C-dimensional vectors to generate new C-dimensional output vectors that reflect the interaction between patches. Thereafter, the transformer block shifts the intermediate output feature vector according to a certain rule and uses the shifted vector as an input to the transformer block again to perform additional learning.

The dimension merging unit (302c) reduces the dimension by merging adjacent vectors among the output vectors generated from the transformer block (302b). For example, among the N/4 × H/4 × W/4 C-dimensional output vectors, adjacent vectors can be merged to reduce the dimension to N/8 × H/8 × W/8 2C-dimensional vectors. The merged vector is repeatedly processed through the embedding unit (302a) and the transformer block (302b), and this process is performed until one patch is merged into one entire image set. Finally, the dimension merging unit (302c) generates a final FFR prediction vector for one image. This prediction vector is used to predict the fractional flow reserve (FFR) of a vascular segment for the entire medical image set.

FIG. 4 is another example of the configuration of a data preprocessing unit and an artificial intelligence model unit in a system or server according to embodiments.

FIG. 3 illustrates an example of a system according to embodiments in which one transformer block is used repeatedly several times within an artificial intelligence model section. Below, FIG. 4 illustrates an example of a system according to embodiments in which a plurality of transformer blocks are provided within an artificial intelligence model section, and embedding and dimension merging are configured to be performed according to individual settings.

Referring to FIG. 4, the medical image set (400) refers to medical images acquired by the medical image acquisition unit (200) of FIG. 2, and may refer to OCT images of blood vessels or cardiovascular systems. Referring to FIG. 4, the 3D patch segmentation unit (401) performs part or all of the operations of the data preprocessing unit (202) of FIG. 2.

Referring to FIG. 4, the artificial intelligence model unit (302) includes a plurality of stage units (402-1, 402-2, 402-3, …, 402-N). Each stage unit receives feature data or patches derived from a previous stage, performs embedding, transformer block passing, and dimension merging, and transfers the data to the next stage unit or derives a final FFR value. Each stage unit (402-1, 402-2, 402-3, …, 402-N) includes an embedding unit (402-1a, …, 402-Na), a transformer block (402-1b, …, 402-Nb), and a dimension merging unit (402-1c, …, 402-Nc). The embedding unit (402-1a, …, 402-Na), the transformer block (402-1b, …, 402-Nb), and the dimension merging unit (402-1c, …, 402-Nc) respectively refer to the embedding unit (302a), the transformer block (302b), and the dimension merging unit (302c) mentioned in Fig. 3.

Figure 5 shows a specific structure of a transformer block according to embodiments.

Figure 5 shows a specific structure of a transformer block according to embodiments.

The transformer block (50) can be included in the artificial intelligence model section (203) of Fig. 2, and refers to the transformer block (302b) of Fig. 3 and the transformer blocks (402-1b, ..., 402-Nb) of Fig. 4.

The transformer block (50) includes a first multi-head self-attention module (500) and a second multi-head self-attention module (501). The transformer block (50) learns relationships between input 3D patches and generates output vector data through this.

A first multi-head self-attention module (500) is described. The first multi-head self-attention module (500) includes a first layer normalization unit (500a), a first 3D multi-head self-attention unit (500b), a second layer normalization unit (500c), and a multi-artificial neural network layer unit (500d).

The first layer normalization unit (500a) performs normalization on the input N/4 × H/4 × W/4 C-dimensional vectors to adjust the distribution of the data. Thereafter, the vector normalized by the first layer normalization unit (500a) is transferred to the first 3-dimensional multi-head self-attention unit (500b). The first 3-dimensional multi-head self-attention unit (500b) learns the local spatial relationship of each segmented patch. For example, the first 3-dimensional multi-head self-attention unit (500b) performs a 3-dimensional attention operation between patches segmented into a size of (4, 4, 4) to generate a new C-dimensional vector including interaction information between the patches.

The second layer normalization unit (500c) normalizes the generated vector again and transfers it to the multi-artificial neural network layer unit (500d). The multi-artificial neural network layer unit (500d) may be a neural network model in which multiple neural network layers are connected. The artificial neural network layer unit (500d) may preferably be composed of two feed-forward networks and includes a module that performs a nonlinear activation function and dropout, etc.

Meanwhile, in the first layer normalization unit (500a), residual connection can be performed to prevent the vanishing gradient problem after passing through the first 3D multi-head self-attention unit (500b) and the multi-artificial neural network layer unit (500d).

The intermediate output features derived by passing through the first multi-head self-attention module (500) are input again, and the intermediate output features are shifted and passed through the second multi-head self-attention module (501). The second multi-head self-attention module (501) again passes through the first layer normalization unit (501a), the three-dimensional multi-head self-attention unit (501b), the second layer normalization unit (501c), and the multiple artificial neural network layer unit (501d).

Here, the second 3D multi-head self-attention unit (501b) included in the second multi-head self-attention module (501) additionally performs a window shift operation to combine local information learned in the previous step with new global context information. That is, the second 3D multi-head self-attention unit (501b) is configured to perform multi-head self-attention based on the result of additionally performing a window shift operation, unlike the first 3D multi-head self-attention unit (500b).

This configuration allows the transformer block according to the embodiments to learn the relationships between patches more deeply and helps generate C-dimensional output vectors that reflect consistent spatial information within the entire medical image set.

Thereafter, the output vector is normalized in the second layer normalization unit (501c) and converted into a final output vector through the multi-layer unit (501d). This final output vector is repeatedly used in the subsequent transformer block or passed to the final prediction step.

The system according to the embodiments enables effective learning from detailed features to overall contextual information within a medical image set due to this configuration, thereby processing complex image data and enabling high-accuracy prediction.

Figure 6 illustrates the structure of medical image data to be input into the artificial intelligence model unit according to embodiments.

Specifically, FIG. 6 illustrates a method of dividing a medical image set according to embodiments. FIG. 6(A) illustrates dividing a medical image set according to embodiments according to a first method, which is a general method, and FIG. 6(B) illustrates dividing a medical image set according to embodiments according to a second method, which is a shifted method.

Some or all of the operations shown in Fig. 6 may be performed in the data preprocessing unit (202) and/or the artificial intelligence model unit (203) of Fig. 2, the 3D patch segmentation unit (301) and/or the transformer block (302b) of Fig. 3, the first 3D MSA unit (500b) and the second 3D MSA unit (501b) of Fig. 5. Figs. 6(A) and 6(B) illustrate different methods of segmenting a medical image set.

Meanwhile, the operation shown in Fig. 6(A) can be performed in the first 3D MSA section (500b) of Fig. 5, and the operation shown in Fig. 6(B) can be performed in the second 3D MSA section (501b) of Fig. 5.

For example, according to one embodiment, the system may first pass each of the first patch sets, which are obtained by segmenting the medical image set according to the first method, through a first multi-head self-attention module (500), and derive spatial associations (spatial relationships) between patches in the segmented first patch set. Thereafter, the system according to the embodiments may again pass each of the second patch sets, which are obtained by segmenting the medical image set according to the second method, through a second multi-head self-attention module (501), and derive spatial associations (spatial relationships) between patches in the second patch set according to the second method.

For example, according to another embodiment, the system may first pass the first patch sets, which are divided into a medical image set according to a first method, through a first multi-head self-attention module (500), so as to derive spatial correlations (spatial relationships) between patches in the divided first patch sets. Thereafter, the system may pass the second patch sets, which are divided (i.e., shifted) again according to a second method, through a second multi-head self-attention module (501), so as to derive spatial correlations (spatial relationships) between patches in the second patch sets, which are divided according to the second method, through a second multi-head self-attention module (501).

The system according to the embodiments integrates global context information by considering correlations between patches divided in different ways as well as correlations between predetermined patches in a medical image set by passing patches divided in different ways through a transformer block, and effectively learns relationships between input patches to derive more sophisticated high-dimensional features.

FIG. 6(A) illustrates an example in which a system according to embodiments divides a block consisting of a medical image set according to a first method to derive patches (600). Here, the first method is a method in which data including a width, a height, and a number of frames at a certain interval are defined as one patch based on the top, the side, and the first image frame (i.e., the position corresponding to (0, 0, 0)) of the block, and the data is divided according to a certain standard as described above.

FIG. 6(A) illustrates a system according to embodiments, for dividing a medical image set into a plurality of patches according to a first method, determining a basic unit size of a patch (i.e., a width, a width, and a number of frames of one patch). The size of the basic unit of a patch may be determined by a value obtained by dividing the width, a width, and a number of frames of the entire medical image set by an integer. Preferably, the size of the basic unit of a patch may be determined by a value obtained by dividing the width, a width, and a number of frames of the entire medical image set by a power of 2 (e.g., 8, 16, etc.).

FIG. 6(B) illustrates an embodiment in which a system according to embodiments divides a block consisting of a medical image set according to a second method to derive patches (601). Here, the second method is a method in which a portion of a medical image set is extracted by maintaining the size of a basic unit of a patch determined by the first method, but shifting the start position of the patch by a specific width, a specific width, and a specific number of frames.

FIG. 6(B) illustrates a system according to embodiments, for dividing a medical image set into a plurality of patches according to a second method, determining a basic unit size of a patch (i.e., a width, a width, and a number of frames of one patch). The size of the basic unit of a patch may be determined by a value obtained by dividing the width, a width, and a number of frames of the entire medical image set by an integer. Preferably, the size of the basic unit of a patch may be determined by a value obtained by dividing the width, a width, and a number of frames of the entire medical image set by a power of 2 (e.g., 8, 16, etc.).

Additionally, the degree of shift in the width direction (601a), the degree of shift in the height direction (601b), and the degree of shift in the frame direction (601c) can be determined based on the width of the entire medical image set, the number of widths, and frames.

Preferably, the degree of shifting in the width direction (601a) may be a value smaller than the width of the basic unit of the patch divided according to the first method, and more preferably may be one of 1/2, 1/3, 1/4, ..., 1/N of the width of the basic unit of the patch divided according to the first method. Preferably, the degree of shifting in the width direction (601b) may be a value smaller than the width of the basic unit of the patch divided according to the first method, and more preferably may be one of 1/2, 1/3, 1/4, ..., 1/N of the width of the basic unit of the patch divided according to the first method. Also preferably, the degree of shifting in the frame direction (601c) may be a value smaller than the number of frames of the basic unit of the patch divided according to the first method.

Most preferably, the degree of shifting in the width direction (601a) is 1/3 or 2/3 of the width of the basic unit of the patch divided according to the first method. Also most preferably, the degree of shifting in the width direction (601b) is 1/3 or 2/3 of the width of the basic unit of the patch divided according to the first method. Also most preferably, the degree of shifting in the frame direction (601c) is 1/2 of the number of frames of the basic unit of the patch divided according to the first method.

FIG. 7 illustrates the most desirable structure of medical image data to be input into an artificial intelligence model unit according to embodiments and a method for generating the most desirable data structure.

Specifically, FIG. 7 illustrates the most preferred examples of an embodiment in which a medical image set according to embodiments is divided according to the first method (FIG. 7(A)) and an embodiment in which the medical image set is divided according to the second method (FIG. 7(B)).

The most desirable medical image set includes medical images (OCT images) of a section from a proximal end to a distal end of a specific blood vessel, and each medical image is an image captured during a pullback operation of an imaging catheter. In addition, the size of the most desirable medical image set is as follows. The size of each image in the most desirable medical image set is 128 X 128 pixels (i.e., W=512, H=512). In addition, the number of image frames in the most desirable medical image set is 128. That is, the system according to the embodiments resizes the medical image set to a medical image set of (N, W, H) = (128, 128, 128).

The system according to the embodiments (in particular, the data preprocessing unit according to the embodiments) can maintain the most suitable operation speed of the transformer modules by standardizing (N, W, H) = (128, 128, 128).

For example, the system according to the embodiments (in particular, the data preprocessing unit according to the embodiments, etc.) may resize each medical image (OCT image) to a size of 128 X 128 before inputting the medical image set to the artificial intelligence model unit according to the embodiments. For example, the system according to the embodiments may resize each medical image (OCT image) to a size of 128 X 128 when each medical image (OCT image) is configured to a size of 512 X 512.

In addition, the system according to the embodiments (particularly, the data preprocessing unit according to the embodiments, etc.) may adjust the number of image frames to 128 before inputting the medical image set into the artificial intelligence model unit according to the embodiments. That is, if the number of frames in the medical image set is 128 or more, 128 of them may be evenly selected, and if the number of frames is 128 or less, some of them may be duplicated to generate 128 frames.

In addition, the system according to the embodiments (in particular, the data preprocessing unit according to the embodiments) can evenly select 256 frames among the medical image set when the number of frames in the medical image set is 256 or more, and select 128 frames by selecting only odd-numbered frames or even-numbered frames among the selected 256 frames.

Figures 7(A) and 7(B) show examples of patches segmented according to the most preferred method.

First, the system according to the embodiments performs the first stage (1 ^st stage). This is performed by the first stage unit (402-1) shown in Fig. 4, and the specific operation is as follows.

As shown in Fig. 7(A), a medical image set is divided (700a) into patches of size (N, W, H) = (4, 4, 4) by the first method, thereby deriving 32768 (32*32*32) patches. Here, each patch (700a) is made not to overlap with each other. Thereafter, the system according to the embodiments passes each of the patches (700a) through a first multi-head self-attention module, thereby deriving a local spatial relationship between pixels included in each patch.

Next, the system according to the embodiments shifts the medical image sets (or, 32768 intermediate feature data derived by passing through the first multi-head self-attention module) by the second method as shown in Fig. 7(B) to generate a new patch having the size of (N, W, H) = (4, 4, 4). Here, the shift means moving (702a, 702b, 702c) a part of the medical image set as shown in Fig. 7(B), and the system according to the embodiments divides the image (701) generated by moving again according to the first method. Here, the individual patches are made not to overlap each other. Thereafter, the system according to the embodiments passes each of the patches through the second multi-head self-attention module to derive the local spatial relationship (final feature data) between the pixels included in each new patch.

Here, the degree of shifting in the width direction (601a) is most preferably 1/3 or 2/3 of the width of the basic unit of the patch divided according to the first method. Also, most preferably, the degree of shifting in the width direction (601b) is 1/3 or 2/3 of the width of the basic unit of the patch divided according to the first method. Also, most preferably, the degree of shifting in the frame direction (601c) is 1/2 of the number of frames of the basic unit of the patch divided according to the first method.

In the following embodiments, the system merges (700b) eight adjacent final feature data (two adjacent patches in the width direction, two adjacent patches in the height direction, and two adjacent patches in the frame direction) within the final feature data derived by passing the second multi-head self-attention module. That is, the system according to the embodiments concatenates 32768 adjacent C-dimensional final feature data into eight units within the final feature data, and embeds 4096 concatenated feature data of the 8C dimensions back into the C-dimensional feature data. In addition, this is the result of dividing into patches of the size of (N, W, H) = (8, 8, 8) and is explained by 4096 (16*16*16) merged patches.

The system according to the following embodiments passes the local spatial relationship (final feature data) through the artificial neural network layers included in the transformer block (402-1b) of the first stage, thereby deriving 4096 (16*16*16) final feature data to be input to the second stage.

The system according to the following embodiments performs the ^second stage. This is performed by the second stage unit (402-2) shown in Fig. 4, and the specific operation is as follows.

First, 4096 (16*16*16) patches are derived, which are divided into patches of the size of (N, W, H) = (8, 8, 8). Here, each patch does not overlap with each other. Each patch according to the embodiments is passed through the first multi-head self-attention module, and the local spatial relationship between the pixels included in each patch is derived.

Next, the system according to the embodiments shifts the medical image sets (or, 4096 intermediate feature data derived by passing through the first multi-head self-attention module) by the second method as shown in Fig. 7(B) to generate a new patch of the size of (N, W, H) = (8, 8, 8). Thereafter, the system according to the embodiments passes each of the patches through the second multi-head self-attention module to derive local spatial relationships (final feature data) between pixels included in each new patch.

The system according to the embodiments herein merges (700b) eight adjacent final feature data (two adjacent patches in the width direction, two adjacent patches in the height direction, and two adjacent patches in the frame direction) within the final feature data derived by passing the data through the second multi-head self-attention module. That is, the system according to the embodiments concatenates 4096 adjacent C-dimensional final feature data into eight units within the final feature data, and embeds 512 concatenated feature data of 8C dimensions into 2C-dimensional feature data. Here, the number of dimensions of the feature data embedded in performing the second stage may be greater than the number of dimensions of the feature data embedded in performing the first stage, and preferably may be twice as large. This is the result of splitting into patches of size (N, W, H) = (16, 16, 16) and is explained by 512 (8*8*8) merged patches.

The system according to the following embodiments passes the local spatial relationship (final feature data) through the artificial neural network layers included in the transformer block (402-2b) of the second stage, thereby deriving 512 (8*8*8) final feature data to be input to the third stage.

The system according to the following embodiments performs the ^third stage. This is performed by the third stage unit (402-3) shown in Fig. 4, and the specific operation is as follows.

(N, W, H) = 512 (8*8*8) patches are derived, each of which is divided into patches of (16, 16, 16) size. Each of the patches according to the embodiments is passed through the first multi-head self-attention module, and the local spatial relationship between the pixels included in each patch is derived.

Next, the system according to the embodiments first shifts the medical image sets (or, 2048 intermediate feature data derived by passing through the first multi-head self-attention module) by the second method as shown in Fig. 7(B) to generate a new patch of the size of (N, W, H) = (16, 16, 16). Thereafter, the system according to the embodiments passes each of the patches through the second multi-head self-attention module to derive local spatial relationships (final feature data) between pixels included in each new patch.

In the following embodiments, the system merges (700b) eight adjacent final feature data (two adjacent patches in the width direction, two adjacent patches in the height direction, and two adjacent patches in the frame direction) within the final feature data derived by passing them through the second multi-head self-attention module. That is, the system according to the embodiments is the result of embedding 64 concatenated feature data of 16C dimensions into 4C dimension feature data by concatenating 512 adjacent final feature data of 2C dimensions into eight units within the final feature data. In addition, this is the result of dividing into patches of the size of (N, W, H) = (32, 32, 32) and is explained by 64 (4*4*4) merged patches.

The system according to the following embodiments passes the local spatial relationship (final feature data) through the artificial neural network layers included in the transformer block (402-3b) of the third stage, thereby deriving 64 (4*4*4) final feature data to be input to the fourth stage.

The system according to the following embodiments performs the fourth stage, which is ^the last stage. This is performed by the fourth stage unit (402-4) shown in Fig. 4, and the specific operation of the last stage is as follows.

(N, W, H) = 64 (4*4*4) patches divided into patches of size (32, 32, 32) are derived. Each of the patches according to the embodiments is passed through the first multi-head self-attention module, and the local spatial relationship between the pixels included in each patch is derived.

Next, the system according to the embodiments shifts the medical image sets (or, 64 intermediate feature data derived by passing through the first multi-head self-attention module) by the second method as shown in Fig. 7(B) to generate a new patch of the size of (N, W, H) = (32, 32, 32). Thereafter, the system according to the embodiments passes each of the patches through the second multi-head self-attention module to derive local spatial relationships (final feature data) between pixels included in each new patch.

The system according to the embodiments passes local spatial relationships (final feature data) through artificial neural network layers included in the transformer block (402-4b) of the fourth stage to derive one FFR value.

The system according to the embodiments enables effective learning from detailed features to overall contextual information within a medical image set due to this configuration, thereby processing complex image data and enabling high-accuracy prediction.

The system according to the embodiments integrates global context information by considering correlations between patches divided in different ways as well as correlations between predetermined patches in a medical image set by passing patches divided in different ways through a transformer block, and effectively learns relationships between input patches to derive more sophisticated high-dimensional features.

The system according to the embodiments maintains the most suitable computational speed of the transformer modules by normalizing (N, W, H) = (128, 128, 128).

FIG. 8 illustrates an example of a screen of an interface or software provided to a user by a vascular medical image analysis device according to embodiments.

Referring to FIG. 8, the vascular medical image analysis device according to the embodiments provides software that provides various functions to a user terminal through a user interface, and the screen shown in FIG. 8 shows an example of a UI displayed on a user terminal. The software shown in FIG. 8 helps the user to effectively analyze vascular images and utilize them for diagnosis.

The image (or UI) illustrated in Fig. 8 can be displayed through software installed in the terminal device (105) of the user or medical staff (106) of Fig. 1.

Referring to FIG. 8, 800 represents a main interface of software that appears on a user terminal. The main interface (800) includes a first component (801) for setting and editing layers, a second component (802) for displaying a medical image, a third component (803) for displaying a cross-section of a blood vessel, and a fourth component (804) which is an area for displaying bibliographic information about a set of medical images.

The first component (801) is a component configured to display or edit one or more layers (802a) in a medical image. The first component (801) may include one or more detailed interfaces that can set each layer (802a) to be visible or displayed opaquely or transparently. The first component (801) provides a function that allows a user to select and display or hide a specific layer (802a) of interest (e.g., lumen, stent, plaque, calcification, branch vessel, etc.) in a medical image.

The second component (802) represents an area that displays an actually captured medical image of a patient and one or more layers (802a) that are overlaid on the medical image. In this area, a selected layer can be highlighted and displayed as a translucent overlay. For example, the layer indicated by 802a corresponds to an area that represents an object detected by a vascular medical image analysis device according to embodiments as a specific layer selected by a user.

The third component (803) represents an image (map) that represents a cross-sectional view of a blood vessel. The third component (803) visually represents a cross-sectional image along the entire length of the blood vessel. At this time, when a user clicks (803a) a specific location within the third component (803), a device according to the embodiments (such as a user's terminal device or a vascular medical image analysis device) can display a cross-sectional image corresponding to that location on the second component (802). At this time as well, selected specific areas within the cross-sectional image are displayed as a semitransparent layer.

The fourth component (804) is an area that displays bibliographic information about a set of medical images and is located within a screen of an interface or software. The fourth component (804) may be located as a subcomponent within the first component (801) to the third component (803), or may be located separately from the first component (801) to the third component (803).

The fourth component (804) is an area that displays the value of the predicted branch flow reserve (FFR) according to the embodiments described in FIGS. 1 to 7.

Due to this configuration, the vascular medical image analysis device according to the embodiments helps the user to intuitively analyze the OCT image and provides a useful function for easily identifying the area of interest to establish a diagnosis and treatment plan. In addition, the user can select a specific location to check a detailed cross-sectional image, thereby enabling more accurate analysis.

FIG. 9 illustrates another example of a screen of an interface or software provided to a user by a vascular medical image analysis device according to embodiments.

Referring to FIG. 9, the vascular medical image analysis device according to the embodiments illustrates the UI/UX of software configured to enable a user to check and analyze OCT images.

The image (or UI) illustrated in Fig. 9 can be displayed through software installed in the terminal device (105) of the user or medical staff (106) of Fig. 1.

Referring to FIG. 9, 900 represents a main interface of software that appears on a user terminal. The main interface (900) includes a first component (901) for setting and editing layers, a second component (902) for displaying an OCT image for a specific location of a blood vessel, a third component (906) for displaying a cross-section of the blood vessel, and a fourth component (908) which is an area for displaying bibliographic information for a medical image set. The first component (901), the second component (902), the third component (906), and the fourth component (908) may respectively refer to the first component (801), the second component (802), the third component (803), and the fourth component (804) described in FIG. 9.

Meanwhile, the main interface (900) according to the embodiments may further include at least one of a fifth component (903) that displays an image of a different format (e.g., a FLIm (Fluorescence Lifetime) image) for a specific location of a blood vessel, a sixth component (904) that displays an enface map for an OCT image, and a seventh component (905) that displays an enface map for an image of a different format.

The fifth component (903) represents an image of a different format (e.g., a FLIm image) corresponding to the corresponding location in the blood vessel. That is, the fifth component (903) represents an image of a lumen of a different format corresponding to a specific location in the blood vessel, and the image visually provides biochemical characteristics of the location in the blood vessel, for example.

The sixth component (904) represents an enface map that synthesizes images of different formats for the entire blood vessel and projects them onto a plane to create a 2D image. The vertical axis of the enface map displayed in the fifth component is a color representation of a FLIm image of the lumen surface captured in a 360-degree direction centered on a camera positioned at the end of a guide wire or imaging catheter, and the horizontal axis represents the location of the imaging catheter or camera within the blood vessel. This map allows the entire structure of the blood vessel to be viewed at a glance and visually provides biochemical information for a specific location.

The seventh component (905) of FIG. 9 represents an in-face map that is expanded into a 2D image by synthesizing or projecting an OCT image of the entire blood vessel. The vertical axis of the in-face map displayed in the sixth component represents an OCT image of the lumen surface captured in a 360-degree direction centered on a camera located at the end of a guide wire or imaging catheter, and the horizontal axis represents the location of the imaging catheter or camera within the blood vessel. This map allows the entire structure of the blood vessel to be viewed at a glance and visually provides OCT information of a specific location.

Meanwhile, referring to FIG. 9, the main interface (900) according to the embodiments provides a function (907a, 907b) that allows the user to select a first location and a second location within a blood vessel. If the user selects the first location (907a) and the second location (907b), the system according to the embodiments predicts FFR information for the corresponding section based on the method shown in FIGS. 1 to 7. Thereafter, the system according to the embodiments displays the predicted FFR information at the location of the fourth component (908).

Due to this configuration, the vascular medical image analysis device according to the embodiments helps the user to intuitively analyze the OCT image, and is useful for easily identifying the area of interest to establish a diagnosis and treatment plan. In addition, the user can select a specific location to check a detailed cross-sectional image and images of different formats of the location, enabling more accurate analysis.

FIG. 10 is a flowchart showing a process for predicting fractional blood flow reserve by a vascular medical image analysis device according to embodiments.

Specifically, FIG. 10 is a flowchart of a method in which a vascular medical image analysis device trains an artificial intelligence model unit (203) according to embodiments and predicts fractional flow reserve (FFR) of a vascular section corresponding to a medical image set using the trained artificial intelligence model unit (203). Steps 1000 to 1004 are processes in which the vascular medical image analysis device trains the artificial intelligence model unit (203) according to embodiments, particularly, artificial neural network layers within the transformer block(s) within the artificial intelligence model unit (203), and may be performed by the learning unit (204) of FIG. 2. Steps 1005 to 1009 may be performed by the artificial intelligence model unit (203) of FIG. 2.

Referring to FIG. 10, the device according to the embodiments learns the artificial intelligence model unit (203) according to the embodiments based on learning data. Here, the learning data may include OCT images corresponding to the first location to the second location of the blood vessel and label information indicating the fractional blood flow reserve value for the first location to the second location of the blood vessel. That is, the device according to the embodiments obtains the learning data (1000) as described above.

Next, the device according to the embodiments may optionally verify (1001) the suitability of the medical image set of learning data, and may further extract (1002) feature information to more accurately learn the medical image set. For example, the device according to the embodiments may further extract feature information, such as a guidewire, a calcification area, an area representing a shadow caused by a catheter, an area covered by a catheter, a lumen area, a bifurcation or branch area, from the OCT image, and further annotate these to the learning data so that they can be utilized for learning an artificial neural network model.

Thereafter, the device according to the embodiments can first predict (1003) the FFR (Fractional Flow Reserve) value for the cardiovascular section based on the medical image set (and the feature information) using the learned or initialized artificial intelligence model. Thereafter, the device according to the embodiments can update (1004) the artificial intelligence model based on the predicted or identified result and the label information.

Meanwhile, referring to FIG. 10, the device according to the embodiments can obtain (1005) a medical image set including a plurality of OCT images for a first location to a second location of a second blood vessel. The device according to the embodiments can verify the suitability of the medical image set as in steps 1001 and 1002, and extract feature information from the medical image set (1007), thereby generating additional information that is additionally helpful for prediction.

Thereafter, referring to FIG. 10, the device according to the embodiments can divide the medical image set into a plurality of patches of a fixed size based on the first method (1008). The operation of step 1008 may mean, for example, the data preprocessing unit (202) of FIG. 2, the three-dimensional patch division unit (301) of FIG. 3, the three-dimensional patch division unit (401) of FIG. 4, and the operations described in FIGS. 6 and 7.

Thereafter, the vascular medical image analysis device according to the embodiments can predict the fractional blood flow reserve from the first location to the second location based on the plurality of patches.

FIG. 11 is an example of a configuration diagram of a system or server according to embodiments.

Referring to FIG. 11, the server (1100) includes an input unit (1110), an output unit (1120), a control unit (1130), a storage unit (1140), and a communication unit (1150).

The input unit (1110) receives commands or information from an administrator. The input unit (1110) may include one or more of a microphone and a key input unit for receiving audio signals.

The output unit (1120) outputs the command processing result or various information to the manager. For example, the output unit (1120) outputs information on the predicted result (for example, the value or range of fractional flow reserve (FFR)) based on the vascular OCT image. To this end, the output unit (1120) may include a display, a speaker, a haptic output unit (1120), and an optical output unit (1120), although not illustrated in the drawing. The display may be provided in the form of a flat panel display, a flexible display, an opaque display, a transparent display, electronic paper (E-paper), or any other form well known in the technical field to which the present invention belongs. A touch pad may be laminated on the display to form a touch screen, and a touch key may be implemented through this touch screen. In addition to the display and the speaker, the output unit (1120) may further include any other form of output means well known in the technical field to which the present invention belongs.

The control unit (1130) connects and controls components within the server (1100). For example, the control unit (1130) controls each component so that information generated from a system that predicts a result based on a vascular OCT image (e.g., a value or range of fractional flow reserve (FFR)) can be output through the output unit (1120). As another example, when judgment information is input by an administrator, the control unit (1130) generates a response signal including judgment information. The control unit (1130) may be configured to include a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an MCU (Micro Controller Unit), a GPU (Graphics Processing Unit), or any other form of processor well known in the art of the present invention.

The storage unit (1140) stores data, programs, applications, etc. required for the server (1100) to operate. The storage unit (1140) may include nonvolatile memory, volatile memory, a hard disk, an optical disk, a magneto-optical disk, or any type of computer-readable recording medium well known in the art to which the present invention pertains.

The communication unit (1150) communicates with a system or other system that predicts fractional flow reserve (FFR) information based on vascular OCT images via a wired or wireless network.

The embodiments of the present invention disclosed in this specification and drawings are only specific examples presented to easily explain the technical content of the present invention and to help understand the present invention, and are not intended to limit the scope of the present invention. It will be apparent to those skilled in the art that other modified examples based on the technical idea of the present invention can be implemented in addition to the embodiments disclosed herein.

Although the present invention has been described above with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various modifications and changes may be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below.

Claims (10)

In a device for predicting fractional blood flow reserve based on OCT (Optical Coherence Tomography) images for cardiovascular systems,
A medical image acquisition unit for acquiring a medical image set including multiple OCT images from a first location to a second location of the cardiovascular system;
A three-dimensional patch segmentation unit for segmenting the above medical image set into a plurality of patches of fixed size based on the first method;
A first classification model unit for detecting an area of cardiovascular calcification within the above medical image set;
A second classification model unit for detecting the location and shape of a stent installed in the cardiovascular system within the above medical image set;
An artificial intelligence model unit including an artificial intelligence model that predicts fractional blood flow reserve from the first location to the second location based on the plurality of patches and the detected calcification area; and
A medical information provider providing the predicted fractional blood flow reserve and the location and shape of the stent;
The above artificial intelligence model includes multiple transformer models,
The above artificial intelligence model is trained based on a training data set consisting of a learning OCT image for the cardiovascular system and an annotated calcified area in the learning OCT image, a guidewire of a catheter for photographing the cardiovascular system, and a shadow area of the catheter.
A device for predicting fractional blood flow reserve. In the first paragraph, the first transformer model
An embedding unit that converts each of the above-mentioned divided patches into a C-dimensional vector;
A first multi-head self-attention module that generates intermediate feature data representing the spatial relationship between the above embedded vectors; and
A first multi-artificial neural network layer unit for learning weight information for learning the above spatial relationship; including;
A device for predicting fractional blood flow reserve. In the second paragraph, the first transformer model
A second multi-head self-attention module for generating final feature data representing the spatial relationship of the segmented patches based on the second method, based on the medical image set or the intermediate feature data; further comprising;
A device for predicting fractional blood flow reserve. In the third paragraph,
The second method is a method of shifting the medical image set or the intermediate feature data and dividing the shifted result according to the first method.
A device for predicting fractional blood flow reserve. In the third paragraph, the first transformer model
A dimension merging unit further comprising: merging at least some of the generated final feature data and re-embedding the merged result into a C-dimensional vector;
A device for predicting fractional blood flow reserve. In the fifth paragraph, the dimension merging unit concatenates eight adjacent feature data among the generated final feature data.
A device for predicting fractional blood flow reserve. In the first paragraph,
The above artificial intelligence model is a model learned based on learning data.
The above learning data includes OCT images corresponding to the first to second locations of the cardiovascular system and label information indicating fractional blood flow reserve values for the first to second locations of the cardiovascular system.
A device for predicting fractional blood flow reserve. In Article 7,
The above artificial intelligence model section includes four transformer models,
Using a loss function derived from the absolute value of the difference between the fractional blood flow reserve value predicted based on the OCT images in the learning data and the correct fractional blood flow reserve value included in the label information, the first multi-artificial neural network layer and the second multi-artificial neural network layer included in each of the four transformer models are trained.
A device for predicting fractional blood flow reserve. In the second paragraph,
The above fractional blood flow reserve prediction device is,
A data preprocessing unit further comprising: a data preprocessing unit for selecting OCT images equal to the number of the first value among the plurality of OCT images when the number of OCT images included in the medical image set is greater than the first value, and resizing the width and height of the selected OCT images by the first value pixels;
A device for predicting fractional blood flow reserve. A method for predicting fractional flow reserve, performed by at least one processor included in a device for predicting fractional flow reserve based on an OCT (Optical Coherence Tomography) image for cardiovascular,
A step of acquiring a medical image set including multiple OCT images from a first location to a second location of the cardiovascular system;
A step of dividing the above medical image set into a plurality of patches of fixed size based on the first method;
A step of detecting an area of calcification of the cardiovascular system within the above medical image set;
A step of detecting the location and shape of the stent installed in the cardiovascular system within the above medical image set;
A step of predicting the fractional blood flow reserve from the first location to the second location based on the plurality of patches and the detected calcification area using an artificial intelligence model; and
A step of providing the predicted fractional blood flow reserve, the position and shape of the stent; including;
The above artificial intelligence model includes multiple transformer models,
The above artificial intelligence model is trained based on a training data set consisting of a learning OCT image for the cardiovascular system and an annotated calcified area in the learning OCT image, a guidewire of a catheter for photographing the cardiovascular system, and a shadow area of the catheter.
Method for predicting fractional blood flow reserve.

Images