KR102332472B1 - Tumor automatic segmentation using deep learning based on dual window setting in a medical image - Google Patents

Abstract

An automatic tumor segmentation method using deep learning based on a dual window setup in a medical image is provided. The automatic tumor segmentation method using deep learning based on the dual window setting in the medical image applies a first window setting and a second window setting to a two-dimensional medical image including a tumor, so as to apply a first window setting based on the two-dimensional medical image. generating a window image and a second window image; The first window image and the second window image are input to a first deep neural network and a second deep neural network, respectively, and a first pre-shape model and generating a second pre-shape model; generating a weighted shape-focused prior by combining the first pre-shape model and the second pre-shape model; and inputting the 3D medical image including the tumor and the weighted shape-centered dictionary model into a third deep neural network, and obtaining a segmentation result of the tumor through the third deep neural network. do.

Description

Tumor automatic segmentation method using deep learning based on dual window setting in medical images {TUMOR AUTOMATIC SEGMENTATION USING DEEP LEARNING BASED ON DUAL WINDOW SETTING IN A MEDICAL IMAGE}

The present invention relates to an automatic tumor segmentation method using deep learning based on dual window settings in a medical image, and more particularly, to a two-dimensional deep neural network by inputting a first window image and a second window image to which different window settings are applied. It relates to a method for automatically segmenting a tumor using the results and a three-dimensional deep neural network.

Cancer is the most common cause of death from disease. For patients who have received radiation therapy, which is one of the methods for treating cancer, it is essential to measure the tumor size in order to evaluate the cancer prognosis and treatment response. Although tumor size is quantified using (a) one-dimensional RECIST (b) two-dimensional WHO or (c) 3D volume, there is no substantiated evidence suggesting that line-length measurements may be misleading compared to 3D measurements of tumors. keeps coming out In addition, volumetric segmentation is required to analyze image features and determine tumor volume for treatment planning. However, the tumor depends on the location where the tumor is placed (e.g., upper and lower lungs, chest wall, mediastinum, base of the lung, pulmonary vessels), the size of the tumor (e.g., 1 cm to 18 cm), the shape of the tumor, and the type of tumor (e.g., solid, Since sub-solid and necrosis (necrosis) were very diverse, there was a problem in that it was difficult to perform the segmentation operation.

In addition, various window settings may be applied to the medical image according to an object to be checked in the image. For example, when applying a window setting for viewing the entire organ area to a medical image, it is difficult to check the exact boundary of a tumor included in an organ. There was a problem in that it was difficult to confirm contextual information between the tumor and the internal organs.

(Patent Document 0001) JP 4919408 (Patent Document 0002) US 8494238

The problem to be solved by the present invention is to determine the location, size, shape and type of a tumor by inputting two window images and a weighted shape-centered dictionary model and a 3D medical image generated through a 2D deep neural network into a 3D deep neural network. It is to provide a method of accurately dividing at the boundary with the surroundings regardless of

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

The automatic tumor segmentation method using deep learning based on the setting of two windows in a medical image for solving the above-described problem is a method for automatically segmenting a tumor by applying a first window setting and a second window setting to a two-dimensional medical image including a tumor, generating a first window image and a second window image based on the 3D medical image; The first window image and the second window image are input to a first deep neural network and a second deep neural network, respectively, and a first pre-shape model and generating a second pre-shape model; generating a weighted shape-focused prior by combining the first pre-shape model and the second pre-shape model; and inputting the 3D medical image including the tumor and the weighted shape-centered dictionary model into a third deep neural network, and obtaining a segmentation result of the tumor through the third deep neural network. do.

Other specific details of the invention are included in the detailed description and drawings.

According to the present invention as described above, it has various effects as follows.

The present invention can accurately perform automatic segmentation of tumors.

In addition, the present invention can consider global context information and local context information together by using a two-dimensional deep neural network and a three-dimensional deep neural network.

In addition, the present invention can overcome the low contrast between the tumor and the surrounding tissue by using two images to which different window settings are applied.

Also, according to the present invention, shape information of a tumor can be obtained by using a weighted shape-centered dictionary model.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a block diagram illustrating an apparatus for automatically dividing a tumor according to an embodiment of the present invention.
2 is a flowchart illustrating an automatic tumor segmentation method according to an embodiment of the present invention.
3 is an exemplary view for explaining a first window image and a second window image.
4 is a block diagram illustrating an automatic tumor segmentation method according to an embodiment of the present invention.
5 is an exemplary diagram for explaining a weighted shape-centered dictionary model according to an embodiment of the present invention.
6 and 7 are tables for explaining the effect of the automatic tumor segmentation method according to an embodiment of the present invention.
8 is an exemplary view for explaining the effect of the automatic tumor segmentation method according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully understand the scope of the present invention to those skilled in the art, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. As used herein, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other components in addition to the stated components. Like reference numerals refer to like elements throughout, and "and/or" includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first component mentioned below may be the second component within the spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used herein will have the meaning commonly understood by those of ordinary skill in the art to which this invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Spatially relative terms "below", "beneath", "lower", "above", "upper", etc. It can be used to easily describe the correlation between a component and other components. A spatially relative term should be understood as a term that includes different directions of components during use or operation in addition to the directions shown in the drawings. For example, when a component shown in the drawing is turned over, a component described as “beneath” or “beneath” of another component may be placed “above” of the other component. can Accordingly, the exemplary term “below” may include both directions below and above. Components may also be oriented in other orientations, and thus spatially relative terms may be interpreted according to orientation.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Semantic segmentation is a technology that divides semantically the same part and determines which category it belongs to, rather than just dividing it into similar regions in terms of certain characteristics or calculated properties, like general image segmentation. say

That is, it may correspond to a technique of classifying all pixels of an image to which category they belong within a predefined category, and may correspond to pixelwise classification.

Methods for semantic image segmentation are divided into two main categories. The first is a technique of extracting hand-crafted features from an input image, dividing the image into super-pixel units, and then segmenting the image based on meaning. More specifically, the user can design and extract features that can be clues by analyzing the given image data. Thereafter, segmentation may be performed in units of super pixels based on the pattern of the extracted features. This process can lead to improvements in accuracy and speed. Thereafter, semantic image segmentation may be performed using a support vector machine in units of each super-pixel to determine which category the corresponding pixel or super-pixel belongs to. This method has a disadvantage in that the range of system utilization is limited because a homemade feature must be redesigned every time the type of image input to the system changes.

The second is a technique for extracting features using deep learning and then classifying them in units of pixels based on this. As the performance of deep learning-based classification has been proven to be excellent, an approach using a convolutional neural network (CNN) structure has been proposed even in semantic-based image segmentation. Fully Convolutional Networks (FCNs), which have changed the CNN structure, show excellent performance in image segmentation. After segmentation in units of superpixels, a CNN filter is trained using a training dataset, and an image is segmented, followed by post-processing such as CRF (Conditional Random Field).

Hereinafter, segmentation described in the present invention may mean semantic image segmentation, and a deep learning-based semantic image segmentation method may be applied.

1 is a block diagram illustrating an apparatus for automatically dividing a tumor according to an embodiment of the present invention.

Referring to FIG. 1 , the automatic tumor segmentation apparatus 100 according to an embodiment of the present invention may segment a tumor from an acquired medical image. For example, the automatic tumor segmentation apparatus 100 may generate a weighted shape center model by inputting a first window image and a second window image based on a 2D medical image to a 2D deep neural network, and a weighted shape center model and a 3D medical image can be input to a 3D deep neural network to obtain a tumor segmentation result.

In an embodiment, the automatic tumor segmentation apparatus 100 may be an electronic device such as a server or a PC, and a dedicated program for setting an automatic tumor segmentation method may be installed. For example, the automatic tumor segmentation apparatus 100 obtains a medical image, sets a window, an image controller 110 capable of inputting a medical image to a deep neural network, and uses a two-dimensional deep neural network to machine the received medical image. A two-dimensional deep learning unit 120 that performs learning and generates a plurality of prediction maps, a probabilistic model generator 130 that generates a pre-shape model and a weighted shape-centered model, and a weighted shape-centered model and a 3D medical image 3 A 3D deep learning unit 140 that performs machine learning using a dimensional deep neural network and generates a tumor segmentation result, a tumor determination unit 150 that determines the prognosis of a tumor based on the tumor segmentation result, and a machine learning result , a database 160 capable of storing the structure and filter values of the deep neural network, patient data, tumor segmentation results, and the like into big data.

2 is a flowchart illustrating an automatic tumor segmentation method according to an embodiment of the present invention. 3 is an exemplary view for explaining a first window image and a second window image. 4 is a block diagram illustrating an automatic tumor segmentation method according to an embodiment of the present invention. 5 is an exemplary diagram for explaining a weighted shape-centered dictionary model according to an embodiment of the present invention. 6 and 7 are tables for explaining the effect of the automatic tumor segmentation method according to an embodiment of the present invention. 8 is an exemplary view for explaining the effect of the automatic tumor segmentation method according to an embodiment of the present invention.

2 to 8 , in an embodiment, in operation 21 , the image controller 110 may acquire a 2D medical image including a tumor. For example, the 2D medical image may include a plurality of axial images, a plurality of coronal images, and a plurality of sagittal images. For example, medical imaging includes two-dimensional medical images (eg, x-ray images, CT cross-sectional images, MRI cross-sectional images) and/or three-dimensional medical images (eg, CT images, MRI, PET images), There are no special restrictions on video. "Image" means a subject collected by computed tomography (CT), magnetic resonance imaging (MRI), ultrasound or any other medical imaging system known in the art. may be a medical image of A medical image is voxel data, and may include a plurality of slices, that is, a plurality of unit images.

In an embodiment, in operation 22 , the image controller 110 generates the first window image 41 and the second window image 42 by applying the first window setting and the second window setting to the 2D medical image, respectively. can do.

For example, adjusting a medical image for cancer diagnosis to an appropriate brightness and contrast is an essential process for accurate reading. Excessively bright or dark images, or images with excessive contrast, interfere with observing anatomical structures or lesions. It is important. Brightness and contrast can be adjusted mechanically on the monitor for reading, but in the case of cross-sectional images such as CT or MRI, the brightness and contrast of the image can be adjusted through windows setting. Window setting consists of window level and window width, and different window setting values can be set according to the organs to be viewed. For example, the window level may be a median value of gray scale, and the window width may be a range that can be expressed in gray scale, which is several levels of black and white. Specifically, the window level is directly related to the pre-sensitization coefficient, absorption value, and concentration. When the value is set low, it is easy to observe tissues with a low attenuation coefficient, such as air. easy to In addition, the window width is directly related to the contrast of the image, and when the value is set wide, the contrast of the image becomes worse, and when the value is set narrow, the contrast of the image becomes good. lose

In an embodiment, the first window setting may be a case in which the window width and window level are adjusted so that the entire region of the organ is clearly visible in the 2D medical image, and the second window setting is the area where the tumor is located in the 2D medical image. It may be the case that the window width and window level are adjusted so that the boundary between the tumor and the surrounding area (or the important area where the tumor is located) is clearly visible. For example, the first window setting may be a lung window setting, and the second window setting may be a mediastinal window setting.

In an embodiment, the first window image 41 may include a plurality of axial images, a plurality of coronal images, and a plurality of sagittal images to which the first window setting is applied, and the second window image 42 is the second window. It may include a plurality of axial images, a plurality of coronal images, and a plurality of sagittal images to which the settings are applied.

For example, the first window image 41 may be an image including the entire and wide area of the tumor (FIG. 3(a)), and the second window image 42 has a clear difference in brightness values between the tumor and surrounding structures. It may be an image ((b) of FIG. 3).

In an embodiment, the region of interest of the first window image 41 to which the first window setting is applied may be the entire region of the organ including the tumor, and the region of interest of the second window image to which the second window setting is applied. The region may be an important region in which the tumor is located. For example, the first window image may be a lung window setting image, and the second window image may be a mediastinal window setting image.

In an embodiment, in operation 23, the two-dimensional deep learning unit 120 generates the first window image 41 and the second window image 42 respectively for the first deep neural network 43 and the second deep neural network ( 44) can be entered. For example, a plurality of axial images, a plurality of coronal images, and a plurality of sagittal images included in the first window image 41 may be input to the first deep neural network 43, respectively, and the second window image 42 ), a plurality of axial images, a plurality of coronal images, and a plurality of sagittal images may be input to the second deep neural network 44 , respectively.

In an embodiment, the first deep neural network 43 and the second deep neural network 44 calculate a probability value of a region likely to be a tumor in the first window image 41 and the second window image 42 . It may be a two-dimensional deep neural network that has been machine-learned in advance. For example, the first deep neural network 43 and the second deep neural network 44 may include a 2D U-net.

In one embodiment, the 2D U-net and 3D U-net shown in FIG. 4 may correspond to the convolutional network structure proposed in "3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation", MICCAI 2016. . For example, a 3D U-Net may include a contracting path and an expansive path. The contraction path follows the typical structure of a convolutional neural network, which involves the iterative application of two 3x3 unpadded convolutions, each of which has a rectified linear unit. ; ReLU) and a 2x2 max pooling operation of stride 2 for downsampling is followed. For each downsampling step, the number of feature channels is doubled. All steps in the dilation path are correspondingly cropped by upsampling of the feature map followed by a 2x2 convolution (“up-convolution”) that halves the number of feature channels. ) consists of concatenation with the feature map from the constriction path, and two 3x3 convolutions, each of which is followed by a ReLU. The aforementioned truncation is necessary because of the loss of border pixels in every convolution. In the final layer, 1x1 convolution is used to map each 64-component feature vector to a desired number of classes. A total of 22 convolutional layers are included in this exemplary neural network, which is arbitrary. Choose the input tile size so that all 2x2 max-pooling operations are applied to layers with even x and y sizes, so that the output segmentation map leads neatly It will be appreciated by those skilled in the art that it is important to

In an embodiment, in operation 24 , the two-dimensional deep learning unit 120 and the probabilistic model generating unit 130 perform a first dictionary shape through the first deep neural network 43 and the second deep neural network 44 , respectively. A model 47 and a second pre-shape model 48 may be created.

In an embodiment, the two-dimensional deep learning unit 120 inputs the first window image 41 and the second window image 42 to the first deep neural network 43 and the second deep neural network 44, respectively. Accordingly, a plurality of first prediction maps 45 and a plurality of second prediction maps 46 may be generated through the first deep neural network 43 and the second deep neural network 44 , respectively. For example, the first plurality of prediction maps 45 may include a plurality of prediction maps corresponding to each of a plurality of axial images, a plurality of coronal images, and a plurality of sagittal images included in the first window image 41 . In addition, the plurality of second prediction maps 46 may include a plurality of prediction maps corresponding to each of a plurality of axial images, a plurality of coronal images, and a plurality of sagittal images included in the second window image 42 .

In an embodiment, the first and second plurality of prediction maps 45 and 46 may have a plurality of channels although resolution is lower than that of the input first and second window images 41 and 42 . Through the generated first and second plurality of prediction maps 45 and 46, the regions may be divided by sharing them in the region classification step and the object detection step, and the region of interest may be detected. That is, by performing pixel labeling from the first and second plurality of prediction maps 45 and 46 , regions of an image may be divided. In this case, pixel labeling may be performed by attaching the same number to all adjacent pixels having similar features of the input image. Also, the pixel labeling result may include probability distribution information for each pixel with respect to the class to be classified of the image.

In an embodiment, the generation of the first and second plurality of prediction maps 45 and 46 may use a convolutional neural network (CNN), but is not limited thereto, and a deep neural network (DNN) , Recurrent Neural Network (RNN), etc. may vary depending on the purpose of use of the image processing device. Here, as an example, a plurality of first and second prediction maps 45 and 46 may be generated according to 2D U-net.

In an embodiment, the probabilistic model generator 130 combines the plurality of first prediction maps 45 and the plurality of second prediction maps 46 using max probability value voting, respectively, to obtain the first A pre-shape model 47 and a second pre-shape model 48 may be generated.

Here, the first pre-shape model 47 may be generated by combining a plurality of prediction maps having a maximum probability value among a plurality of first prediction maps 45 having a reference probability value or more. Also, the second dictionary shape model 48 may be generated by combining a plurality of prediction maps having a maximum probability value among a plurality of second prediction maps having a reference probability value or more. For example, the reference probability value may be 0.5, and the first pre-shape model 47 and the second pre-shape model 48 may be generated by connecting pixels that are 0.5 or more and are labeled with only maximum probability values. Since tumors vary in location and size, the margin of error can be reduced by using the maximum probability value rather than the average probability. Meanwhile, the probability value may mean a probability value of the presence of a tumor.

Meanwhile, the first pre-shape model 47 may include three-dimensional spatial shape information of the entire area of the organ including the tumor in the form of a probability map, and the second pre-shape model 48 is the location of the tumor. 3D spatial shape information of an important region may be included in the form of a probability map. For example, the first pre-shape model 47 and the second pre-shape model 48 may include information meaningful for classifying regions in a medical image and detecting an object, for example, illuminance, color, outline, etc. of the medical image. may include.

In an embodiment, in operation 25 , the probabilistic model generator 130 may generate the weighted shape-centered dictionary model 50 by combining the first pre-shape model 47 and the second pre-shape model 48 . . For example, the weighted shape-centered dictionary model 50 may include 3D spatial shape information of the tumor in the form of a probability map.

For example, the probabilistic model generating unit 130 combines the first pre-shape model 47 and the second pre-shape model 48 using weighted average voting to generate the weighted shape-centered dictionary model 50 ) can be created.

For example, the weight shape center dictionary model 50 may be calculated by Equation 1 below.

[Equation 1]

Here, Map _i is the probability value of the i-th pixel in the weighted shape center model 50 , P ⁱ _L is the probability value of the i-th pixel in the first pre-shape model 47 , and P ⁱ _M is the second dictionary shape It may be the probability value of the i-th pixel in the model 48 .

That is, since the probability information obtained through the second window image 42 in which the difference in brightness values between the tumor and the surrounding structures is clear can be said to be core information of the tumor, the second pre-shape model 48 generated through the second window 42 . ) can be given a high weight, and a low weight can be given to the probability value of the first pre-shape model 47 generated through the first window 41 including the entire and wide area of the tumor. have. Meanwhile, the range of the probability value is 0.5 or more because, as described above, the first pre-shape model 47 and the second pre-shape model 48 consist only of pixels having a probability value of 0.5 or more.

In summary, the weighted shape center model 50 may be generated by considering the main region where the tumor is located and the entire organ region to more accurately reflect the probability information related to the region of interest where the tumor will be located. Accordingly, as shown in FIG. 5 , the images included in the weighted shape center model 50 may accurately reflect the size and shape of the tumor. For example, the brightness value of the medical image (eg, white in FIG. 5 ) may be higher than that of the surrounding area so that the shape of the tumor is clearly distinguished from the surrounding area. It may have a lower brightness value (eg, gray in FIG. 5 ) than the surrounding area. 4 different tumors are shown in FIG. 5 , and it can be confirmed that all four tumors having different shapes and sizes are divided in the medical image to be clearly distinguished from the surrounding area by the present invention.

In an embodiment, in operation 26 , the 3D deep learning unit 140 inputs the 3D medical image 49 including the tumor and the weighted shape-centered dictionary model 50 to the third deep neural network 51 . can For example, the third deep neural network 51 uses the 3D medical image 49 and the weighted shape-centered dictionary model 50 to perform machine learning in advance to calculate the probability value of a region likely to be a tumor. It may be a three-dimensional deep neural network. For example, the third deep neural network 51 may include a 3D U-net.

Meanwhile, although not shown in the drawings, according to an embodiment of the present invention, at least a portion of a region other than a region of interest in the 3D medical image 49 may be cropped. For example, in order to reduce the operation process of the third deep neural network 51 , a portion of the boundary region in the 3D medical image 49 may be cropped. Specifically, for example, a boundary portion completely different from the brightness value information of the tumor and surrounding structures may be cropped using brightness value information.

In an embodiment, in operation 27 , the 3D deep learning unit 140 may obtain the tumor segmentation result 52 through the third deep neural network 51 . The tumor determination unit 150 may determine a cancer prognosis and treatment result by using the tumor division result 52 .

That is, the present invention uses the second window image 42 to which the second window setting is applied so that the region where the tumor is located can be seen clearly and the first window image 41 to which the second window setting is applied so that the entire organ can be seen well. By doing so, a more sophisticated pre-model can be generated and the accuracy of tumor segmentation can be increased.

Meanwhile, the data set used to confirm the effect of the present invention is contrast-enhanced chest CT data from 263 patients with non-small cell lung cancer, and 100-130 kVp; 73~575mAs; slice thickness 1.0 ~ 7.5mm, pixel space 0.54 ~ 0.83mm set as parameters, lung window setting is window width 1500HU (Hounsfield unit), window level 600HU, mediastinal window setting is window width 360HU, window level 60HU set

For accuracy evaluation in FIG. 6 , Dice similarity coefficient (DSC), sensitivity (Sensitivity, Sens), and specificity (Specificity, Spec) for each method were calculated and compared through Equation (2).

[Equation 2]

In this case, TP (True Positive) is the number of pixels in the automatically segmented region in the manually divided organ region, TN (True Negative) is the number of pixels in the non-automatically segmented region in the non-manually divided organ region, FP (False Positive) ) denotes the number of pixels in an auto-segmented region in a non-manually divided organ region, and FN (False Negative) denotes the number of pixels in a non-auto-segmented region in a manually-segmented organ region.

When compared with the 2D, 2.5D and 3D U-Net results, the segmentation performance of the tumor segmentation method (Coupling-Net) of the present invention was 75.06% in DSC, and the result was 0.14% higher than the 3D U-Net result. p It is only a low number, 3D U-Net recorded the highest sensitivity of 75.66% due to over-segmentation, and 2D U-Net had the highest specificity at 99.96% due to under-segmentation. Therefore, the tumor segmentation method of the present invention shows the most stable results.

7 , class 1 is a tumor isolated from the periphery, class 2 is a tumor attached to the chest wall, class 3 is a tumor attached to the mediastinum, and class 4 is a tumor attached to the chest wall or liver in the upper and lower parts of the lung. A tumor surrounded by peripheral structures. The number of each class is 8, 10, 13 and 4, respectively. The table of FIG. 7 discloses the DSC values of each class. The present invention (Coupling-Net) showed the best performance of 87.28%, 65.28%, and 86.03% of classes 1, 3, and 4 in which lung tumors were isolated and attached to the mediastinum or surrounded by the chest wall or surrounding organs. In the case of class 1, the accuracy of segmentation can be increased in the present invention (Coupling-Net) because small blood vessels around the lung tumor were removed in the mediastinal window setting. For class 3, unlike the lung window setting, the difference in intensity between the lung tumor and the mediastinum was large in the mediastinal window setting, making it possible to distinguish the lung tumor from the mediastinum. However, the relatively low performance of grade 3 compared to other grades is due to the variety of lung tumor shapes. In the case of class 4, it showed excellent performance because it was able to accurately target the tumor according to the large difference in strength between the tumor and the surrounding backbone in the mediastinal window setting. Since the present invention (Coupling-Net) was trained including the area invaded up to the chest wall through the weighted shape-centered prior model, only class 2 showed the highest result in 77.27% of 3D U-Net.

8 shows a segmentation result in a chest CT image. Figure 8 (a) is the original chest CT image, Figure 8 (b) shows the result using 2D U-Net, Figure 8 (c) shows the result using the 2.5D U-Net, Figure 8 ( d) shows the results using 3D U-Net, Figure 8 (e) shows the results using the present invention (Coupling-Net), red is overlapping region with expert annotation, green is under-segmentation region, Blue is the over-segmentation region.

Referring to FIG. 8 , the 2D U-Net results were under-segmented in all classes when considering only the axial plane, and it was confirmed that leakage occurred in the surrounding blood vessels and chest wall with intensity values similar to those of lung tumors when only the lung window setting was considered. In Fig. 8 (d), the overall segmentation performance of the 3D U-Net was improved, but the performance was lowered in Class 4 surrounded by the chest wall, and in Fig. , showed stable segmentation results in all classes regardless of location, shape, and type.

In the method for automatically segmenting a tumor using deep learning based on dual window setting in a medical image according to an embodiment of the present invention, the first window setting and the second window setting are respectively applied to the 2D medical image including the tumor, generating a first window image and a second window image based on the 3D medical image; The first window image and the second window image are input to a first deep neural network and a second deep neural network, respectively, and a first pre-shape model and generating a second pre-shape model; generating a weighted shape-focused prior by combining the first pre-shape model and the second pre-shape model; and inputting the 3D medical image including the tumor and the weighted shape-centered dictionary model into a third deep neural network, and obtaining a segmentation result of the tumor through the third deep neural network. can do

According to various embodiments, the 2D medical image includes a plurality of axial images, a plurality of coronal images, and a plurality of sagittal images, and the first window image is the first window setting. a plurality of axial images, a plurality of coronal images, and a plurality of sagittal images applied, and the second window image includes a plurality of axial images, a plurality of coronal images and a plurality of sagittal images to which the second window setting is applied. can

According to various embodiments, a region of interest of the first window image to which the first window setting is applied is the entire region of the organ including the tumor, and a region of interest of the second window image to which the second window setting is applied. The region of interest may be an important region in which the tumor is located.

According to various embodiments, the first deep neural network and the second deep neural network may include a 2D U-net, and the third deep neural network may include a 3D U-net.

According to various embodiments, by inputting the first window image and the second window image to a first deep neural network and a second deep neural network, respectively, through the first deep neural network and the second deep neural network, respectively generating a plurality of first prediction maps and a plurality of second prediction maps; and generating the first pre-shape model and the second pre-shape model by combining the plurality of first prediction maps and the plurality of second prediction maps using max probability value voting, respectively. may include.

According to various embodiments, the first prior shape model is generated by combining a plurality of prediction maps having a maximum probability value among a plurality of first prediction maps having a reference probability value or more, and the second prior shape model is It may be generated by combining a plurality of prediction maps having a maximum probability value among a plurality of second prediction maps having a reference probability value or more.

According to various embodiments, the method may include generating the weighted shape-centered dictionary model by combining the first dictionary shape model and the second dictionary shape model using weighted average voting.

According to various embodiments, the weighted shape center dictionary model is calculated by Equation 1 below, Map _i is the probability value of the i-th pixel in the weighted shape center model, and P ⁱ _L is the weighted shape center model in the first dictionary shape model. It is the probability value of the i-th pixel, and P ⁱ _M may be the probability value of the i-th pixel in the second pre-shape model.

[Equation 1]

According to various embodiments, the first pre-shape model includes three-dimensional spatial shape information of the entire region including the tumor in the form of a probability map, and the second pre-shape model includes the location of the tumor. The three-dimensional spatial shape information of the important region may be included in the form of a probability map, and the weighted shape-centered dictionary model may include the three-dimensional spatial shape information of the tumor in the form of a probability map.

According to various embodiments, an automatic tumor segmentation program using deep learning based on two window settings in a medical image is combined with a computer that is hardware, and stored in a medium to execute the method of any one of claims 1 to 9 can be

The steps of a method or algorithm described in relation to an embodiment of the present invention may be implemented directly in hardware, as a software module executed by hardware, or by a combination thereof. A software module may contain random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any type of computer-readable recording medium well known in the art to which the present invention pertains.

As mentioned above, although embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can realize that the present invention can be embodied in other specific forms without changing its technical spirit or essential features. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

100: tumor automatic segmentation device
110: video controller
120: 2D deep learning unit
130: probability model generation unit
140: 3D deep learning unit
150: tumor judgment unit
160: database

Claims (10)

In the automatic tumor segmentation method using deep learning based on the dual window setting in the medical image,
generating a first window image and a second window image based on the 2D medical image by applying a first window setting and a second window setting to a 2D medical image including a tumor, respectively;
By inputting the first window image and the second window image to a first deep neural network and a second deep neural network, respectively, a first pre-shape model and generating a second pre-shape model;
generating a weighted shape-focused prior by combining the first pre-shape model and the second pre-shape model; and
inputting the three-dimensional medical image including the tumor and the weighted shape-centered dictionary model into a third deep neural network, and obtaining a segmentation result of the tumor through the third deep neural network; An automatic tumor segmentation method using deep learning based on a dual window setting in a medical image, characterized in that.
According to claim 1, wherein the two-dimensional medical image comprises a plurality of axial (Axial) images, a plurality of coronal (Coronal) images, and a plurality of sagittal (Sagittal) images,
The first window image includes a plurality of axial images, a plurality of coronal images, and a plurality of sagittal images to which the first window setting is applied,
The second window image includes a plurality of axial images, a plurality of coronal images, and a plurality of sagittal images to which the second window setting is applied. A method for automatic tumor segmentation using deep learning based on a dual window setting in a medical image .
3. The method of claim 2,
A region of interest in the first window image to which the first window setting is applied is the entire region of the organ including the tumor;
An automatic tumor segmentation method using deep learning based on a dual window setting in a medical image, characterized in that the region of interest of the second window image to which the second window setting is applied is an important region in which the tumor is located.
According to claim 1,
The first deep neural network and the second deep neural network include a 2D U-net, and the third deep neural network includes a 3D U-net Deep learning based on a dual window setting in a medical image, characterized in that Automated tumor segmentation method using
According to claim 1,
By inputting the first window image and the second window image to a first deep neural network and a second deep neural network, respectively, a first plurality of prediction maps through the first deep neural network and the second deep neural network, respectively (Prediction map) and generating a plurality of second prediction maps; and
generating the first pre-shape model and the second pre-shape model by combining the plurality of first prediction maps and the plurality of second prediction maps using max probability value voting, respectively; An automatic tumor segmentation method using deep learning based on a dual window setting in a medical image, comprising:
The method of claim 5, wherein the first pre-shape model is generated by combining a plurality of prediction maps having a maximum probability value among a plurality of first prediction maps having a reference probability value or more,
The second pre-shape model is deep learning based on a dual window setting in a medical image, characterized in that it is generated by combining a plurality of prediction maps having a maximum probability value among a plurality of second prediction maps having a reference probability value or more. Automated tumor segmentation method using
The method of claim 1, further comprising generating the weighted shape-centered dictionary model by combining the first dictionary shape model and the second dictionary shape model using weighted average voting. Automated tumor segmentation method using deep learning based on dual window setup in medical images.
The method of claim 7, wherein the weighted shape center dictionary model is calculated by Equation 1 below,
[Equation 1]

Map _i is the probability value of the i-th pixel in the weighted shape center model, P ⁱ _L is the probability value of the i-th pixel in the first pre-shape model, and P ⁱ _M is the i-th pixel in the second pre-shape model An automatic tumor segmentation method using deep learning based on a dual window setting in a medical image, characterized in that it is a probability value of .
The method of claim 1, wherein the first pre-shape model includes three-dimensional spatial shape information of the entire region of the organ including the tumor in the form of a probability map,
The second pre-shape model includes three-dimensional spatial shape information of an important region in which the tumor is located in the form of a probability map,
The weighted shape-centered dictionary model includes the three-dimensional spatial shape information of the tumor in the form of a probability map. An automatic tumor segmentation method using deep learning based on a dual window setting in a medical image.
In combination with a computer that is hardware, stored in the medium to execute the method of any one of claims 1 to 9, an automatic tumor segmentation program using deep learning based on dual window settings in a medical image.

Images