JP7270626B2 - Medical image processing apparatus, medical image processing system, operating method of medical image processing apparatus, program, and storage medium - Google Patents

Description

The present invention relates to a medical image processing apparatus, a medical image processing system, a medical image processing method, and a program, and more particularly to automatic classification of lesions in medical images.

In the medical field, examinations using modalities such as endoscope systems are performed. In recent years, techniques have been known for analyzing medical images such as endoscopic images, which are time-series images captured using an endoscope, automatically classifying lesions included in the endoscopic images, and providing classification results. It is In this specification, image classification and image discrimination are treated as the same concept.

Patent Literature 1 describes an endoscope image processing system that supports detection of a characteristic lesion site in an image captured using an endoscope. When the endoscopic image processing system described in the document executes the lesion estimation function, the position estimated to be the lesion site is surrounded by a dotted line in the image displayed on the monitor.

Patent Document 2 discloses an image processing system that performs texture analysis on the overall gradation of a captured image in which blood vessels are captured, and uses the results of the texture analysis to perform classification corresponding to pathological diagnoses such as non-tumor and adenoma. Are listed. The document describes an example of displaying the correctness probability of classification.

Patent Literature 3 describes an endoscope apparatus that removes images inappropriate for diagnosis from images of body cavities captured using an endoscope apparatus.

Patent Literature 4 describes an image processing device that processes time-series images obtained by taking time-series images of a lumen in a living body using a medical observation device. The image processing apparatus described in the document identifies a region of normal mucous membranes in images constituting time-series images as a specific region. Also, the image processing apparatus described in the document calculates the reliability of the specific region.

Japanese Patent No. 5528255 JP 2017-70609 A Japanese Patent No. 4615963 Japanese Patent No. 5576711

When AI (Artificial Intelligence) is used to automatically classify the disease types of lesions discovered during examination of body cavities using an endoscope, the AI that performs automatic classification is based on the images acquired during the examination. is input and the classification result is output.

However, the images acquired during the examination may not show part of the lesion, such as when the lesion is hidden by some object, even though the lesion is present, or may be blurred even if the lesion is shown. It also includes images that are not suitable for classification using AI, such as those with images. In such cases, AI may output inappropriate classification results.

The inventions described in Patent Documents 1 to 4 may output inappropriate classification results when classifying images that are not suitable for automatic classification as described above. On the other hand, it is difficult for an observer to determine whether the classification result is appropriate.

SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances. , and to provide programs.

In order to achieve the above object, the following aspects of the invention are provided.

A medical image processing apparatus according to a first aspect includes an image acquisition unit that acquires a captured image generated by imaging a living body, a classification unit that classifies the captured image into two or more classes, and a classification target of the classification unit. An image generator that generates an area image that visualizes the position of the area that contributed to the classification using the classifier in the captured image, and a display signal that represents the captured image, the classification result derived using the classifier, and the area image. a display signal transmission unit for transmitting to the display device, the display signal transmission unit being a medical image processing apparatus for transmitting to the display device a display signal for displaying the region image separately from the captured image.

According to the first aspect, a captured image is classified into two or more classes, an area image is generated by visualizing an area that has contributed to the classification in the captured image, and the area image is displayed separately from the captured image using a display device. . This allows the observer to grasp the regions that contributed to the classification, and to grasp whether the classification is appropriate.

A captured image may be a medical image captured using a medical device. A medical image represents an image of a living body generated using a modality such as an endoscope device, a CT device, an MRI device, and an X-ray imaging device.

As a class, defined medical classifications may apply.

According to a second aspect, in the medical image processing apparatus of the first aspect, the image generator may change the visualization form of the region image according to the classification result of the captured image.

According to the 2nd aspect, it becomes easy to recognize the distinction of classification.

Examples of visualization features include visualization features using color and visualization features that change position. A visualization form that uses these together is also possible.

A third aspect is the medical image processing apparatus according to the first aspect or the second aspect, wherein the classification unit classifies the captured image based on the feature amount acquired from the captured image, and the image generation unit classifies the region based on the feature amount. An image may be generated.

According to the third aspect, it is possible to generate an area image based on the feature amount of the captured image.

In the third aspect, the captured image may be divided into a plurality of areas, the feature amount may be calculated for each area, and the area contributing to the classification in the captured image may be specified based on the feature of each area.

A fourth aspect is the medical image processing apparatus according to the first aspect or the second aspect, wherein the classifying unit is applied with a trained deep learning device, and the image generating unit generates a region based on intermediate layer information in the deep learning device. An image may be generated.

According to the fourth aspect, it is possible to generate the region image based on the information of the intermediate layer of the trained deep learning device.

A fifth aspect is the medical image processing apparatus according to any one of the first aspect to the fourth aspect, wherein the classification unit calculates a degree of belonging to a class for each of a plurality of regions set in the captured image, A configuration may be adopted in which the captured images are classified based on the degree.

According to the fifth aspect, classification based on the degree of belonging is possible.

Examples of the degree of membership include the probability of belonging to a class and the score of the class.

A sixth aspect may be configured such that, in the medical image processing apparatus of the fifth aspect, the image generation unit generates a region image based on the degree of belonging.

According to the sixth aspect, it is possible to generate an area image based on the degree of belonging.

A seventh aspect is the medical image processing apparatus according to any one of the first aspect to the sixth aspect, wherein the classifying unit performs classification exception determination based on the area image, and the display signal transmitting unit uses the classifying unit. A display signal indicating the result of the exception determination may be transmitted to the display device.

According to the seventh aspect, it is possible to suppress the output of inappropriate classification results for captured images that are difficult to classify.

The display signal transmission unit can transmit a display signal representing the exception determination result to the display device instead of the classification result. The display signal transmission unit can transmit a display signal representing the classification result and the exception determination result to the display device.

An eighth aspect is the medical image processing apparatus of the seventh aspect, wherein the classification unit calculates the reliability of the classification result derived using the classification unit based on the region image, and the display signal transmission unit calculates the reliability may be configured to transmit a display signal representing to the display device.

According to the eighth aspect, it is possible to grasp the reliability of the classification.

A ninth aspect is the medical image processing apparatus according to the eighth aspect, comprising a storage instruction acquisition unit that acquires an instruction to store the captured image, and a storage unit that stores the captured image, wherein the storage unit stores the captured image. A configuration in which at least one of the classification result, the exception determination result, and the reliability of the classification result is associated with the captured image when the captured image is stored in the storage unit based on an image storage instruction. good too.

According to the ninth aspect, the captured image and the information associated with the captured image can be used. Also, information associated with the captured image can be confirmed.

According to a tenth aspect, in the medical image processing apparatus according to any one of the first to ninth aspects, the display signal transmission unit may transmit a display signal representing character information of the classification result to the display device.

According to the tenth aspect, the classification result can be grasped based on the character information.

The character information can be of any language type. Textual information may apply abbreviations.

A medical image processing system according to an eleventh aspect includes an image acquisition unit that acquires a captured image generated by imaging a living body, a classification unit that classifies the captured image into two or more classes, and a classification target of the classification unit. An image generation unit that generates an area image that visualizes the position of the area that contributed to the classification using the classification unit in the captured image, a display device that displays the captured image and the area image, and the captured image and the area image. a display signal transmission unit that transmits a display signal to a display device, and the display signal transmission unit is a medical image processing system that transmits a display signal for displaying a region image separately from a captured image to the display device .

According to the eleventh aspect, the same effects as those of the first aspect can be obtained.

In the eleventh aspect, the same matters as those specified in the second to tenth aspects can be appropriately combined. In that case, the components that perform the specified processes and functions in the medical image processing apparatus can be understood as the components of the medical image processing system that perform corresponding processes and functions.

A medical image processing method according to a twelfth aspect includes an image acquisition step of acquiring a captured image generated by imaging a living body, a classification step of classifying the captured image into two or more classes, and a classification target of the classification step. An image generation step of generating an area image that visualizes the position of the area that contributed to the classification in the classification step in the captured image, and transmitting the captured image, the classification result derived in the classification step, and a display signal representing the area image to the display device. and a display signal transmission step for displaying the region image separately from the captured image.

According to the twelfth aspect, the same effects as those of the first aspect can be obtained.

In the twelfth aspect, the same items as those specified in the second to tenth aspects can be appropriately combined. In this case, the components that perform the specified processes and functions in the medical image processing apparatus can be grasped as the components of the medical image processing method that perform the corresponding processes and functions.

A program according to a thirteenth aspect provides a computer with an image acquisition function of acquiring a captured image generated by imaging a living body, a classification function of classifying the captured image into two or more classes, and a captured image to be classified by the classification function. 3, an image generation function that generates an area image that visualizes the position of the area that contributed to the classification using the classification function, and a display device that displays the captured image, the classification result derived using the classification function, and the display signal representing the area image It is a program for realizing the display signal transmission function for transmitting a display signal for displaying an area image separately from a captured image to a display device.

According to the thirteenth aspect, the same effects as those of the first aspect can be obtained.

In the thirteenth aspect, the same matters as those specified in the second to tenth aspects can be appropriately combined. In such a case, the components responsible for the specified processes and functions in the medical image processing apparatus can be grasped as components of a program responsible for the corresponding processes and functions.

According to the present invention, a captured image is classified into two or more classes, an area image is generated by visualizing areas that have contributed to the classification in the captured image, and the area image is displayed separately from the captured image using a display device. This allows the observer to grasp the regions that contributed to the classification, and to grasp whether the classification is appropriate.

FIG. 1 is an overall configuration diagram of an endoscope system including a medical image processing apparatus according to an embodiment. FIG. 2 is a block diagram showing the hardware configuration of the medical image processing apparatus. FIG. 3 is a functional block diagram of the medical image processing apparatus according to the first embodiment. FIG. 4 is an explanatory diagram of a display screen applied to the first embodiment. FIG. 5 is an explanatory diagram of a display screen according to a comparative example. FIG. 6 is an explanatory diagram of a display screen when the reliability of classification results is low. FIG. 7 is an explanatory diagram of a display screen showing another example when the reliability of classification results is low. FIG. 8 is an explanatory diagram of a display screen when there are multiple lesions. FIG. 9 is an explanatory diagram of a display example of classification results. FIG. 10 is an explanatory diagram of an example of displaying belonging probabilities as classification results. FIG. 11 is an explanatory diagram of an example of displaying scores as classification results. FIG. 12 is a flow chart showing the procedure of the medical image processing method. FIG. 13 is an explanatory diagram of a display screen according to a modification of the area image. FIG. 14 is an explanatory diagram of a display screen according to the first modified example of classification results. FIG. 15 is an explanatory diagram of a display screen according to the second modified example of classification results. FIG. 16 is an explanatory diagram of an area image to which shading according to the degree of contribution is applied. FIG. 17 is an explanatory diagram of an area image to which a heat map according to the degree of contribution is applied. FIG. 18 is a block diagram showing a configuration example of a classifier to which a convolutional neural network is applied. FIG. 19 is a conceptual diagram of shape transition of the feature quantity of the convolution neural network. FIG. 20 is a conceptual diagram of visualization based on the information of the intermediate layer of the convolutional neural network. FIG. 21 is an explanatory diagram of the segmentation technique. FIG. 22 is an explanatory diagram of a display screen indicating that detection is impossible. FIG. 23 is an explanatory diagram of a display screen showing determination impossibility. FIG. 24 is a functional block diagram of a medical image processing apparatus according to the third embodiment.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In this specification, the same components are denoted by the same reference numerals, and overlapping descriptions are omitted as appropriate.

[Overall Configuration of Endoscope System]
FIG. 1 is an overall configuration diagram of an endoscope system including a medical image processing apparatus according to an embodiment. An endoscope system 9 shown in FIG. 1 includes an endoscope 10 , a light source device 11 , a processor device 12 , a display device 13 , a medical image processing device 14 , an input device 15 and a monitor device 16 .

The endoscope 10 is an electronic endoscope. Also, the endoscope 10 is a flexible endoscope. The endoscope 10 has an insertion section 20 , an operation section 21 and a universal cord 22 . The insertion section 20 is inserted into the subject. The insertion portion 20 is formed in an elongated shape with a small diameter as a whole.

The insertion section 20 includes a flexible section 25 , a curved section 26 and a distal section 27 . The insertion portion 20 is configured by connecting a flexible portion 25, a bending portion 26, and a distal end portion 27. As shown in FIG. The flexible portion 25 has flexibility in order from the proximal end side of the insertion portion 20 toward the distal end side. The bending portion 26 has a structure capable of bending when the operation portion 21 is operated. The distal end portion 27 incorporates an imaging optical system, an imaging element 28, and the like (not shown).

The imaging device 28 is applied with a CMOS type imaging device or a CCD type imaging device. CMOS is an abbreviation for Complementary Metal Oxide Semiconductor. CCD is an abbreviation for Charge Coupled Device.

An observation window (not shown) is arranged on the distal end surface 27a of the distal end portion 27 . The observation window is an opening formed in the tip surface 27 a of the tip portion 27 . A cover (not shown) is attached to the observation window. An imaging optical system (not shown) is arranged behind the observation window. The imaging surface of the imaging element 28 receives the image light of the site to be observed via the observation window, the imaging optical system, and the like. The imaging device 28 captures image light of the site to be observed that has entered the imaging surface of the imaging device 28 and outputs an imaging signal. Imaging as used herein includes the meaning of converting reflected light from a site to be observed into an electrical signal.

The operation portion 21 is provided continuously on the proximal end side of the insertion portion 20 . The operation unit 21 includes various operation members operated by the operator. Specifically, the operation section 21 includes two types of bending operation knobs 29 . The bending operation knob 29 is used when bending the bending portion 26 . Note that an operator may be called a doctor, an operator, an observer, a user, or the like.

The operation unit 21 includes an air/water supply button 30 and a suction button 31 . The air/water supply button 30 is used when the operator performs an air/water supply operation. The suction button 31 is used when the operator performs a suction operation.

The operation unit 21 includes a still image capturing instruction unit 32 and a treatment instrument introduction port 33 . The still image capturing instruction unit 32 is operated by the operator when capturing a still image of the site to be observed. The treatment instrument introduction port 33 is an opening through which a treatment instrument is inserted into a treatment instrument insertion passage passing through the insertion portion 20 . Illustration of the treatment instrument insertion passage and the treatment instrument is omitted. A still image is shown in FIG.

The universal cord 22 is a connection cord that connects the endoscope 10 to the light source device 11 . The universal cord 22 includes a light guide 35 passing through the insertion portion 20, a signal cable 36, and a fluid tube (not shown).

The distal end of the universal cord 22 is provided with a connector 37 a connected to the light source device 11 and a connector 37 b branched from the connector 37 a and connected to the processor device 12 .

When the connector 37 a is connected to the light source device 11 , the light guide 35 and fluid tube (not shown) are inserted into the light source device 11 . Thereby, necessary illumination light, water, and gas are supplied from the light source device 11 to the endoscope 10 via the light guide 35 and the fluid tube (not shown).

As a result, illumination light is emitted from an illumination window (not shown) of the distal end surface 27a of the distal end portion 27 toward the site to be observed. Further, in response to the pressing operation of the air/water button 30, gas or water is jetted from an air/water nozzle (not shown) on the distal end surface 27a of the distal end portion 27 toward an observation window (not shown) on the distal end surface 27a.

When the connector 37b is connected to the processor device 12, the signal cable 36 and the processor device 12 are electrically connected. As a result, an imaging signal of the site to be observed is output from the imaging device 28 of the endoscope 10 to the processor device 12 via the signal cable 36 , and a control signal is output from the processor device 12 to the endoscope 10 .

In the present embodiment, a flexible endoscope has been described as an example of the endoscope 10. However, as the endoscope 10, various types of electronic endoscopes, such as rigid endoscopes, capable of capturing a moving image of a site to be observed can be used. A scope may be used.

The light source device 11 supplies illumination light to the light guide 35 of the endoscope 10 via the connector 37a. Illumination light can be white light or light in a specific wavelength band. The illumination light may be a combination of white light and light of a specific wavelength band. The light source device 11 is configured such that light in a wavelength band according to the purpose of observation can be appropriately selected as illumination light.

The white light may be either white wavelength band light or multiple wavelength band light. The specific wavelength band is a narrower band than the white wavelength band. As for the light in the specific wavelength band, light in one kind of wavelength band may be applied, or light in a plurality of wavelength bands may be applied. Light in a specific wavelength band is sometimes called special light.

The processor device 12 controls the operation of the endoscope 10 via the connector 37b and the signal cable 36. FIG. The processor device 12 also acquires an imaging signal from the imaging device 28 of the endoscope 10 via the connector 37b and the signal cable 36 . The processor device 12 acquires the imaging signal output from the endoscope 10 by applying a prescribed frame rate.

The processor device 12 generates an endoscopic image, which is an observed image of a site to be observed, based on the imaging signal acquired from the endoscope 10 . The endoscopic image 38 referred to here includes a moving image. Endoscopic image 38 may include still image 39 . The moving image is shown in FIG. 3 with reference numeral 38a. The endoscopic image 38 shown in the embodiment is an example of a captured image.

When the still image imaging instruction section 32 of the operation section 21 is operated, the processor device 12 generates a still image 39 of the site to be observed based on the imaging signal acquired from the imaging device 28 in parallel with the generation of the moving image. . The still image 39 may be generated with a resolution higher than that of the moving image.

When generating the endoscopic image 38, the processor unit 12 corrects the image quality by applying digital signal processing such as white balance adjustment and shading correction. The processor unit 12 may add additional information defined by the DICOM standard to the endoscopic image 38 . Note that DICOM is an abbreviation for Digital Imaging and Communications in Medicine.

The endoscopic image 38 is an in-vivo image obtained by imaging the inside of the subject, that is, the inside of the living body. If the endoscopic image 38 is an image obtained by imaging using light in a specific wavelength band, the endoscopic image 38 is a special light image. The processor device 12 then outputs the generated endoscopic image 38 to the display device 13 and the medical image processing device 14, respectively. The processor device 12 may output the endoscopic image 38 to a storage device (not shown) via a network (not shown) according to a communication protocol conforming to the DICOM standard. In addition, the network can apply the network 140 illustrated in FIG.

A display device 13 is connected to the processor device 12 . The display device 13 displays the endoscopic image 38 transmitted from the processor device 12 . The operator can advance and retreat the insertion section 20 while confirming the endoscope image 38 displayed on the display device 13 . When a lesion or the like is detected in the site to be observed, the operator can operate the still image imaging instruction section 32 to capture a still image of the site to be observed.

A computer is used for the medical image processing apparatus 14 . As the input device 15, a keyboard, a mouse, etc., which can be connected to a computer are used. The connection between the input device 15 and the computer may be wired connection or wireless connection. Various monitors connectable to a computer are used as the monitor device 16 .

As the medical image processing device 14, a diagnosis support device such as a workstation and a server device may be used. In this case, the input device 15 and the monitor device 16 are provided for each of a plurality of terminals connected to workstations or the like. Furthermore, as the medical image processing device 14, a medical service support device that supports preparation of medical reports and the like may be used.

The medical image processing device 14 acquires endoscopic images 38 and stores endoscopic images 38 . The medical image processing device 14 performs reproduction control of the monitor device 16 . Note that the term "image" in the specification includes the meaning of an electric signal representing an image and image data such as information representing an image. The term image as used herein means the image itself and/or the image data.

Also, the term "image storage" can be read as "image storage." The storage of images here means non-temporary storage of images. The medical image processing apparatus 14 may include a memory for temporary storage for temporarily storing images.

The input device 15 is used to input operation instructions to the medical image processing apparatus 14 . The monitor device 16 displays an endoscopic image 38 under the control of the medical image processing device 14 . The monitor device 16 may function as a display unit for various information in the medical image processing device 14 .

The medical image processing apparatus 14 can be connected to a storage device (not shown) via a network (not shown in FIG. 1). The DICOM standard, a protocol complying with the DICOM standard, and the like can be applied to the image storage format and communication between devices via the network.

The storage device (not shown) can be a storage that stores data non-temporarily. The storage device may be managed using a server device (not shown). A computer that stores and manages various data can be applied to the server device.

A configuration including the medical image processing device 14 and the monitor device 16 shown in the embodiment is an example of a medical image processing system.

[Description of Medical Image Processing Apparatus According to First Embodiment]
[Hardware Configuration of Medical Image Processing Apparatus]
FIG. 2 is a block diagram showing the hardware configuration of the medical image processing apparatus. The medical image processing apparatus 14 shown in FIG. 2 includes a control unit 120, a memory 122, a storage device 124, a network controller 126, a power supply device 128, a display controller 130, an input/output interface 132, and an input controller . Note that I/O shown in FIG. 2 represents the input/output interface 132 .

The control unit 120, memory 122, storage device 124, network controller 126, display controller 130, and input/output interface 132 are connected via a bus 136 so as to be capable of data communication.

<Control unit>
The control unit 120 functions as an overall control unit, various calculation units, and a storage control unit of the medical image processing apparatus 14 . The control unit 120 executes programs stored in ROM (read only memory) provided in the memory 122 .

The control unit 120 may download a program from an external storage device (not shown) via the network controller 126 and execute the downloaded program. The external storage device may be communicably connected to the medical image processing apparatus 14 via the network 140 .

The control unit 120 uses a RAM (random access memory) provided in the memory 122 as a calculation area, and cooperates with various programs to execute various processes. As a result, various functions of the medical image processing apparatus 14 are realized.

The control unit 120 controls reading data from the storage device 124 and writing data to the storage device 124 . The control unit 120 may acquire various data from an external storage device via the network controller 126 . The control unit 120 can execute various types of processing such as computation using the various types of acquired data.

The controller 120 may include one or two or more processors. Examples of processors include FPGAs (Field Programmable Gate Arrays) and PLDs (Programmable Logic Devices). FPGAs and PLDs are devices whose circuitry can be changed after manufacturing.

Another example of the processor is an ASIC (Application Specific Integrated Circuit). An ASIC comprises circuitry specifically designed to perform specific processing.

The controller 120 can apply two or more processors of the same type. For example, the controller 120 may use two or more FPGAs, or two PLDs. The controller 120 may employ two or more processors of different types. For example, the controller 120 may employ one or more FPGAs and one or more ASICs.

When a plurality of control units 120 are provided, the plurality of control units 120 may be configured using one processor. As an example of configuring a plurality of control units 120 with one processor, one processor is configured using a combination of one or more CPUs (Central Processing Units) and software, and this processor functions as a plurality of control units 120 There is a form to do. Note that software in this specification is synonymous with a program.

Another example of configuring the plurality of controllers 120 with a single processor is a form of using a processor that implements the functions of the entire system including the plurality of controllers 120 with a single IC chip. SoC (System On a Chip) is a representative example of a processor that implements the functions of the entire system including a plurality of control units 120 with a single IC chip. Note that IC is an abbreviation for Integrated Circuit.

In this way, the control unit 120 is configured using one or more of various processors as a hardware structure.

<memory>
The memory 122 includes ROM (not shown) and RAM (not shown). The ROM stores various programs executed in the medical image processing apparatus 14 . The ROM stores parameters, files, and the like used for executing various programs. The RAM functions as a temporary storage area for data, a work area for the control unit 120, and the like.

<Storage device>
The storage device 124 non-temporarily stores various data. The storage device 124 may be externally attached to the medical image processing apparatus 14 . A large-capacity semiconductor memory device may be applied instead of or in combination with the storage device 124 .

<Network controller>
Network controller 126 controls data communication with external devices. Controlling data communications may include managing data communications traffic. The network 140 connected via the network controller 126 can apply a known network such as LAN.

<Power supply>
A large-capacity power supply such as a UPS (Uninterruptible Power Supply) is applied to the power supply 128 . The power supply device 128 supplies power to each part of the medical image processing apparatus 14 when commercial power supply is interrupted due to a power failure or the like.

<Display controller>
The display controller 130 functions as a display driver that controls the monitor device 16 based on command signals sent from the controller 120 .

<Input/output interface>
The input/output interface 132 communicably connects the medical image processing apparatus 14 and an external device. The input/output interface 132 may apply a communication standard such as USB (Universal Serial Bus).

<Input controller>
The input controller 134 converts the format of the signal input using the input device 15 into a format suitable for processing by the medical image processing apparatus 14 . Information input from the input device 15 via the input controller 134 is transmitted to each section via the control section 120 .

Note that the hardware configuration of the medical image processing apparatus 14 shown in FIG. 2 is an example, and additions, deletions, and changes can be made as appropriate. Note that the hardware configuration of the medical image processing apparatus 14 shown in FIG. 2 can also be applied to other embodiments of the first embodiment.

[Description of Functional Blocks of Medical Image Processing Apparatus]
FIG. 3 is a functional block diagram of the medical image processing apparatus according to the first embodiment. The medical image processing apparatus 14 includes an image acquisition section 40 , an image processing section 42 , a display control section 44 and a storage section 46 . The image acquisition unit 40 acquires the endoscopic image 38 from the processor device 12 . The image acquisition unit 40 stores the endoscopic image 38 in the endoscopic image storage unit 46a.

The image acquisition unit 40 may acquire the endoscopic image 38 from the processor device 12 via an information storage medium such as a memory card. The image acquisition unit 40 may acquire the endoscopic image 38 via the network 140 shown in FIG.

That is, the image acquiring unit 40 can acquire the moving image 38a composed of the time-series frame images 38b. The image acquiring unit 40 can acquire the still image 39 when the still image is captured during the capturing of the moving image 38a.

The image processing unit 42 includes a classification unit 48 and an area image generation unit 50 . A classification unit 48 automatically classifies lesions from the endoscopic image 38 . The term classification in this specification can be read interchangeably with identification.

That is, the classification unit 48 can classify the endoscopic images 38 into prescribed classes and derive classification results. When automatically classifying the endoscopic images 38, the classifying unit 48 gives class information to the frame images 38b forming the moving image 38a. The classification unit 48 may assign class information to all frame images 38b, or may assign class information to frame images 38b every few frames. The classification unit 48 may assign class information to the still image 39 . Classes and class information can be read as classification results.

The classification unit 48 stores the classification result in the classification storage unit 46b in association with the frame image 38b. Table 1 below shows examples of classes applied to the classifier 48 .

NICE described in Table 1 above is an abbreviation for NBI International Colorectal Endoscopic Classification. NBI is an abbreviation for Narrow Band Imaging. JNET is an abbreviation for The Japan NBI Expert Team.

The area image generation unit 50 generates area images representing areas that have contributed to the classification in the endoscopic image 38 . The area image generation unit 50 stores the area image in the area image storage unit 46c. A regional image is illustrated in FIG. 4 using reference numeral 208 .

The display control unit 44 transmits to the monitor device 16 a display control signal that causes the monitor device 16 to display the endoscopic image 38 and the area image generated using the area image generation unit 50 . The display control unit 44 transmits to the monitor device 16 a display control signal that causes the monitor device 16 to display text information representing the classification result of the endoscopic image 38 derived using the classification unit 48 . The display control unit 44 shown in the embodiment is an example of a display signal transmission unit.

The monitor device 16 displays the endoscopic image 38, the area image, and character information representing the classification results on the same screen. The monitor device 16 displays the endoscopic image 38 and the area image in different areas on the screen. Textual information representing the classification result is illustrated in FIG. 4 using reference numeral 212 . The details of the screen displayed using the monitor device 16 will be described later.

The storage unit 46 includes an endoscope image storage unit 46a, a classification storage unit 46b, and a region image storage unit 46c. The endoscope image storage unit 46 a stores the endoscope image 38 acquired using the image acquisition unit 40 .

The image processing unit 42 reads the endoscopic image 38 stored in the endoscopic image storage unit 46 a and performs image processing on the endoscopic image 38 . The display control unit 44 reads the endoscopic image 38 stored in the endoscopic image storage unit 46 a and causes the monitor device 16 to display the endoscopic image 38 .

The classification storage unit 46 b stores the classes of the endoscopic images 38 classified by the classifying unit 48 in association with the endoscopic images 38 . Specifically, the classification storage unit 46b stores the classes of the frame images 38b forming the moving image 38a in association with the frame images 38b. The display control unit 44 reads out the classification result from the classification storage unit 46b and causes the monitor device 16 to display character information or the like representing the classification result.

The area image storage unit 46c stores the area image generated using the area image generation unit 50. FIG. The display control unit 44 reads the area image from the area image storage unit 46c and causes the monitor device 16 to display the area image.

The storage unit 46 shown in FIG. 3 can apply one or more storage elements. That is, the storage unit 46 can include three storage elements corresponding to the endoscopic image storage unit 46a, the classification storage unit 46b, and the region image storage unit 46c. Also, the endoscope image storage unit 46a, the classification storage unit 46b, and the area image storage unit 46c can each apply a plurality of storage elements. Furthermore, two or all of the endoscope image storage unit 46a, the classification storage unit 46b, and the region image storage unit 46c can be configured using one storage element.

[Description of the display screen displayed on the monitor device]
<Description of area image>
FIG. 4 is an explanatory diagram of a display screen applied to the first embodiment. The display screen 200 shown in FIG. 4 includes an endoscopic image display area 202 , an area image display area 204 , and a classification result display area 206 .

The endoscopic image display area 202 is an area in which the endoscopic image 38 is displayed. Endoscopic image display area 202 may display still image 39 . The endoscopic image display area 202 can switch between the endoscopic image 38 and the still image 39 for display. A reference numeral 209 represents a classification contribution region 209 which is a region that contributed to the classification of the endoscopic image 38 . FIG. 4 schematically shows the classification contribution area 209 .

An area image 208 is displayed in the area image display area 204 . A region image 208 has a classification contribution corresponding region 210 highlighted. Classification contribution corresponding region 210 is a region in region image 208 that corresponds to classification contribution region 209 .

The classification result display area 206 displays character information 212 representing the classification result. FIG. 4 shows an example in which neoplastic, which is an English notation for tumor, is displayed in the classification result display area 206 as the character information 212 . Note that the character information 212 can be written in any language. That is, the character information 212 can be written in Japanese or written in a foreign language other than English. Also, the textual information 212 may apply abbreviations.

FIG. 5 is an explanatory diagram of a display screen according to a comparative example. A comparison screen 220 displays the results of the automatic classification, and is an example in which a tumor 222 is found in the endoscopic image 38 and the endoscopic image 38 is classified as a tumor. The comparison screen 220 displays the endoscopic image 38 and character information 212 representing the classification result. Note that FIG. 5 schematically illustrates the tumor 222 .

The comparison screen 220 shown in FIG. 5 displays text information 212 representing the classification result, but if the endoscopic image 38 is difficult to classify, an incorrect classification result may be output. On the other hand, the display screen 200 shown in FIG. 4 displays an area image 208 corresponding to the endoscopic image 38 and displays a classification contribution corresponding area 210 in the area image 208 . Thereby, the observer can visually grasp based on which region of the endoscopic image 38 the classification unit 48 performs classification. Also, the area image 208 can be an indicator of the reliability of the classification result.

FIG. 6 is an explanatory diagram of a display screen when the reliability of classification results is low. A display screen 200a shown in FIG. 6 shows an example in which the classification result is erroneous, and the classification is performed based on a region other than the tumor 222 of the endoscopic image 38. FIG.

In the area image 208a shown in FIG. 6, an area 230 different from the area to be the classification contribution corresponding area 210a is displayed as the classification contribution corresponding area.

FIG. 7 is an explanatory diagram of a display screen showing another example when the reliability of classification results is low. A display screen 200b shown in FIG. 7 is an example when the tumor 222 in the endoscopic image 38 cannot be found.

In the area image 208b shown in FIG. 7, the area 210b that should be the classification contribution corresponding area is not displayed. When the display screen 200b shown in FIG. 7 is displayed, it is conceivable that the classification unit 48 cannot find a classification target because the tumor 222 is blurred or the size of the tumor 222 is small. .

In such a case, the operator operates the endoscope 10 shown in FIG. 1 to focus or enlarge and display the lesion and the surroundings of the lesion, so that the classification unit 48 can perform correct classification. becomes. That is, when the region image 208 is displayed together with the endoscopic image 38, it can suggest an operation to the operator for causing the classification unit 48 to derive a correct classification result.

In this embodiment, an example of displaying the endoscopic image display area 202 and the area image display area 204 on the display screen 200 displayed on one monitor device 16 has been described. The display screen 200 including the endoscopic image display area 202 may be displayed, and the display screen 200 including the area image display area 204 may be displayed on the other side.

Also, in one display screen 200 displayed on one monitor device 16, the endoscopic image display area 202 and the area image display area 204 may be switched in a time division manner and displayed. Furthermore, the regional image display area 204 may be superimposed on the endoscopic image display area 202 . For example, the area image display area 204 can be superimposed and displayed at a position such as the lower left corner of the endoscopic image display area 202 shown in FIG.

Observers tend to dislike movement of the viewpoint in order to observe endoscopic images while performing precise operations. When the regional image display area 204 is superimposed on the endoscopic image display area 202, the endoscopic image display area 202 and the regional image display area 204 are arranged closer to each other on the display screen 200b. This has the effect of reducing the movement of the observer's viewpoint.

FIG. 8 is an explanatory diagram of a display screen when there are multiple lesions. A first classification contribution region 209a and a second classification contribution region 209b are extracted from the display screen 200d shown in FIG.

A first classification contribution corresponding area 210g corresponding to the first classification contribution area 209a and a second classification contribution corresponding area 210h corresponding to the second classification contribution area 209b are displayed in the area image 208g. A distinguishable display mode is applied to the first classification contribution corresponding area 210g and the second classification contribution corresponding area 210h.

Further, the classification result display area 206a of the display screen 200d displays the first character information 212e representing the first classification result corresponding to the first classification contribution corresponding area 210g and the second classification result corresponding to the second classification contribution corresponding area 210h. The second character information 212f representing is displayed.

When there are different classifications of lesions such as non-neoplastic lesions and neoplastic lesions, it is difficult for a system that outputs one classification result from the endoscopic image 38 to return an appropriate classification result. On the other hand, in the medical image processing apparatus 14 according to the present embodiment, as shown in FIG. 8, the first classification contribution region 209a and the second classification contribution region 209b are visualized using the display mode for each classification in the region image 208g. . This makes it possible to obtain an appropriate classification result even when a plurality of lesions exist in the endoscopic image 38 and the plurality of lesions are classified differently.

<Explanation of classification result display>
FIG. 9 is an explanatory diagram of a display example of classification results. FIG. 9 shows an example of displaying a specific class as a classification result. Character information 212a shown in FIG. 9 indicates that the classification result is NICE1.

FIG. 10 is an explanatory diagram of an example of displaying belonging probabilities as classification results. FIG. 10 shows an example of displaying the belonging probability of each class as a result of classification. The character information 212b shown in FIG. 10 indicates that the belonging probability of NICE1 is 98%, the belonging probability of NICE2 is 2%, and the belonging probability of NICE3 is 0%.

The character information 212b shown in FIG. 10 may be only characters representing that the belonging probability of NICE1 is 98%. The character information 212b shown in FIG. 10 may be characters representing that the probability of belonging to NICE1 is 98% and the probability of belonging to NICE2 is 2%.

FIG. 11 is an explanatory diagram of an example of displaying scores as classification results. FIG. 11 shows an example in which scores of each class are displayed as classification results. The character information 212c shown in FIG. 11 indicates that the score of NICE1 is 1.23, the score of NICE2 is 0.002, and the score of NICE3 is 0.05. The belonging probabilities shown in FIG. 10 and the scores shown in FIG. 11 are examples of degrees of class belonging.

[Procedure of medical image processing method]
FIG. 12 is a flow chart showing the procedure of the medical image processing method. In the endoscope image acquisition step S10, the image acquisition unit 40 shown in FIG. 3 acquires the endoscope image 38. FIG. In the endoscopic image storage step S12, the image acquisition unit 40 stores the endoscopic image 38 acquired in the endoscopic image acquisition step S10 in the endoscopic image storage unit 46a.

In the classification step S14, the classification unit 48 classifies the endoscopic images 38 into prescribed classes. In the classification result storage step S16, the classification unit 48 stores the classification result derived in the classification step S14 in the classification storage unit 46b.

In the area image generation step S18, the area image generation unit 50 generates area images such as the area image 208 shown in FIG. 4 based on the classification result. In the area image storage step S20, the area image generation unit 50 stores the area image generated in the area image generation step S18 in the area image storage unit 46c.

In the display signal transmission step S<b>22 , the display control unit 44 transmits a display signal to the monitor device 16 . The display signals transmitted from the display control unit 44 to the monitor device 16 include display signals representing the endoscopic image 38 and the area image 208 . The display signal transmitted from the display control unit 44 to the monitor device 16 may include a display signal representing the classification result.

[Modified example of display screen]
<Modified example of area image>
FIG. 13 is an explanatory diagram of a display screen according to a modification of the area image. On the display screen 200c shown in FIG. 13, an image obtained by reducing the endoscopic image 38 is combined with the background of the area image 208c. That is, the area image 208 c is displayed by superimposing the classification contribution corresponding area 210 on the reduced endoscopic image 38 . An image obtained by reducing the endoscopic image 38 may have a lower resolution than the endoscopic image 38 .

<Modified example of classification results>
FIG. 14 is an explanatory diagram of a display screen according to the first modified example of classification results. A region image 208d shown in FIG. 14 is superimposed with character information 212d representing the classification result. In the area image 208d, enhancement of the text information 212d, such as changing the color of the text information 212d to the color of the classification contribution corresponding area 210, may be applied.

FIG. 15 is an explanatory diagram of a display screen according to the second modified example of classification results. In the display screen 200e shown in FIG. 15, the frame 201 of the endoscopic image 38 and the frame 207 of the area image 208h are given colors according to the classification results. Note that only the frame 201 may be colored, or only the frame 207 may be colored. That is, on the display screen 200e, at least one of the frame 201 of the endoscopic image 38 and the frame 207 of the region image 208h is given a color according to the classification result.

Since the observer observes the endoscopic image while performing precise operations, there is concern that the observer's viewpoint movement, the observer's recognition of character information, and the like may adversely affect the observer's operations. On the other hand, the operator viewing the display screen 200e shown in FIG. 15 can grasp the classification result without moving the viewpoint.

The medical image processing apparatus 14 not only gives colors to the frame 201 and the like according to the classification result, but also gives a color to the frame 201 and the like when the classification result is a specific classification such as a tumor, and colors other classifications. In this case, it is also possible to adopt a mode in which the frame 201 and the like are not colored. In addition, the medical image processing apparatus 14 may add a color to the frame 201 or the like in the case of exception determination, which will be described later, or change the color according to the reliability when changing the display according to the reliability. good too.

In addition, the mode of changing the color may include the mode of changing the shade of the same color. For example, it may be dark red if classified as tumor and light red if classified as non-tumor. Application of color is not limited to the frame 201 or the like. Areas other than the frame 201 and the frame 207 may be colored. Furthermore, a display mode other than colors such as symbols can be applied as long as it is a mode that suppresses the operator's viewpoint movement and makes it easier for the operator to grasp the classification result than the character information.

<Other Modifications>
The display form of the area image may be changed according to the classification result. For example, when classifying endoscopic images 38 into two classes, tumor or non-tumor, the colors for tumor and non-tumor may be different. In such an example, it may be red if classified as tumor and blue if classified as non-tumor.

Also, the display screen may be configured so that a plurality of area image display areas 204 can be displayed, and the position of the area image 208 may be changed according to the classification result. For example, in a display screen capable of displaying two upper and lower area image display areas 204, the upper area image display area 204 is used when classified as tumor, and the lower area image display area 204 is used when classified as non-tumor. obtain. According to the modification of the display screen described above, it is possible to assist the visibility of the operator. The display form according to the classification result shown in the embodiment is an example of the visualization form.

<Display example of area image according to the degree of contribution to classification>
FIG. 16 is an explanatory diagram of an area image to which shading according to the degree of contribution is applied. In the display screen 200e shown in FIG. 16, gradation corresponding to the degree of contribution is applied to the classification contribution corresponding area 210e of the area image 208e.

A classification-contribution corresponding region 210e shown in FIG. 16 is dark near the center 211a and thin around the periphery 211b. This indicates that the contribution of the central vicinity 211a is high and the contribution of the periphery 211b is low. It should be noted that the classification of the degree of contribution may be three or more stages.

FIG. 17 is an explanatory diagram of an area image to which a heat map according to the degree of contribution is applied. In the display screen 200f shown in FIG. 17, a heat map corresponding to the degree of contribution is applied to the classification contribution corresponding area 210f of the area image 208f.

In the classification contribution corresponding region 210f shown in FIG. 17, red is applied near the center 211a and blue is applied around the periphery 211b. An intermediate area 211c between the central neighborhood 211a and the periphery 211b is applied with a neutral color that changes from red to blue. Neutral colors include orange, yellow, green, and the like.

A classification contribution corresponding region 210f shown in FIG. 17 indicates that the contribution is high near the center 211a, the contribution decreases from the near center 211a toward the periphery 211b, and the contribution of the periphery 211b is the lowest. It should be noted that two or more of the modified examples described here may be appropriately combined.

[Detailed Description of Classification and Visualization of Regions Contributing to Classification]
Next, a specific example of the classification of the endoscopic image 38 and the visualization of the region contributing to the classification will be described.

<Pattern 1>
As a pattern 1, an example of calculating a feature amount from an endoscopic image 38, performing classification based on the feature amount, and visualizing a region that contributes to the classification is shown. A technique such as SVM (support vector machine) can be applied for the classification based on the feature amount. For example, a blood vessel region is extracted from the endoscopic image 38, and a feature amount of the extracted blood vessel region is calculated.

Other methods include a method of performing texture analysis on the endoscopic image 38 and calculating feature amounts using the analysis results, and a method of calculating local feature amounts such as SIFT (scale-invariant feature transform). .

The feature amount calculated using the above method is analyzed for each area when the target image is divided into a plurality of areas. This makes it possible to calculate the class membership probability for each area. It is possible to perform visualization processing for each region based on the class membership probability for each region. A plurality of regions shown in the embodiment is an example of a plurality of regions set in a captured image.

<Pattern 2>
As pattern 2, an example of analyzing the information of the intermediate layer of the convolution neural network to classify the endoscopic image 38 and visualizing the regions contributing to the classification will be shown. Such an approach can handle classification and visualization in parallel.

FIG. 18 is a block diagram showing a configuration example of a classifier to which a convolutional neural network is applied. Hereinafter, the convolutional neural network may be abbreviated as CNN (Convolutional Neural Network). A classifier to which the convolutional neural network shown in the embodiment is applied is an example of a deep learning device.

A classification unit 300 shown in FIG. 18 is an example of the classification unit 48 shown in FIG. The classifier 300 comprises a CNN 302 , an error calculator 304 and a parameter updater 306 .

The CNN 302 recognizes the type of lesion appearing in the endoscopic image 38 . The CNN 302 has a multiple layer structure and holds multiple weight parameters. By updating the weighting parameters of the CNN 302 from the initial values to the optimum values, the unlearned model can change to the trained model.

CNN 302 comprises an input layer 310 , an intermediate layer 312 and an output layer 314 . Intermediate layers 312 comprise sets of convolutional layers 320 and pooling layers 322 . Intermediate layer 312 also includes a fully bonded layer 324 . Each layer has a structure in which multiple nodes are connected by edges.

The input layer 310 receives an endoscopic image 38 as a learning target. The intermediate layer 312 extracts features from the endoscopic image 38 input from the input layer 310 . The convolutional layer 320 filters the nearby nodes in the previous layer to get the feature map. Filter processing is synonymous with convolution operation using a filter.

Pooling layer 322 reduces the feature map output from convolutional layer 320 to generate a new feature map. Convolutional layer 320 is responsible for feature extraction, such as edge extraction, from endoscopic image 38 .

The pooling layer 322 is responsible for providing robustness so that the extracted features are not affected by translations and the like. Note that the intermediate layer 312 is not limited to the case where the convolutional layer 320 and the pooling layer 322 are set as one set, but the case where the convolutional layers 320 are continuous, and the configuration including the normalization layer (not shown) is also possible.

The output layer 314 classifies the types of lesions in the endoscopic image 38 based on the features extracted by applying the intermediate layer 312 . The trained CNN 302 may, for example, classify medical images into two classes: tumor or non-tumor. Recognition results can be two types of scores corresponding to tumor or non-tumor.

Arbitrary initial values are set for the coefficients of the filter applied to the convolutional layer 320 in the CNN 302 before learning, the offset value, and the weight of the connection with the next layer in the fully connected layer 324 .

The error calculator 304 acquires the recognition result output from the output layer 314 of the CNN 302 and the correct data 370 for the endoscopic image 38, and calculates the error between them. Examples of error calculation methods include softmax cross entropy and sigmoid.

The parameter updating unit 306 adjusts the weight parameters of the CNN 302 by applying the error backpropagation method based on the error calculated by applying the error calculating unit 304 . The parameter updating unit 306 repeatedly performs parameter adjustment processing, and performs repeated learning until the difference between the output of the CNN 302 and the correct data 370 becomes small.

The classification unit 300 generates a trained model by performing learning to optimize each parameter of the CNN 302 using a data set of endoscopic images 38 stored in a database (not shown).

FIG. 19 is a conceptual diagram of shape transition of the feature quantity of the convolution neural network. Reference numeral 330 indicates schematic information. Input data represents the information 330 a of the input layer 310 . Conv. 1 to Conv. 5 represent the intermediate layer 312 . FC6 through FC8 represent all-connected layers 324. FIG. In FIG. 19, the information amount of the information 330a of the input layer 310 and the information amount of the information 330b of the intermediate layer 312 are expressed using width×height×number of channels.

A convolutional neural network produces a final output while aggregating the spatial information of the input image. As shown in FIG. 19, the convolutional neural network aggregates spatial information with width and height as it progresses from the input layer 310 to the hidden layer 312, and after the information 330c of the fully connected layer 324, spatially lose information.

In other words, by analyzing the information 330b of the intermediate layer 312, it is possible to extract features that hold spatial information. However, the information 330b of the intermediate layer 312 generally has more channels than the information 330a of the input layer 310. FIG.

In the example shown in FIG. 19, the number of channels of the information 330a in the input layer 310 is 3, whereas Conv . 4, the number of channels of the information 330b is increased to 384. Therefore, the problem is how to aggregate the information in the channel direction to visualize the classification contribution region. An example of visualizing a classification contribution region by aggregating information in the channel direction is shown below.

《First example》
In general, channels that contribute greatly to the final output tend to have large absolute values. Therefore, the absolute value of the information in each channel of the hidden layer 312 is compared, and the channel with the larger absolute value is extracted. Visualize the extracted channels. This allows visualization of areas that contribute more to the final output.

When extracting channels, multiple channels may be extracted. As an example of extracting a plurality of channels, there is an example of extracting a specified number of channels in descending order of absolute value. If multiple channels are extracted, the multiple channels may be averaged.

《Second example》
Principal component analysis may be performed in the channel direction of the intermediate layer 312 to extract principal components and visualize the extracted principal components. For example, by reducing the dimension of the channel to one dimension, it is possible to visualize the extracted principal components. In the second example, information on all channels is visualized, so more accurate visualization is possible than in the first example.

《Third example》
The final output result will be the score for each class label in the classification. The contribution of each class label of the classification to the score can be derived using differentiation. For example, Gradient-weighted Class Activation Mapping (Grad-CAM) may be applied to derive the contribution of each class label of the classification to the score.

Let ^yc be the score of any class c. Let A ^k be the k-th channel feature map of an arbitrary hidden layer. Let A ^k _ij be the value of the coordinates (i, j) of the feature map A ^k . A map L ^C _Grad-CAM that visualizes the contribution of class c is obtained by the following equation.

The map L ^C _Grad-CAM represents an area image visualizing the classification contribution area that aggregates information in the channel direction.

FIG. 20 is a conceptual diagram of visualization based on the information of the intermediate layer of the convolutional neural network. The example shown in FIG. 20 shows an example in which a processing target image 380 including two types of animals is classified into one animal 381a or the other animal 381b, and the regions serving as the grounds for classification are visualized.

A first classified image 382 is classified into one animal 381a, and shows an example in which a region 384 serving as a basis for classification is visualized. A second classified image 386 shows an example in which the animal is classified into the other animal 381b and a region 388 that serves as the basis for the classification is visualized.

When pattern 2 is applied to the classification of the endoscopic image 38, the endoscopic image 38 is used as the processing target image 380, and two kinds of animals are classified into two kinds of classes. Two types of animals may be used as two types of feature regions.

<Pattern 3>
As pattern 3, an example of visualizing a region by applying a segmentation technique is shown. FIG. 21 is an explanatory diagram of the segmentation technique. FIG. 21 shows an example of classifying a classification target image 390 including a person 391a and an animal 391b into a person area 394, an animal area 396, and a background area 398. FIG. Reference numeral 392 indicates the classified image.

Segmentation techniques yield class membership probabilities for each region. Each region can be visualized based on the membership probability. It is also possible to classify the classification target image 390 based on the regions obtained by applying the segmentation technique. For example, among the classes applied for classification, the class occupying the largest area in the classification target image 390 may be the classification result.

[Action and effect of the first embodiment]
According to the medical image processing apparatus according to the first embodiment, it is possible to obtain the following effects.

[1]
The endoscopic image 38 is displayed in the endoscopic image display area 202 on the display screen 200 displayed on the monitor device 16 . An area image 208 is displayed in an area image display area 204 different from the endoscopic image display area 202 of the display screen 200 . Area image 208 includes classification-contributing corresponding areas 210 corresponding to classification-contributing areas 209 that contributed to the classification of endoscopic image 38 . Thereby, the observer can visually grasp based on which region of the endoscopic image 38 the classification unit 48 performs classification. Also, the area image 208 can be an indicator of the reliability of the classification result.

[2]
The classification result is displayed on the monitor device 16 . Thereby, the observer can visually recognize the classification result of the endoscopic image 38 .

[3]
The degree of membership of each class is derived as a classification result. The degree of belonging to each class is displayed on the monitor device 16 . The degree of membership can apply the membership probability of each class and the score of each class. This allows the observer to visually recognize the degree of belonging to each class.

[4]
The display form of the area image 208 can be changed according to the classification result. This can improve the visibility of the classification results.

[Description of Medical Image Processing Apparatus According to Second Embodiment]
Next, a medical image processing apparatus according to a second embodiment will be described. In the medical image processing apparatus according to the second embodiment, exception determination is added to the classification of the endoscopic image 38 . That is, after visualizing the lesion area of the endoscopic image 38 using pattern 1, pattern 2, and pattern 3 shown in the first embodiment, the area image is analyzed to perform exception determination.

For example, if the area of the lesion area detected from the endoscopic image 38 is less than or equal to a specified threshold, it may be undetectable. If there are a plurality of lesion areas detected from the endoscopic image 38, and the area of each lesion area is equal to or greater than a specified threshold value, the judgment may be impossible.

The same hardware and functional blocks as those of the medical image processing apparatus 14 according to the first embodiment are applied to the medical image processing apparatus according to the second embodiment. A description of the hardware and functional blocks of the medical image processing apparatus according to the second embodiment is omitted.

[Display screen when detection is not possible]
FIG. 22 is an explanatory diagram of a display screen indicating that detection is impossible. A display screen 400 shown in FIG. 22 displays undetectable as character information 412 indicating undetectable. When the character information 412 representing undetectable is displayed, the classification contribution corresponding area 410 is not displayed in the area image 408 . In FIG. 22, a lesion area 407 that cannot be detected is schematically illustrated using a solid line. A lesion area 407 when detectable is schematically illustrated using a two-dot chain line.

The character information 412 can apply a form in which the character information representing the classification result is overwritten with the character information representing undetectable. The display screen 400 can apply a form representing undetectable, apart from the character information representing the classification result.

The classification unit 48 shown in FIG. 3 may analyze the region image 408 to quantify the reliability of the classification, and the display control unit 44 may cause the display screen 400 to display a numerical value such as a score representing the reliability of the classification. . Reference numeral 402 shown in FIG. 22 represents an endoscope image display area. Reference numeral 404 represents an area image display area. Reference numeral 406 represents a classification result display area.

[Display screen when judgment is impossible]
FIG. 23 is an explanatory diagram of a display screen showing determination impossibility. A display screen 400a shown in FIG. 23 displays undeterminable as character information 412a, which indicates that determination is impossible. When the character information 412a indicating that determination is impossible is displayed, the area image 408a displays a classification contribution corresponding area 410a indicating that determination is impossible.

As an example of the classification contribution corresponding area 410a representing the inability to determine, there is a mixture of two types of display forms corresponding to different classifications. FIG. 23 illustrates a classification contribution corresponding area 410a in which two colors are mixed when the color is changed according to the classification. Note that reference numeral 409 in FIG. 23 represents a classification contribution area.

[Action and effect of the second embodiment]
According to the medical image processing apparatus according to the second embodiment, it is possible to obtain the following effects.

[1]
If the lesion in the endoscopic image 38 cannot be detected or determined, character information 412 indicating that detection or determination is impossible is displayed. This allows the operator to grasp that the classification is inappropriate.

[2]
A classification reliability is calculated based on the area image. The confidence level is displayed on the display screen. This allows recognition of the reliability of the classification.

[Description of Medical Image Processing Apparatus According to Third Embodiment]
FIG. 24 is a functional block diagram of a medical image processing apparatus according to the third embodiment. The medical image processing apparatus 14a shown in FIG. 24 has a storage instruction acquisition unit 41 in contrast to the medical image processing apparatus 14 shown in FIG. A storage instruction obtaining unit 41 obtains an instruction to store the endoscopic image 38 transmitted from the processor device 12 shown in FIG. The storage instruction acquisition unit 41 may acquire an instruction to store the endoscopic image 38 from the endoscope 10 .

When the storage instruction acquisition unit 41 acquires the storage instruction for the endoscopic image 38, the endoscopic image 38 acquired using the image acquisition unit 40 is obtained by at least the classification result, the area image, the exception determination result, and the reliability. It is stored in the endoscopic image storage unit 46a in association with any of them.

The medical image processing apparatus 14a may combine the classification result and the like with the endoscopic image 38 and store them, or may store the classification result and the endoscopic image 38 separately. An instruction to store the endoscopic image 38 may be transmitted from the processor device 12 when the operator operates an operation button or the like (not shown), or the instruction to store the endoscopic image 38 may be sent to the processor based on the classification result or the like. It may be automatically sent from the device 12 . If the reliability is equal to or higher than the prescribed threshold, the processor device 12 determines that the medical image processing device 14a has obtained an appropriate classification result, and automatically transmits an instruction to store the endoscopic image 38. good.

[Action and effect of the third embodiment]
According to the medical image processing apparatus according to the third embodiment, it is possible to obtain the following effects.

[1]
After the endoscopy, when the user prepares the report, he/she can check whether the classification results, etc. are appropriate.

[2]
At the time of re-examination, the previous inspection results can be referred to.

[Modification of Endoscope System]
[Modification of Processor Device]
The processor device 12 may have the functionality of the medical image processing device 14 . That is, the processor device 12 may be configured integrally with the medical image processing device 14 . In such an embodiment, the display device 13 can also serve as the monitor device 16 . Processor unit 12 may include connection terminals for connecting input device 15 .

[Modified example of illumination light]
As an example of a medical image that can be acquired using the endoscope system 9 shown in the present embodiment, a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as the light in the white band can be mentioned. be done.

Another example of a medical image that can be acquired using the endoscope system 9 shown in this embodiment is an image obtained by irradiating light in a specific wavelength region. A narrower band than the white band can be applied to the specific wavelength band. The following modifications are applicable.

<First example>
A first example of a specific wavelength band is the visible blue or green band. The first example wavelength band includes a wavelength band of 390 nm or more and 450 nm or less, or 530 nm or more and 550 nm or less, and the first example light is 390 nm or more and 450 nm or less, or It has a peak wavelength within a wavelength band of 530 nm or more and 550 nm or less.

<Second example>
A second example of a specific wavelength band is the visible red band. A second example wavelength band includes a wavelength band of 585 nm or more and 615 nm or less, or a wavelength band of 610 nm or more and 730 nm or less, and the second example light is 585 nm or more and 615 nm or less, or It has a peak wavelength within a wavelength band of 610 nm or more and 730 nm or less.

<Third example>
A third example of the specific wavelength band includes a wavelength band in which oxyhemoglobin and reduced hemoglobin have different absorption coefficients, and the light in the third example has a peak wavelength in the wavelength band in which oxidized hemoglobin and reduced hemoglobin have different absorption coefficients. have The wavelength band of this third example includes a wavelength band of 400 ± 10 nanometers, 440 ± 10 nanometers, 470 ± 10 nanometers, or a wavelength band of 600 to 750 nanometers, and the light of the third example is It has a peak wavelength in the wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or 600 nm or more and 750 nm or less.

<Fourth example>
A fourth example of the specific wavelength band is a wavelength band of excitation light that is used for observing fluorescence emitted by a fluorescent substance in vivo and that excites this fluorescent substance. For example, it is a wavelength band of 390 nm or more and 470 nm or less. Observation of fluorescence is sometimes referred to as fluorescence observation.

<Fifth example>
A fifth example of the specific wavelength band is the wavelength band of infrared light. The wavelength band of this fifth example includes a wavelength band of 790 nm or more and 820 nm or less, or a wavelength band of 905 nm or more and 970 nm or less, and the light of the fifth example is 790 nm or more and 820 nm or less, Alternatively, it has a peak wavelength in a wavelength band of 905 nm or more and 970 nm or less.

[Generation example of special light image]
The processor device 12 may generate a special light image having information of a specific wavelength band based on a normal light image obtained by imaging using white light. Note that the generation here includes acquisition. In this case, the processor device 12 functions as a special light image acquisition unit. Then, the processor device 12 obtains a signal in a specific wavelength band by performing an operation based on the color information of red, green, blue or cyan, magenta, and yellow normally included in the optical image.

Note that red, green, and blue are sometimes expressed as RGB (Red, Green, Blue). Also, cyan, magenta, and yellow are sometimes expressed as CMY (Cyan, Magenta, Yellow).

[Generation example of feature quantity image]
As a medical image, at least one of a normal light image obtained by irradiating light in a white band, or light in a plurality of wavelength bands as white band light, and a special light image obtained by irradiating light in a specific wavelength band. A feature image may be generated using an operation based on.

[Example of application to a program that causes a computer to function as an image processing device]
The image processing method described above can be configured as a program that implements functions corresponding to each step in the image processing method using a computer. For example, it is possible to configure a program that causes a computer to realize an endoscopic image acquisition function, an image processing function, a display signal transmission function, and a storage function. Image processing functions may include classification functions and area image generation functions.

It is possible to store a program that causes a computer to implement the image processing function described above in a computer-readable information storage medium that is a tangible, non-temporary information storage medium, and to provide the program through the information storage medium. be.

Further, instead of providing the program by storing it in a non-temporary information storage medium, it is also possible to provide a program signal via a network.

[Regarding combinations of embodiments and modifications]
The components described in the above embodiments and the components described in the modified examples can be used in combination as appropriate, and some of the components can be replaced.

[Example of application to other devices]
In the above embodiment, an endoscopic image was exemplified as a medical image, but the automatic classification shown in this embodiment can also be applied to medical images acquired using a CT device, an MRI device, an X-ray imaging device, or the like. is. The medical image shown in the embodiment is an example of a captured image.

In the embodiments of the present invention described above, it is possible to appropriately change, add, or delete constituent elements without departing from the gist of the present invention. The present invention is not limited to the embodiments described above, and many modifications are possible within the technical concept of the present invention by those skilled in the art.

9 endoscope system 10 endoscope 11 light source device 12 processor device 13 display device 14 medical image processing device 14a medical image processing device 15 input device 16 monitor device 20 insertion section 21 operation section 22 universal cord 25 flexible section 26 bending section 27 Distal portion 27a Distal surface 28 Imaging device 29 Bending operation knob 30 Air/water supply button 31 Suction button 32 Still image imaging instruction unit 33 Treatment instrument introduction port 35 Light guide 36 Signal cable 37a Connector 37b Connector 38 Endoscope image 38a Moving image 38b Frame image 39 Still image 40 Image acquisition unit 41 Storage instruction acquisition unit 42 Image processing unit 44 Display control unit 46 Storage unit 46a Endoscope image storage unit 46b Classification storage unit 46c Area image storage unit 48 Classification unit 50 Area image generation unit 120 Control unit 122 Memory 124 Storage device 126 Network controller 128 Power supply device 130 Display controller 132 Input/output interface 134 Input controller 136 Bus 140 Network 200 Display screen 200a Display screen 200b Display screen 200c Display screen 200d Display screen 200e Display screen 201 Frame 202 Inside view Mirror image display area 204 Area image display area 206 Classification result display area 206a Classification result display area 207 Frame 208 Area image 208a Area image 208b Area image 208c Area image 208d Area image 208e Area image 208f Area image 208g Area image 208h Area image 209 Classification Contribution region 209a First classification contribution region 209b Second classification contribution region 210 Classification contribution corresponding region 210a Region 210b to be a classification contribution corresponding region Region 210e to be a classification contribution corresponding region Classification contribution corresponding region 210f Classification contribution corresponding region 210g First classification contribution corresponding region 210h Second classification contribution corresponding region 211a Near center 211b Surrounding region 211c Intermediate region 212 Character information 212a Character information 212b Character information 212c Character information 212d Character information 212e First character information 212f Second character information 213a First lesion 213b Second lesion 220 Comparison screen 222 Tumor 230 Region 300 Classification unit 302 CNN
304 error calculator 306 parameter updater 310 input layer 312 intermediate layer 314 output layer 320 convolution layer 322 pooling layer 324 fully connected layer 330 information 330a input layer information 330b hidden layer information 330c fully connected layer information 370 correct data 380 processing Target image 381a One animal 381b The other animal 382 First classified image 384 Classification basis area 386 Second classified image 388 Classification basis area 390 Classification target image 391a Person 391b Animal 392 Classified image 394 Person Area 396 Animal area 400 Display screen 400a Display screen 402 Endoscope image display area 404 Area image display area 406 Classification result display area 408 Area image 410 Classification contribution corresponding area 412 Character information 412a Character information S10 to S22 Medical image processing methods process

Claims (14)

an image acquisition unit that acquires a captured image generated by imaging a living body;
a classification unit that classifies lesion areas included in the captured image into classes representing two or more types of lesions ;
an image generation unit that generates an area image that visually represents an area that has contributed to the classification of the lesion area by the classification unit on a background image that indicates the area of the captured image, wherein the classification is performed using the classification unit. , an image generation unit that generates the region images in which the regions that contributed to each classification are visualized in different forms according to the classification results ;
a display signal transmission unit that transmits a display signal representing the captured image and the area image to a display device;
with
The display signal transmission unit transmits the display signal for displaying the area image separately from the captured image to the display device. 2. The medical image processing apparatus according to claim 1, wherein the classification unit classifies a lesion area included in the captured image into a neoplastic lesion and a non- neoplastic lesion . The classification unit classifies the captured image based on the feature amount acquired from the captured image,
3. The medical image processing apparatus according to claim 1, wherein the image generator generates the area image based on the feature amount. The classifying unit is applied with a trained deep learner,
3. The medical image processing apparatus according to claim 1, wherein the image generator generates the area image based on intermediate layer information in the deep learning device. 5. The classifying unit according to any one of claims 1 to 4, wherein the classification unit calculates a degree of belonging to the class for each of a plurality of areas set in the captured image, and classifies the captured image based on the degree of belonging. 10. The medical image processing apparatus according to claim 1. 6. The medical image processing apparatus according to claim 5, wherein the image generator generates the area image based on the degree of belonging. The classification unit performs classification exception determination based on the area image,
The medical image processing apparatus according to any one of claims 1 to 6, wherein the display signal transmission section transmits a display signal representing a result of exception determination using the classification section to the display device. The classification unit calculates the reliability of the classification result derived using the classification unit based on the area image,
8. The medical image processing apparatus according to claim 7, wherein the display signal transmission section transmits a display signal representing the reliability to the display device. a storage instruction acquisition unit that acquires a storage instruction for the captured image;
a storage unit that stores the captured image;
with
The storage unit stores at least the classification result, the exception determination result, and the reliability of the classification result in the captured image when storing the captured image in the storage unit based on the instruction to store the captured image. 9. The medical image processing apparatus according to claim 8, wherein any one is associated. The medical image processing apparatus according to any one of claims 1 to 9, wherein the display signal transmission section transmits a display signal representing character information of the classification result derived using the classification section to the display device. an image acquisition unit that acquires a captured image generated by imaging a living body;
a classification unit that classifies lesion areas included in the captured image into classes representing two or more types of lesions ;
an image generation unit that generates an area image that visually represents an area that has contributed to the classification of the lesion area by the classification unit on a background image that indicates the area of the captured image, wherein the classification is performed using the classification unit. , an image generation unit that generates the region images in which the regions that contributed to each classification are visualized in different forms according to the classification results ;
a display device that displays the captured image and the area image;
a display signal transmission unit that transmits a display signal representing the captured image and the area image to a display device;
with
In the medical image processing system, the display signal transmission unit transmits the display signal for displaying the area image separately from the captured image to the display device. A method of operating a medical image processing apparatus, the medical image processing apparatus comprising an image acquisition unit, a classification unit, an image generation unit, and a display signal transmission unit,
an image acquisition step in which the image acquisition unit acquires a captured image generated by imaging a living body;
a classification step in which the classification unit classifies the lesion area included in the captured image into classes representing two or more lesion types ;
An image generating step in which the image generating unit generates an area image visibly representing the area that contributed to the classification of the lesion area by the classifying unit on a background image indicating the area of the captured image, In the classification using the classification step, an image generation step of generating the region images in which the regions that contributed to each classification are visualized in different forms according to the classification results ;
a display signal transmission step in which the display signal transmission unit transmits a display signal representing the captured image and the area image to a display device;
including
The display signal transmitting step transmits the display signal for displaying the area image separately from the captured image to the display device. the computer,
an image acquisition unit that acquires a captured image generated by imaging a living body;
a classification unit that classifies lesion areas included in the captured image into classes representing two or more types of lesions;
an image generation unit that generates an area image that visually represents an area that has contributed to the classification of the lesion area by the classification unit on a background image that indicates the area of the captured image, wherein the classification is performed using the classification unit . , an image generation unit that generates the region images in which the regions that contributed to each classification are visualized in different forms according to the classification results;
A display signal transmission unit that transmits a display signal representing the captured image and the area image to a display device, and a display signal transmission unit that transmits the display signal for displaying the area image separately from the captured image to the display device . A program that acts as a A non-transitory computer-readable storage medium, wherein when instructions stored on the storage medium are read by a computer,
an image acquisition function for acquiring a captured image generated by imaging a living body;
A classification function that classifies lesion areas included in the captured image into classes representing two or more types of lesions;
An image generating function that generates an area image that visually represents an area that has contributed to the classification of the lesion area by the classification function on a background image that indicates the area of the captured image, the classification using the classification function. 3, an image generation function for generating the area images in which the areas that contributed to each classification are visualized in different forms according to the classification results; and
A display signal transmission function for transmitting a display signal representing the captured image and the area image to a display device, and a display signal transmission function for transmitting to the display device the display signal for displaying the area image separately from the captured image. A storage medium that allows a computer to execute

Images