JPWO2021033303A1 - Training data generation method, trained model and information processing device - Google Patents

Abstract

The training data generation method acquires an image taken by the endoscope, acquires voice data by the operator of the endoscope recorded in association with the acquired image, and recognizes the acquired voice data. A computer is made to execute a process of generating training data for a deep neural network based on the label corresponding to the recognition result and the image.

Description

This technique relates to a training data generation method, a trained model, and an information processing device.

Computer-aided diagnostic technology has been developed that automatically detects lesion sites using a learning model from medical images such as endoscopic images. A method of generating a learning model by supervised machine learning using training data with a correct answer label is known.

A learning method that combines the first learning using images taken with a normal endoscope for training data and the second learning using images taken with a capsule endoscope for training data. A learning model to be learned and a learning system including the learning model are disclosed (for example, Patent Document 1). The learning system of Patent Document 1 selects in advance an image showing a landmark and an image not showing a landmark from a group of images taken by a gastrointestinal endoscope and a capsule-type endoscope, and records each selected image and a correct answer label. do.

International Publication No. 2017/1752282

However, in order to generate training data for learning the learning model described in Patent Document 1, it is necessary to select each image and give a correct answer label to each of these images based on the selection result. In the learning system of No. 1, the point of efficiently generating the training data is not considered.

One aspect is to provide a training data generation method or the like that efficiently generates training data for a trained model using an image taken by an endoscope.

In the training data generation method according to one aspect of the present disclosure, an image taken by an endoscope is acquired, and voice data by an operator of the endoscope recorded in association with the acquired image is acquired and acquired. A computer is made to execute a process of generating training data for a deep neural network based on the label corresponding to the recognition result and the image according to the recognition result of the voice data.

The trained model in one aspect of the present disclosure is trained using the training data generated by the training data generation method in the one aspect of the present disclosure.

The information processing apparatus according to one aspect of the present disclosure acquires an image taken by the endoscope, and acquires voice data by the operator or the subject of the endoscope recorded in association with the acquired image. It includes an acquisition unit and a generation unit that generates training data for a deep neural network based on the label corresponding to the recognition result and the image according to the recognition result of the acquired voice data.

According to the present disclosure, it is possible to provide a training data generation method or the like that efficiently generates training data for a trained model using an image taken by an endoscope.

It is a schematic diagram which shows the outline of the training data generation system which concerns on Embodiment 1. FIG. It is a block diagram which shows the structural example of the endoscope apparatus (microphone) included in the training data generation system. It is a block diagram which shows the configuration example of the information processing apparatus included in the training data generation system. It is explanatory drawing which illustrates the data layout of the medical term DB. It is a functional block diagram which illustrates the functional part included in the control part of an information processing apparatus or the like. It is a flowchart which shows an example of the processing procedure by the control part of an information processing apparatus. It is explanatory drawing about the generation processing of the 1st trained model. It is a flowchart which shows an example of the processing procedure about the generation processing of the 1st trained model by the control part of an information processing apparatus. It is a flowchart which shows an example of the processing procedure by the control part of the information processing apparatus which concerns on Embodiment 2 (training data at the time of unvoiced sound). It is a flowchart which shows an example of the processing procedure by the control part of the information processing apparatus which concerns on Embodiment 3 (enlargement ratio). It is a flowchart which shows an example of the processing procedure by the control part of the information processing apparatus which concerns on Embodiment 4 (accuracy of presence or absence of a lesion). It is a flowchart which shows an example of the processing procedure by the control part of the information processing apparatus which concerns on Embodiment 5 (voice data of a subject). It is explanatory drawing about the generation process of the 2nd trained model.

(Embodiment 1)
Hereinafter, the present invention will be described in detail with reference to the drawings showing the embodiments thereof. FIG. 1 is a schematic diagram showing an outline of the training data generation system S according to the first embodiment. The training data generation system S includes an endoscope device 10 and an information processing device 6 communicably connected to the endoscope device 10.

The endoscope device 10 transmits an image (captured image) taken by the photographing element of the endoscope to the endoscope processor 20, and the endoscope processor 20 performs gamma correction, white balance correction, shading correction, and the like. By performing various image processing, an endoscopic image that is easy for the operator to see is generated.

A microphone 17 is connected to the endoscope device 10, and the remarks (voice) of a doctor or the like collected by the microphone 17 are recorded as voice data. The endoscope device 10 adds or associates time-related elements such as an imaging start time and a voice recording start time with respect to the endoscope image and the voice data, and associates the time axis in the endoscope image and the voice data with each other. , These endoscopic images and audio data are stored and output to the information processing apparatus 6. The microphone 17 is not limited to a wired microphone, and may be, for example, a wireless microphone using Bluetooth (registered trademark) or the like.

The information processing device 6 that has acquired the endoscopic image and voice data to which a time element is added or associated with is transmitted from the endoscope device 10 performs various information processing based on these endoscopic images and voice data. Perform, generate, record and output training data for deep neural networks.

In the present embodiment, the endoscope device 10 outputs a captured image, that is, an endoscope image corrected with a raw image to the information processing device 6, but the present invention is not limited to this. The endoscope device 10 adds a time element to the captured image, associates the time axis in the captured image and the audio data, stores the captured image and the audio data, and outputs the captured image and the audio data to the information processing apparatus 6. May be good. In this case, the information processing apparatus 6 performs a series of processes based on the captured image to generate training data.

FIG. 2 is a block diagram showing a configuration example of the endoscope device 10 included in the training data generation system S. FIG. 3 is a block diagram showing a configuration example of the information processing apparatus 6 included in the training data generation system S. The endoscope device 10 includes an endoscope processor 20, an endoscope 40, and a display device 50. The display device 50 is, for example, a liquid crystal display device or an organic EL (Electro Luminescence) display device.

The display device 50 is installed on the upper stage of the storage shelf 16 with casters. The endoscope processor 20 is housed in the middle stage of the storage shelf 16. The storage shelf 16 is arranged in the vicinity of the endoscopy bed (not shown). The storage shelf 16 has a pull-out shelf on which the keyboard 15 connected to the endoscope processor 20 is mounted.

The endoscope processor 20 has a substantially rectangular parallelepiped shape and is provided with a touch panel 25 on one surface. A reading unit 28 is arranged at the bottom of the touch panel 25. The reading unit 28 is a connection interface for reading and writing a portable recording medium such as a USB connector, an SD (Secure Digital) card slot, or a CD-ROM (Compact Disc Read Only Memory) drive.

The endoscope 40 has an insertion unit 44, an operation unit 43, a universal cord 49, and a scope connector 48. The operation unit 43 is provided with a control button 431. The insertion portion 44 is long, and one end thereof is connected to the operation portion 43 via the folding portion 45. The insertion portion 44 has a flexible portion 441, a curved portion 442, and a tip portion 443 in this order from the operation portion 43 side. The bending portion 442 bends in response to the operation of the bending knob 433. A physical detection device such as a 3-axis acceleration sensor, a gyro sensor, a geomagnetic sensor, or a magnetic coil sensor is mounted on the insertion unit 44, and when the endoscope 40 is inserted into the body of the subject, the physical detection device is used. It may be the one to acquire the detection result of.

The universal cord 49 is long and has a first end connected to the operation unit 43 and a second end connected to the scope connector 48. The universal cord 49 is flexible. The scope connector 48 has a substantially rectangular parallelepiped shape. The scope connector 48 is provided with an air supply / water supply port 36 (see FIG. 2) for connecting an air supply / water supply tube.

The endoscope device 10 includes an endoscope processor 20, an endoscope 40, and a display device 50. In addition to the touch panel 25 and the reading unit 28, the endoscope processor 20 includes a control unit 21, a main storage device 22, an auxiliary storage device 23, a communication unit 24, a display device I / F (Interface) 26, and an input device I /. It includes an F27, an endoscope connector 31, a light source 33, a pump 34, and a bus. The endoscope connector 31 includes an electric connector 311 and an optical connector 312.

The control unit 21 is an arithmetic control device that executes the program of the present embodiment. For the control unit 21, one or a plurality of CPUs (Central Processing Units), GPUs (Graphics Processing Units), multi-core CPUs, and the like are used. The control unit 21 is connected to each hardware unit constituting the endoscope processor 20 via a bus.

The main storage device 22 is, for example, a storage device such as a SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), or a flash memory. The main storage device 22 temporarily stores information necessary in the middle of processing performed by the control unit 21 and a program being executed by the control unit 21. The auxiliary storage device 23 is, for example, a storage device such as a SRAM, a flash memory, or a hard disk, and is a storage device having a larger capacity than the main storage device 22. In the auxiliary storage device 23, for example, the acquired captured image, the generated endoscopic image, or the audio data data may be stored as intermediate data.

The communication unit 24 is a communication module or communication interface for communicating with an information processing device via a network by wire or wirelessly, and is, for example, a narrow-range wireless communication module such as wifi (registered trademark) or Bluetooth (registered trademark), or It is a wide area wireless communication module such as 4G and LTE. The touch panel 25 includes a display unit such as a liquid crystal display panel and an input unit laminated on the display unit.

The display device I / F 26 is an interface for connecting the endoscope processor 20 and the display device 50. The input device I / F 27 is an interface for connecting the endoscope processor 20 and an input device such as a keyboard 15 and a microphone 17.

The light source 33 is a high-intensity white light source such as a white LED or a xenon lamp. The light source 33 is connected to the bus via a driver (not shown). The lighting, extinguishing, and changing of the brightness of the light source 33 are controlled by the control unit 21. The illumination light emitted from the light source 33 is incident on the optical connector 312. The optical connector 312 engages with the scope connector 48 to supply illumination light to the endoscope 40.

The pump 34 generates pressure for the air supply / water supply function of the endoscope 40. The pump 34 is connected to the bus via a driver (not shown). The on / off and pressure change of the pump 34 are controlled by the control unit 21. The pump 34 is connected to the air supply water supply port 36 provided in the scope connector 48 via the water supply tank 35.

The outline of the function of the endoscope 40 connected to the endoscope processor 20 will be described. A fiber bundle, a cable bundle, an air supply tube, a water supply tube, and the like are inserted inside the scope connector 48, the universal cord 49, the operation unit 43, and the insertion unit 44. The illumination light emitted from the light source 33 is radiated from the illumination window provided at the tip portion 443 via the optical connector 312 and the fiber bundle. The range illuminated by the illumination light is photographed by an image sensor provided at the tip portion 443. The captured image is transmitted from the image pickup element to the endoscope processor 20 via the cable bundle and the electric connector 311.

The information processing device 6 includes a control unit 62, a communication unit 61, a storage unit 63, and an input / output I / F 64. The control unit 62 has an arithmetic processing unit having a timing function such as one or a plurality of CPUs (Central Processing Units), MPUs (Micro-Processing Units), GPUs (Graphics Processing Units), and is stored in the storage unit 63. By reading and executing the program P, various information processing, control processing, and the like related to the information processing unit 6 are performed. Alternatively, the control unit 62 may be composed of a chip for a quantum computer, and the information processing device 6 may be a quantum computer.

The storage unit 63 includes a volatile storage area such as a SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), and a flash memory, and a non-volatile storage area such as an EEPROM or a hard disk. The storage unit 63 stores the program P and the data to be referred to at the time of processing in advance. The program P stored in the storage unit 63 may be a program P read from the recording medium 632 that can be read by the information processing apparatus 6. Further, the program P may be downloaded from an external computer (not shown) connected to a communication network (not shown) and stored in the storage unit 63. The storage unit 63 stores the actual files (instance files of the deep neural network (DNN)) of the first trained model 91 and the second trained model 92, which will be described later. The medical term DB631 (DataBase) described later is stored in the storage unit 63.

The communication unit 61 is a communication module or communication interface for communicating with the endoscope device 10 by wire or wirelessly, for example, a narrow-range wireless communication module such as wifi (registered trademark), Bluetooth (registered trademark), or 4G. , LTE and other wide area wireless communication modules.

The input / output I / F64 conforms to a communication standard such as USB or DSUB, and is a communication interface for serial communication with an external device connected to the input / output I / F64. For example, a display unit 7 such as a display and an input unit 8 such as a keyboard are connected to the input / output I / F 64, and the control unit 62 performs information based on an execution command or event input from the input unit 8. The processing result is output to the display unit 7.

FIG. 4 is an explanatory diagram illustrating the data layout of the medical term DB631. The medical term DB631 includes, for example, a medical term, a jargon flag, a standard term, a term classification, and a medical institution ID as management items (metadata). The medical term DB631 is stored in the storage unit 63 of the information processing device 6, and is configured by database management software such as RDBMS (Relational DataBase Management System) mounted on the information processing device 6.

In the item (field) of medical terms, a term indicating the name or symptom of the lesion (lesion type name) and a term indicating the name of the location, place or internal site where the lesion occurs (lesion location name) are stored. Will be done. The term indicating the name or symptom of the lesion (lesion type name) is a medically defined standard term (standard term) such as cancer, polyp, etc., and further substantially these standard terms are used. Includes jargon to mean. The jargon is, for example, "K", which is a standard term for cancer, and K is synonymous with cancer. The terms (lesion location names) that indicate the location, location, or internal site where the lesion is occurring are, for example, upper esophagus and lower esophagus, and like the lesion type name, these standard terms are substantially used. It may include a meaning hidden word. The terms stored in the items (fields) such as medical terms are not limited to Japanese, and may be multilingual including English, Chinese, German, and the like.

In the item (field) of the jargon flag, flag data indicating whether or not the stored medical term is a jargon is stored. By setting a jargon flag for the stored medical term, it is possible to determine whether or not the medical term is a jargon, and post-processing can be performed based on the determination result.

Standard terms corresponding to jargon are stored in the standard term items (fields). If the stored medical term is a jargon, the standard term corresponding to the jargon is stored. Therefore, the jargon can be read or converted into a standard term. If the stored medical term is not a jargon, there is no need for conversion or the like, so that the item (field) of the standard term may be one in which data is not registered (null).

The item (field) of the term classification stores information for classifying whether the medical term is a term relating to the type of lesion or a term relating to the location of the lesion. Remarks by doctors, etc. include matters related to the type and location of lesions. By classifying medical terms in this way, the type and location of lesions should be classified and included in the correct answer data associated with the images described later. Can be done.

In the item (field) of the medical institution ID, when the medical term is a jargon, the number of the medical institution ID for identifying the medical institution in which the jargon is used is stored. Even if the jargon is the same word, it may have different meanings depending on the region and the institution. However, the medical institution ID is associated with each jargon and registered. Therefore, the jargon to be applied can be determined based on the medical institution ID, and the medical term can be reliably extracted from the remarks of doctors and the like.

FIG. 5 is a functional block diagram illustrating a functional unit included in the control unit 62 of the information processing apparatus 6 or the like. The control unit 21 of the endoscope processor 20 functions as an image processing unit 211, a clock unit 212, and an information linking unit 213 by executing a program stored in the main storage device. The control unit 62 of the information processing device 6 executes the program P stored in the storage unit 63 to execute the acquisition unit 621, the voice recognition unit 622, the medical term extraction unit 623, the image extraction unit 624, and the label derivation unit 625. , And functions as a training data generation unit 626.

The image processing unit 211 performs various image processing such as gamma correction, white balance correction, and shading correction on the image (photographed image) output from the endoscope, and outputs the image as an endoscope image.

The clock unit 212 has a clock function, and for example, acquires an imaging start time of a captured image and a voice recording start time by a microphone 17, and outputs the acquisition to the information linkage unit 213. Alternatively, the clock unit 212 may set a time difference between the imaging start time of the captured image and the audio recording start time, and output the time difference to the information linkage unit 213.

The information linkage unit 213 acquires the medical institution ID and the operator ID input via the keyboard 15. The medical institution ID is an identifier for identifying a medical institution. The operator ID is an identifier for identifying an operator of the endoscope device 10, that is, a doctor who examines a subject using the endoscope device 10.

The information linkage unit 213 acquires endoscope information output from the connected endoscope. Endoscope information includes information regarding the type of endoscope such as, for example, a bronchial endoscope, an upper gastrointestinal general-purpose endoscope, or a laparoscope.

The information linking unit 213 acquires voice data by voice collected by the microphone 17 and an endoscope image output from the image processing unit 211. The information linkage unit 213 acquires time-dependent elements such as the imaging start time of the captured image output from the clock unit 212 and the recording start time of the sound by the microphone 17. The information linkage unit 213 associates the acquired endoscopic image and audio data with a time element. The information linkage unit 213 associates, for example, an imaging start time with an endoscopic image in imparting a time-dependent element. Alternatively, the information linking unit 213 captured the frame in each frame (still image) included in the endoscope image based on the sampling time (frame rate) determined when capturing the endoscope image. Information (time stamp) indicating a time point may be added. The information linkage unit 213 associates, for example, the voice recording start time with the voice data in adding the time element.

By acquiring and associating the imaging start time of the captured image and the recording start time of the voice by the microphone 17, the captured image and the voice data are assumed to proceed on the same time axis, and the endoscope is based on the passage of time. The frame (still image) in the image and the pronunciation of the doctor or the like in the voice data can be linked and corresponded to each other. That is, when the captured image is a moving image, when a predetermined time has elapsed from the recording start time of the sound, the frame (still image) included in the captured image (moving image) captured at that time is specified. Can be done. As described above, since the endoscopic image is a corrected image of the captured image, the frame (still image) included in the endoscopic image captured at that time is specified also in the endoscopic image. It goes without saying that you can do it.

The information linking unit 213 associates the imaging start time with the endoscopic image, and associates the audio recording start time with the audio data, but the present invention is not limited to this. When the imaging start time and the audio recording start time are simultaneous, the information linking unit 213 provides information indicating that the imaging start time and the audio recording start time are simultaneous in the endoscopic image and audio data. It may be an addition.

Although it has been described that the captured image captured by the endoscope and the audio data collected and stored by the microphone 17 are different data, the present invention is not limited to this. The information linkage unit 213 may acquire or generate moving image data in AVI format, for example, in which captured images and audio data are integrated. In this case, the captured image captured by the endoscope becomes a video codec in the video data, the voice data collected and stored by the microphone 17 becomes a voice codec, and the video codec and the voice codec are based on the meta information included in the video data. Is synchronized, and the captured image and audio data are associated by a time factor.

When the endoscope image is a moving image, a still image is taken based on the frame rate. In the voice data, the frequency that becomes a phoneme is sampled based on the sampling rate. Therefore, the time point specified by the elapsed time from the imaging start time of the endoscopic image and the time point specified by the elapsed time from the sound recording start time can be derived based on the frame rate and the sampling rate.

The information linking unit 213 associates the acquired medical institution ID, operator ID, endoscope information, endoscope image with time-dependent elements, and voice data, and outputs the information processing device 6. The information linkage unit 213 may output the plurality of associated information and data as a single archive file.

In the present embodiment, the information linkage unit 213 adds a time element to the endoscopic image generated by correcting the captured image, and associates voice data with other data such as an operator ID, but the present invention is not limited to this. .. The information linkage unit 213 may add a time element to the captured image and output the voice data and other data such as the operator ID in association with each other. In this case, each functional unit in the information processing apparatus 6 performs a series of processes based on the captured image to generate training data.

The acquisition unit 621 acquires the medical institution ID, the operator ID, the endoscope information, the endoscope image and the audio data associated with the time element, which are output from the information linking unit 213 of the endoscope device 10. When the plurality of associated information and data are output as a single archive file, the acquisition unit 621 may decompress the archive file and expand it into individual data.

The acquisition unit 621 outputs the voice data associated with the operator ID and the time element to the voice recognition unit 622. The acquisition unit 621 outputs the endoscope information to the training data generation unit 626. The acquisition unit 621 outputs an endoscopic image associated with a time element to the image extraction unit 624. The acquisition unit 621 outputs the medical institution ID to the medical term extraction unit 623.

The speech recognition unit 622 includes submodules such as an acoustic model, a pronunciation dictionary, and a language model. The acoustic model analyzes frequency components and time changes, calculates the approximate amount of speech, that is, pronunciation, and each phoneme that is the basis of the acoustic model, and identifies the closest phoneme. A phoneme sequence is generated by the specified combination of phonemes. The pronunciation dictionary identifies the pronounced term (word) based on the degree of matching with the generated phoneme sequence. The language model is a model used to evaluate whether a word string (character string) by a specified word is appropriate, and is a data of the appearance probability of a word or a character string. The voice recognition unit 622 converts the input voice into a sound wave by using these submodules, and identifies a phoneme from the sound wave. The specified phoneme sequence is matched with a pre-registered pronunciation dictionary and converted into words, and the document is evaluated by using a language model so that the word string (character string) of the converted words becomes an appropriate sentence. Generate. The voice recognition unit 622 outputs the generated document in the form of character data, for example. The voice recognition unit 622 outputs the word included in the character data in association with the information about the time when the word is pronounced or the time stamp.

The voice of the doctor or the like specified by the operator ID is registered in the acoustic model in advance, and the voice recognition unit 622 indicates whether the recorded voice is the doctor who is the operator of the endoscope or the subject (examinee). It may be one that determines whether it is another person such as a patient) and performs a filtering process so as to extract only the voice by the doctor.

The medical term extraction unit 623 extracts each word included in the character data by, for example, performing morphological analysis on the character data, and the extracted words and the medical terms registered in the medical term DB 631 are each. By comparing or matching the characters, the medical terms contained in the character data are extracted.

By comparing each of the medical terms registered in the medical term DB631 with the pronunciation in this way, when the remarks made by the operator such as a doctor include matters unrelated to the diagnosis of the subject. Even so, medical terms related to diagnosis and the like can be reliably extracted in the statement, and the accuracy of training data generation can be improved.

The medical term extraction unit 623 extracts information (time stamp) about the time when the extracted medical term is pronounced from the character data, and outputs the medical term included in the character data and the time when the medical term is pronounced.

The medical term extraction unit 623 is not limited to outputting a single medical term, and may output a plurality of medical terms. That is, when a predetermined medical term is extracted from the acquired character data, the medical term extraction unit 623 has a plurality of medical terms based on the relevance or contextuality with other medical terms located before and after the extracted medical term. Medical terms may be extracted and output as a set. Alternatively, the medical term extraction unit 623 may output document data by a doctor's remark including a plurality of extracted medical terms. By collectively extracting remarks containing multiple medical terms, the period during which each of the multiple medical terms extracted collectively is pronounced is specified, and the still image (frame) in the endoscopic image captured during that period is specified. Can be identified.

When the medical term included in the character data is a jargon, a standard term synonymous with the jargon may be output as a medical term included in the character data. The remarks made by the doctors who are the operators may include not only standard terms in the medical industry but also peculiar jargon used depending on the region, medical field, etc. The jargon and standard terms that are synonymous with the jargon are registered. Therefore, by referring to the medical term DB631, the medical term extraction unit 623 can absorb the fluctuation of expression due to the use of the jargon or the like, and can surely extract the medical term from the remarks of the doctor or the like.

The medical term extraction unit 623 may determine the applied jargon based on the medical institution ID given to each jargon in the jargon used for comparison with the medical term included in the character data. Even if the jargon is the same word, it may have different meanings depending on the region and institution. However, in the above-mentioned medical term DB631, each jargon is associated with a medical institution ID and registered. ing. Therefore, the medical term extraction unit 623 can determine the jargon to be applied based on the acquired medical institution ID by referring to the medical term DB 631, and can surely extract the medical term from the remarks of doctors and the like.

The image extraction unit 624 extracts a frame (still image) corresponding to the time when the medical term is pronounced from the acquired endoscopic image (moving image). The number of frames (still images) to be extracted is not limited to one, and all frames captured during the period in which the medical term is pronounced may be extracted. The image extraction unit 624 outputs the extracted frame (still image) as the first image. The first image corresponds to the time when the medical term is pronounced, and is a frame (still image) captured at that time, and the presence of a lesion is suspected in the internal part included in the first image. It is a thing.

The label derivation unit 625 generates and outputs a label corresponding to the correct answer data (correct answer label) in the training data based on one or a plurality of medical terms acquired from the medical term extraction unit 623. The label is, for example, information regarding the presence or absence of a lesion, that is, the presence or absence of a lesion. Alternatively, the label may include information on symptoms such as the name of the type of lesion, or information on the location, location or name of the site in the body where the lesion occurred.

The training data generation unit 626 acquires the first image from the image extraction unit 624 and acquires the label from the label derivation unit 625. The training data generation unit 626 generates training data when a lesion is present, using the first image as problem data and the label as correct answer data (correct answer label) based on the acquired first image and label. The training data generation unit 626 outputs the generated training data and stores it in the storage unit 63. The training data generation unit 626 may store the first image and the label in the form of array data.

In the present embodiment, each functional unit in a series of processes has been described separately as a functional unit by the control unit 21 of the endoscope processor 20 and a functional unit by the control unit 62 of the information processing apparatus 6. The division of these functional units is an example, and is not limited to this. The control unit 21 of the endoscope processor 20 may function as all the functional units performed by the control unit 62 of the information processing device 6. That is, the endoscope processor 20 may substantially include the information processing device 6. Alternatively, the control unit 21 of the endoscope processor 20 only outputs the captured image captured by the image pickup element and the audio data collected by the microphone 17, and the control unit 62 of the information processing device 6 thereafter. It may function as all the functional parts that perform the processing of. Alternatively, the control unit 21 of the endoscope processor 20 and the control unit 62 of the information processing device 6 cooperate with each other to function as each functional unit in a series of processes by, for example, performing interprocess communication. There may be.

A single endoscopic examination produces an imaged endoscope image and recorded audio data, and the audio data includes a plurality of remarks by a doctor or the like. These multiple remarks include medical terms, and the frame (still image) of the endoscopic image taken at the same point based on the time or period when each of these medical terms is uttered (pronounced) is the first. Specify as one image. The first image taken at the time the medical term is spoken is the presence of a lesion associated with the medical term. Therefore, it is possible to efficiently extract a large amount of images corresponding to the problem data when the presence of lesions is the correct answer data.

FIG. 6 is a flowchart showing an example of a processing procedure by the control unit 62 of the information processing apparatus 6. The information processing apparatus 6 starts processing the flowchart based on the input contents from the input unit 8 connected to the own apparatus, for example. The flowchart in the present embodiment includes the processing of the endoscope processor, which is a prerequisite processing for the information processing device 6 to acquire an endoscope image or the like from the endoscope device 10 (endoscope processor). ..

The control unit 62 of the endoscope processor acquires the operator ID (S01). The control unit 62 of the endoscope processor acquires the medical institution ID (S02). The control unit 62 of the endoscope processor acquires an operator ID and a medical institution ID input via the keyboard 15 by a doctor or the like who is an operator of the endoscope.

The control unit 62 of the endoscope processor acquires endoscope information (S03). The control unit 62 of the endoscope processor communicates with the connected endoscope, for example, a check sequence, and acquires endoscope information regarding the type of the endoscope.

The control unit 62 of the endoscope processor acquires a captured image and generates an endoscope image (S04). The control unit 62 of the endoscope processor performs various image processing such as shading correction on the acquired captured image to generate an endoscope image in a state that is easy for an operator such as a doctor to see.

The control unit 62 of the endoscope processor acquires voice data (S05). The control unit 62 of the endoscope processor acquires voice data including remarks (voice) of a doctor or the like collected by a microphone 17 and records it in an auxiliary storage device 23 or the like.

The control unit 62 of the endoscope processor adds a time element to the endoscope image and audio data (S06). The control unit 62 of the endoscope processor adds a time element so that the endoscope image and the audio data can be matched on the time axis.

The control unit 62 of the endoscope processor outputs an endoscope image and audio data to which a time element is added, an operator ID, and the like (S07). The control unit 62 of the endoscope processor outputs various data acquired or generated such as an endoscope image and voice data to which a time element is added, an operator ID, and the like to the information processing apparatus 6.

The control unit 62 of the information processing device 6 acquires an endoscope image and voice data to which a time element is added, an operator ID, and the like from the endoscope device 10 (S101). The control unit 62 stores various data acquired from the endoscope device 10 in the storage unit 63.

The control unit 62 of the information processing apparatus 6 performs a voice recognition process included in the voice data (S102). The control unit 62 performs a voice recognition process included in the voice data, and generates character data based on the pronunciation included in the voice. The control unit 62 identifies a doctor or the like who has emitted a voice based on the acquired operator ID in performing voice recognition processing, recognizes only the voice by the specified doctor, and generates character data. May be good. Information about the time when the term included in the character data is pronounced is added to the character data. As described above, the voice of a doctor or the like specified by the operator ID is registered in advance in the acoustic model used for performing the voice recognition process. The control unit 62 determines whether the recorded voice is a doctor who is the operator of the endoscope or another person such as a subject (patient), and extracts only the voice by the doctor. It may be filtered.

The control unit 62 of the information processing apparatus 6 extracts the pronounced medical term based on the result of voice recognition (S103). The control unit 62 extracts each word included in the character data by, for example, performing morphological analysis on the character data generated based on the pronunciation included in the voice. The control unit 62 extracts the medical term included in the character data by comparing or matching each of the extracted words with each of the medical terms registered in the medical term DB 631. The control unit 62 stores the extracted medical term and the time point at which the medical term is spoken in the storage unit 63 in association with each other. As described above, when the extracted medical term is registered in the medical term DB631 as a jargon, the control unit 62 may store the standard term having the same meaning as the jargon as the extracted medical term. In this case, for example, even if the doctor's remark is "K in the lower esophagus", "K" is treated as a jargon, converted into the standard term "cancer" which is synonymous with "K", and "lower esophagus". The medical term is extracted as "cancer".

When applying the jargon registered in the medical term DB 631, the control unit 62 may determine the jargon group to be applied by using the input medical institution ID. As described above, the medical term DB 631 is configured by, for example, an RDBMS, and a medical institution ID that identifies a medical institution in which the jargon is used is registered for each jargon stored in the field of the medical term. Using the input medical institution ID, the control unit 62 extracts the jargon in which the medical institution ID is registered (the jargon that is the same record as the medical institution ID) as the jargon group to be applied. The control unit 62 compares the remarks of the doctor or the like with the extracted jargon group, identifies the jargon included in the jargon of the doctor or the like, converts it into a standard term synonymous with the specified jargon, and converts it into a medical term. Is extracted. In the present embodiment, the operator ID and the medical institution ID are acquired separately, but the present invention is not limited to this. The operator ID includes a number (information) that identifies the medical institution to which the doctor or the like who is the operator belongs, and the control unit 62 acquires only the operator ID and includes it in the operator ID. Extract the number that identifies the medical institution. The control unit 62 may extract a group of jargon to be applied by using the medical term DB631 based on the extracted number (corresponding to the medical institution ID).

The control unit 62 of the information processing apparatus 6 extracts a first image (frame) corresponding to the pronunciation time point of the extracted medical term (S104). The control unit 62 specifies the time of pronunciation of the extracted medical term, that is, the period during which the doctor or the like makes a statement including the medical term, and first sets the frame (still image) of the endoscopic image captured during the period. Extract as an image.

The control unit 62 of the information processing apparatus 6 derives a label based on the extracted medical terms (S105). When the extracted medical term includes a plurality of medical terms such as "cancer in the lower esophagus", the control unit 62 refers to the medical term DB631 to indicate the position of the lesion in the lower esophagus, and the cancer is the lesion. It is determined that it indicates a name (type), and a label is generated (derived) according to the determination result. As an example, when "cancer in the lower esophagus" is extracted, the control unit 62 generates (derives) a label consisting of "lesion: present / symptom: cancer / location: lower esophagus".

The control unit 62 of the information processing apparatus 6 generates training data based on the first image and the label (S106). The control unit 62 uses the first image, which is a frame captured during the period when the extracted medical terms are issued, as problem data, and the label derived based on the extracted medical terms as correct answer data (correct answer level). To generate. For example, if the period during which the extracted medical term is issued is 2 seconds and the frame rate is 50 frames / sec (fps), the number of first images corresponding to the extracted medical term is 100, and these first images Will be given the same label. The control unit 62 generates training data consisting of a first image and a label, for example, in the form of object type array data. Alternatively, the control unit 62 may register the training data including the first image and the label in a predetermined field of the database (training data DB) configured in advance.

The control unit 62 of the information processing device 6 stores the generated training data and the acquired endoscopic information in the storage unit 63 in association with each other (S107). The control unit 62 stores the generated training data in the storage unit 63 in association with the endoscope information including the type of the endoscope. Since the internal part to be examined differs depending on the type of endoscope, individual learning according to the type of endoscope or internal part is performed by storing the training data in association with the endoscopic information. You can generate a finished model.

FIG. 7 is an explanatory diagram relating to the generation process of the first trained model 91. The information processing apparatus 6 inputs an endoscopic image by learning based on the generated training data, and outputs information including at least one of the presence / absence of a lesion, the type (symptom) of the lesion, and the location of the lesion. A deep neural network (first trained model 91) is constructed (generated). As described above, the training data is composed of a first image which is problem data specified based on the remarks of doctors and the like, and a label which is correct answer data derived based on medical terms included in the remarks of doctors and the like. ..

The deep neural network (first trained model 91) trained using the training data is expected to be used as a program module that is a part of artificial intelligence software. The first trained model 91 is used in the information processing device 6 provided with the control unit 62 (CPU or the like) and the storage unit 63 as described above, and is used in the information processing device 6 having arithmetic processing capability as described above. By being executed, a neural network system is constructed. That is, the control unit 62 of the information processing apparatus 6 performs an operation to extract the feature amount of the endoscopic image input to the input layer according to the command from the first trained model 91, and determines the presence or absence of a lesion from the output layer. It operates to output information including at least one of the lesion type (symptom) and the location of the lesion.

The input layer has a plurality of neurons that receive input of the pixel value of each pixel included in the endoscopic image, and passes the input pixel value and distance information to the intermediate layer. The intermediate layer has a plurality of neurons for extracting the image features of the endoscopic image, and passes the active state of the neurons based on the extracted image features to the output layer. For example, when the first trained model 91 is a CNN (Convolutional Neural Network), the intermediate layer has a convolutional layer that convolves the pixel values of each pixel input from the input layer and a pixel value that is convolved by the convolutional layer. It has a configuration in which the pooling layers to be mapped (compressed) are alternately connected, and the feature amount of the endoscopic image is finally extracted while compressing the pixel information of the endoscopic image. The output layer has one or more neurons that output information including at least one of the presence or absence of a lesion, the type (symptom) of the lesion, and the location of the lesion contained in the endoscopic image, and is an intermediate layer. Information on the presence or absence of lesions is output based on the image feature amount and the like output from. Information including the presence or absence of the output lesion, the type (symptom) of the lesion, and at least one of the locations of the lesion is used as diagnostic support information by a doctor or the like.

In the present embodiment, the data input to the first trained model 91 is described as being an endoscopic image, but the data is not limited to this. The data input to the first trained model 91 may be a captured image captured by the image pickup device. That is, the first trained model 91 outputs information including at least one of the presence / absence of a lesion, the type (symptom) of a lesion, and the location of a lesion by inputting a photographed image and distance information. In this case, it is desirable that the training data is generated based on the captured image.

The information processing apparatus 6 compares the value output from the output layer with the information including the presence or absence of a labeled lesion with respect to the problem data (first image), that is, the correct answer value (correct answer data), and the output layer. The parameters used for the arithmetic processing in the intermediate layer are optimized so that the output value from is close to the correct answer value. The parameter is, for example, a weight between neurons (coupling coefficient), a coefficient of an activation function used in each neuron, and the like. The method of optimizing the parameters is not particularly limited, but for example, the information processing apparatus 6 optimizes various parameters by using the error back propagation method. The information processing apparatus 6 performs the above processing using the training data, generates the first trained model 91, and stores the generated first trained model 91 in the storage unit 63.

In the present embodiment, the information processing apparatus 6 is supposed to generate training data and a first trained model 91 using the training data, but the present invention is not limited to this. The generation of the first trained model 91 using the training data may be performed by an information processing device 6 different from the information processing device 6 in which the training data is generated.

FIG. 8 is a flowchart showing an example of a processing procedure related to the generation processing of the first trained model 91 by the control unit 62 of the information processing apparatus 6. The control unit 62 of the information processing apparatus 6 acquires training data (S120). The training data is composed of a first image which is problem data specified based on the remarks of doctors and the like, and a label which is correct answer data derived based on medical terms included in the remarks of doctors and the like.

A large amount of endoscopic images, which are the original data of such training data, and audio data, which stores statements made by doctors and the like, are stored as result data of endoscopic examinations performed at each medical institution, and these results are obtained. By using the data, a large amount of training data can be generated. Furthermore, by extracting medical terms in the remarks of doctors and the like from the voice data, it is considered that there is a lesion in order to identify the frame of the endoscopic image captured at the time when the remarks including the medical terms are made. It is possible to efficiently extract frames, reduce the man-hours for generating a large amount of training data, and save labor.

The control unit 62 of the information processing apparatus 6 generates the first trained model 91 (S121). The control unit 62 constructs (generates) the first trained model 91 by training the deep neural network using the acquired training data. When the first trained model 91 is a neural network, the parameters used for the arithmetic processing in the intermediate layer are optimized by using, for example, an error backpropagation method.

According to the present embodiment, a label having a correlation with the image is derived based on the recognition result of the voice data by the operator of the endoscope such as a doctor recorded in association with the image taken by the endoscope. Generate training data for deep neural networks based on images and labels. In the training data, the image taken by the endoscope corresponds to the problem data, and the label having a correlation with the image corresponds to the correct answer data. Since it is derived based on the above, the correct answer data can be efficiently derived, and the training data can be efficiently generated.

According to this embodiment, the label contains information about at least one of the presence or absence of a lesion, the type of lesion, or the location of a lesion at an internal site included in an image. Therefore, it is possible to efficiently create training data in which the image is used as problem data and information including any one or all of the presence / absence of a lesion, the type of a lesion, or the position of a lesion is used as correct answer data.

According to the present embodiment, a label having a correlation with the image is derived based on the comparison between the pronunciation term recognized from the voice data and the medical term group stored in advance in the medical term DB631, so that the label can be accurately labeled. Can be derived. Further, the medical term group stored in advance in the medical term DB631 includes a jargon and a standard term synonymous with the jargon, and when the recognized pronunciation term is a jargon, it is based on the standard term synonymous with the jargon. , A label having a correlation with the image is derived. Therefore, even when jargon or the like is used in the remarks made by the operator of the endoscope such as a doctor and expression fluctuation occurs, the label can be derived with high accuracy.

According to the present embodiment, information regarding the time when a term included in the medical term group is uttered is acquired, and the first image corresponding to the time when the term is uttered is specified. Therefore, it is possible to efficiently identify the first image at the time when the internal part where the lesion is suspected is photographed. Then, when the specified first image is used as the problem data, the training data indicating that there is a lesion can be efficiently generated by including the information regarding the existence of the lesion in the label corresponding to the correct answer data.

According to the present embodiment, the type of endoscope differs depending on the internal part to be imaged, that is, the internal part to be examined. On the other hand, by acquiring endoscope information including the type of endoscope and storing the endoscope information in association with the training data, the training data can be efficiently managed and each type of endoscope can be stored. It is possible to reliably perform training for a deep neural network suitable for.

According to the present embodiment, by training the deep neural network using the training data efficiently created by the above method, the number of steps required to generate the training data is reduced, and the image taken by the endoscope is taken. When is input, it is possible to generate a trained model that outputs information including the presence or absence of a lesion in an internal part included in the image.

(Embodiment 2)
FIG. 9 is a flowchart showing an example of a processing procedure by the control unit 62 of the information processing apparatus 6 according to the second embodiment (training data at the time of unvoiced sound). The information processing apparatus 6 starts processing the flowchart based on the input contents from the input unit 8 connected to the own apparatus, for example.

The control unit 62 of the information processing apparatus 6 performs the processes from S201 to S205 in the same manner as the processes S101 to S105 of the first embodiment. The control unit 62 of the information processing apparatus 6 derives the first label based on the extracted medical terms (S205). The control unit 62 of the information processing apparatus 6 generates training data with lesions based on the first image and the first label (S206). By performing the processing up to S206, training data based on the first image with the first label including the presence of a lesion as the correct label is generated.

The control unit 62 of the information processing apparatus 6 extracts frames other than the first image as the second image (S207). The control unit 62 extracts and specifies as a second image a frame other than the frame (still image) specified as the first image in the acquired endoscope image.

The control unit 62 of the information processing apparatus 6 derives the information indicating that there is no lesion as the second label of the second image (S208). The second image corresponds to a frame of an endoscopic image taken during a period in which a doctor or the like does not make a statement (a period of unvoiced sound) or a period in which a statement does not include medical terms. Therefore, the control unit 62 can determine that there is no lesion (no lesion) in the internal site included in these second images. The control unit 62 generates a second label consisting of, for example, "lesion: no / symptom: no / place: no".

The control unit 62 of the information processing apparatus 6 generates training data without lesions based on the second image and the second label (S209). The control unit 62 adds the training data (training data without lesions) generated based on the second image and the second label to the training data (training data with lesions) generated in S206.

The control unit 62 of the information processing device 6 stores the generated training data and the acquired endoscopic information in the storage unit 63 in association with each other (S210). The control unit 62 stores the generated training data, that is, the training data in which the training data with lesions and the training data without lesions are combined, in the storage unit 63 as in the process (S107) of the first embodiment.

According to the present embodiment, at the time when the second image other than the first image is captured, the terms included in the medical term group are not emitted. Therefore, when these second images are used as problem data, the correct answer data is obtained. By including information on the absence of lesions in the label corresponding to, it is possible to efficiently generate training data for the absence of lesions.

(Embodiment 3)
FIG. 10 is a flowchart showing an example of a processing procedure by the control unit 62 of the information processing apparatus 6 according to the third embodiment (enlargement ratio). Similar to the first embodiment, the information processing apparatus 6 starts processing of the flowchart based on the input contents from the input unit 8 connected to the own apparatus, for example. The control unit 62 of the information processing apparatus 6 performs the processes from S301 to S304 in the same manner as the processes S101 to S104 of the first embodiment.

The control unit 62 of the information processing apparatus 6 extracts an image (frame) that is an image of the same internal part as the extracted first image and has a different enlargement ratio as the first image (S305). The control unit 62 is a frame other than the extracted first image, and the frame is a frame in which the same internal part as the first image is imaged with respect to the frames located before and after the first image, and the enlargement ratio. Is different. The control unit 62 extracts a feature amount corresponding to the lesion from the extracted first image, and is a frame containing a feature amount similar to or similar to the feature amount extracted in a frame other than the first image, and is the first frame. Extract frames that have a different magnification from the image. The control unit 62 extracts a feature amount corresponding to the lesion from the extracted first image by, for example, edge detection, pattern matching with a color or shape corresponding to a pre-registered lesion, and the like. The control unit 62 recognizes the difference in the enlargement ratio by, for example, comparing the sizes of the extracted feature quantities in the frame.

The control unit 62 is an image (frame) in which the same internal part as the extracted first image is captured, and extracts a frame having an enlargement ratio smaller than the enlargement ratio of the first image as the first image. You may. When a doctor or the like refers to an endoscopic image for examination or diagnosis, if the image (frame) has a small magnification, if the presence of a lesion is suspected in the image (frame), refrain from making definitive statements. It may be silent, and it is assumed that the presence of the lesion is recognized by increasing the enlargement rate and the statement including medical terms is included. In such a case, the audio data at the time when the image (frame) having a small enlargement ratio is captured becomes silent, and the image (frame) is not extracted as the first image. Based on the relationship with the first image having a large enlargement ratio, the image (frame) having a small enlargement ratio can be extracted as the first image.

The control unit 62 of the information processing apparatus 6 performs the processes from S306 to S308 in the same manner as the processes S105 to S107 of the first embodiment.

According to the present embodiment, when a plurality of images having different enlargement ratios include the same internal part, or when any image of a plurality of images having different enlargement ratios is specified as a first image, any of the specified images is specified. An image having a smaller enlargement ratio than the image is also specified as the first image. Therefore, the remarks of doctors and the like at the time when an image with a large enlargement ratio is captured are applied to an image having a smaller enlargement ratio than the image, and a label having a correlation with these images having a smaller enlargement ratio is accurately derived. be able to.

(Embodiment 4)
FIG. 11 is a flowchart showing an example of a processing procedure by the control unit 62 of the information processing apparatus 6 according to the fourth embodiment (accuracy of the presence or absence of a lesion). Similar to the first embodiment, the information processing apparatus 6 starts processing of the flowchart based on the input contents from the input unit 8 connected to the own apparatus, for example. The control unit 62 of the information processing apparatus 6 performs the processes from S401 to S404 in the same manner as the processes S101 to S104 of the first embodiment.

The control unit 62 of the information processing apparatus 6 derives information regarding the certainty of the presence or absence of a lesion based on the amount of change in a plurality of images (frames) located before and after the extracted first image (S405). The control unit 62 derives the amount of change in pixel units between the extracted first image and a plurality of images (frames) located before and after the first image, and the accuracy of the presence or absence of a lesion based on the derived amount of change. To derive information about. That is, when the number of frames whose change amount from the extracted first image is less than or equal to a predetermined value is, for example, 200 and the frame rate is 50 fps, the doctor refers to the same internal part for 4 seconds. Become. That is, during this period, the endoscopic image is apparently stopped. In such a stopped state, that is, when the frames in which the amount of change from the extracted first image is a predetermined value or less are continuous, the doctor may be at a loss. Therefore, the control unit 62 derives information on the accuracy such as the presence or absence of a lesion according to the period determined based on the number of consecutive frames. For example, the control unit 62 may be derived by lowering the accuracy of the presence or absence of a lesion as the period becomes longer.

The control unit 62 of the information processing apparatus 6 derives a label based on the extracted medical term and the derived accuracy (S406). As an example, the control unit 62 generates a label consisting of "lesion: yes / symptom: cancer / location: lower esophagus / accuracy: 80%". The control unit 62 performs the processes of S407 and S408 in the same manner as the processes S106 and S107 of the first embodiment.

According to this embodiment, when it is difficult to determine the presence or absence of a lesion, doctors and the like tend to stop the movement of the endoscope and gaze at a specific internal part. On the other hand, by deriving information on the certainty of the presence of lesions based on the amount of change in the moving image over a unit time and including it in the label, information including the degree of gaze of a specific site by a doctor or the like is included in the label. be able to.

(Embodiment 5)
FIG. 12 is a flowchart showing an example of a processing procedure by the control unit 62 of the information processing apparatus 6 according to the fifth embodiment (voice data of the subject). Similar to the first embodiment, the information processing apparatus 6 starts processing of the flowchart based on the input contents from the input unit 8 connected to the own apparatus, for example. The control unit 62 of the information processing apparatus 6 performs the processes S501 to S504 in the same manner as the processes S101 to S104 of the first embodiment. In the present embodiment, the voice data collected and recorded by the microphone 17 includes utterance by the subject. Alternatively, the control unit 62 of the information processing apparatus 6 may acquire the voice data by the subject as data separate from the voice data by the doctor or the like.

The control unit 62 of the information processing apparatus 6 acquires the voice data of the subject corresponding to the first image (frame) (S505). The control unit 62 acquires the generated voice data of the subject at the time when the extracted first image (frame) is captured. The acquisition of the voice data of the subject is performed by cutting out the partial data of the period during which the voice of the subject is uttered from the voice data output from the endoscope processor 20. You may. Alternatively, the control unit 62 identifies the phoneme of the voice of the subject by using the above-mentioned acoustic model for the voice data output from the endoscope processor 20, and the voice of the subject is performed. The voice data of the subject may be acquired by cutting out the partial data of the period. The control unit 62 may specify the corresponding first image by matching or approaching the time points at which the images are taken, based on the period during which the subject makes a voice.

The control unit 62 of the information processing apparatus 6 performs the processing of S506 in the same manner as the processing S105 of the first embodiment.

The control unit 62 of the information processing apparatus 6 generates training data based on the first image, the voice data of the subject, and the label (S507). The control unit 62 generates training data using the problem data as the first image and the voice data of the subject, and using the label from which the correct answer data is derived. The control unit 62 of the information processing apparatus 6 performs the processing of S508 in the same manner as the processing S107 of the first embodiment.

According to the present embodiment, by acquiring the voice data by the subject of the endoscope, training data combining the image taken by the endoscope and the voice data by the subject of the endoscope is generated. Can be done. By including the voice data of the subject in the problem data in this way, training data for the second trained model 92 (see FIG. 13) effective for lesions related to, for example, pharyngeal cancer is efficiently generated. be able to.

FIG. 13 is an explanatory diagram relating to the generation process of the second trained model 92. The information processing apparatus 6 uses the first image and the voice data of the subject as problem data, and learns based on the training data in which the presence / absence of a lesion, the symptom, and the location of the lesion are correct data, so that the first image and the subject can be examined. A deep neural network (second trained model 92) is constructed (generated) by inputting the voice data of the person and outputting the presence / absence of a lesion, the symptom, and the location of the lesion.

The second trained model 92 includes an input layer, an intermediate layer, and an output layer like the first trained model 91. The voice data may be arranged using a voice analysis library such as LibROSA and used as input data of CNN in the same manner as the image data. Alternatively, using multimodal deep learning, an autoregressive layer by LSTM (Long short term memory) or the like is provided next to the input layer into which audio data is input, and the output from the autoregressive layer and the endoscopic image by CNN are used. The output from the intermediate layer from which the feature amount of the above is extracted may be combined by the fully connected layer and passed to the output layer.

According to the present embodiment, by including the voice data by the subject in the problem data, it is possible to efficiently generate the second trained model 92 that is effective for lesions related to, for example, pharyngeal cancer.

The embodiments disclosed this time should be considered to be exemplary in all respects and not restrictive. The technical features described in each embodiment can be combined with each other and the scope of the invention is intended to include all modifications within the scope of the claims and the scope of the claims. ..

S Training data generation system 10 Endoscope device 15 Keyboard 16 Storage shelf 17 Microscope 20 Endoscope processor 21 Control unit 211 Image processing unit 212 Clock unit 213 Information linkage unit 22 Main storage device 23 Auxiliary storage device 24 Communication unit 25 Touch panel 26 Display device I / F
27 Input device I / F
28 Reading unit 31 Endoscope connector 311 Electrical connector 312 Optical connector 33 Light source 34 Pump 35 Water supply tank 36 Air supply water supply port 40 Endoscope 43 Operation unit 431 Control button 433 Curved knob 44 Insertion part 441 Flexible part 442 Curved part 443 Tip 45 Folded connector 48 Scope connector 49 Universal code 50 Display device 6 Information processing device 61 Communication unit 62 Control unit 621 Acquisition unit 622 Voice recognition unit 623 Medical term extraction unit 624 Image extraction unit 625 Label derivation unit 626 Training data generation Department 63 Memory part 631 Medical term DB
632 Recording medium P program 64 I / O I / F
7 Display unit 8 Input unit 91 1st trained model 92 2nd trained model

Claims (11)

Acquire the image taken by the endoscope and
The voice data by the operator of the endoscope recorded in association with the acquired image is acquired, and the voice data is acquired.
A training data generation method for causing a computer to execute a process of generating training data for a deep neural network based on a label corresponding to the recognition result and the image according to the recognition result of the acquired voice data. The training data generation according to any one of claims 1, wherein the label contains information on any one of the presence or absence of a lesion, the type of lesion, or the location of a lesion in an internal site included in the image. Method. The recognition of the voice data includes a process of recognizing the pronunciation in the voice data.
Identify medical terms based on the contrast between the recognized pronunciation terms and the pre-memorized medical term groups.
The training data generation method according to claim 1 or 2, wherein the label is generated based on the specified medical term. The medical terminology includes jargon and standard terms synonymous with jargon.
The training data generation method according to claim 3, wherein when the recognized pronunciation term is the jargon, the medical term is specified based on a standard term synonymous with the jargon. The association between the image and the voice data of the operator is based on a time factor.
In the image, the first image corresponding to the utterance included in the voice data of the operator is specified.
The label associated with the identified first image includes information about the presence of the lesion.
The training data generation method according to any one of claims 1 to 4, wherein in the image, information regarding the absence of a lesion is included in a label associated with a second image that is an image other than the first image. The image includes a plurality of images having the same internal part and different magnifications.
The training data generation method according to any one of claims 1 to 5, wherein the same label is associated with the plurality of images having different enlargement ratios. The image taken by the endoscope is a moving image and is a moving image.
The training data generation method according to any one of claims 1 to 6, wherein information regarding the probability of the presence of a lesion is included in the label based on the amount of change in the moving image in a unit time. Obtaining voice data from the subject of the endoscope,
The invention according to any one of claims 1 to 7, wherein the training data for the deep neural network is generated based on the generated label, the image, and the voice data by the subject of the endoscope. Training data generation method. Obtaining endoscope information including the type of endoscope,
The training data generation method according to any one of claims 1 to 8, wherein the acquired endoscopic information is stored in association with the training data. A trained model trained using the training data generated by the training data generation method according to any one of claims 1 to 9. An acquisition unit that acquires an image taken by an endoscope and acquires voice data by an operator or a subject of the endoscope recorded in association with the acquired image.
An information processing device including a generation unit that generates training data for a deep neural network based on a label corresponding to the recognition result and the image according to the recognition result of the acquired voice data.

Images