JP7304508B2 - Information processing system and information processing program - Google Patents

Description

The present invention relates to an information processing system and an information processing program for identifying objects that are difficult to identify.

Eye surgery is commonly performed to treat eye diseases such as cataracts. When performing eye surgery, medical personnel cover the patient's face with a drape and expose the eye to be operated on through the opening of the drape (see Non-Patent Document 1).

https://www. tsukazaki-hp. jp/care/ophthalmology/ai

When the patient's face is draped, it can be difficult to distinguish whether the eye exposed through the drape is the left or right eye. Therefore, when the non-operating eye is erroneously exposed, the medical staff may misunderstand that the operating eye is exposed. Such a problem of misidentification is a problem that can occur even in sites other than medical sites. Therefore, it is desirable to provide an information processing system and an information processing program capable of identifying objects that are difficult to identify.

A first information processing system according to an embodiment of the present invention includes an acquisition section and a display section. The acquisition unit acquires the position information of the identification target or the region including the identification target in the captured image by performing object detection on the segmentation image obtained from the captured image including the identification target. The display unit displays an image in which the marker generated based on the position information acquired by the acquisition unit is superimposed on the captured image.

A second information processing system according to one embodiment of the present invention includes an acquisition unit and a synthesis unit. The acquisition unit acquires the position information of the identification target or the region including the identification target in the captured image by performing object detection on the segmentation image obtained from the captured image including the identification target. The synthesizing unit superimposes the marker generated based on the position information acquired by the acquiring unit on the image of the surgical field including the identification target formed by the optical system.

A third information processing system according to an embodiment of the present invention includes an acquisition section and a transmission section. The acquisition unit acquires the position information of the identification target or the region including the identification target in the captured image by performing object detection on the segmentation image obtained from the captured image including the identification target. The transmission unit transmits the position information acquired by the acquisition unit to the information processing device provided with the display unit.

A first information processing program according to an embodiment of the present invention causes a computer to execute the following two steps.
(1) Acquisition step of acquiring position information of an identification target or a region including an identification target in a captured image by performing object detection on a segmentation image obtained from a captured image including the identification target (2) Acquisition step A generation step of generating a video signal for displaying an image in which the marker generated based on the position information obtained in step is superimposed on the captured image

A second information processing program according to an embodiment of the present invention causes a computer to execute the following two steps.
(1) Acquisition step of acquiring position information of an identification target or a region including an identification target in a captured image by performing object detection on a segmentation image obtained from a captured image including the identification target (2) Acquisition step A generation step of generating a video signal for generating projection light for superimposing the marker generated based on the position information acquired in step on the image of the operative field including the identification target formed by the optical system

A third information processing program according to an embodiment of the present invention causes a computer to execute the following two steps.
(1) Acquisition step of acquiring position information of an identification target or a region including an identification target in a captured image by performing object detection on a segmentation image obtained from a captured image including the identification target (2) Acquisition step A transmission step of transmitting the position information acquired in to an information processing device equipped with a display unit

In the first information processing system, the second information processing system, the third information processing system, the first information processing program, the second information processing program, and the third information processing program according to one embodiment of the present invention, By performing object detection on the segmentation image obtained from the captured image including the identification target, the position information of the identification target or the region including the identification target in the captured image is obtained. As a result, for example, the marker generated based on the acquired position information is displayed as an image superimposed on the captured image, or superimposed on the image of the surgical field including the identification target formed by the optical system. becomes possible. As a result, it is possible to reliably include the elements necessary for identifying the identification target in the captured image, so that the difficult-to-identify target can be identified.

FIG. 10 is a diagram showing an example of an operating field image on which markers for position adjustment of an imaging region are superimposed; FIG. 10 is a diagram showing an example of an operating field image on which markers for position adjustment of an imaging region are superimposed; 1 is a diagram showing an example of a schematic configuration of a left/right eye identification system according to a first embodiment of the present invention; FIG. 4 is a diagram showing an example of functional blocks of the left/right eye identification system of FIG. 3; FIG. FIG. 3 is a diagram showing an example of image data stored in a memory of a server device; FIG. It is a figure showing an example of a Segmentation image. 5 is a flow chart showing an example of left/right eye identification procedures in the left/right eye identification system of FIG. 4; FIG. 7 is a diagram showing an example of a detection image obtained from the segmentation image of FIG. 6; FIG. 4 is a diagram showing an example of an image displayed on the display surface of FIG. 3; FIG. 4 is a diagram showing an example of an image displayed on the display surface of FIG. 3; FIG. 4 is a diagram showing an example of an image displayed on the display surface of FIG. 3; FIG. 4 is a diagram showing an example of an image displayed on the display surface of FIG. 3; FIG. 4 is a diagram showing an example of an image displayed on the display surface of FIG. 3; FIG. 4 is a diagram showing an example of an image displayed on the display surface of FIG. 3; FIG. FIG. 5 is a diagram showing a modified example of the functional blocks of the left/right eye identification system of FIG. 4; FIG. 16 is a flow chart showing an example of left and right eye identification procedures in the left and right eye identification system of FIG. 15; FIG. FIG. 4 is a diagram showing a modified example of the schematic configuration of the right and left eye identification system of FIG. 3; FIG. 18 is a diagram showing an example of functional blocks of the left/right eye identification system of FIG. 17; FIG. 19 is a flow chart showing an example of left and right eye identification procedures in the left and right eye identification system of FIG. 18; FIG. FIG. 4 is a diagram showing a modified example of the schematic configuration of the right and left eye identification system of FIG. 3; FIG. 21 is a diagram showing an example of functional blocks of the left/right eye identification system of FIG. 20; FIG. 21 is a diagram showing an example of functional blocks of the left/right eye identification system of FIG. 20; FIG. 4 is a diagram showing a modified example of the schematic configuration of the right and left eye identification system of FIG. 3; FIG. 24 is a diagram showing an example of functional blocks of the left/right eye identification system of FIG. 23; FIG. 10 is a diagram showing an example of a schematic configuration of a left/right eye identification system according to a second embodiment of the present invention; FIG. 26 is a diagram showing an example of functional blocks of the left/right eye identification system of FIG. 25; FIG. 27 is a diagram showing an example of an image projected from the projection unit of FIG. 26; FIG. 27 is a diagram showing an example of an image projected from the projection unit of FIG. 26; FIG. 27 is a diagram showing an example of an image projected from the projection unit of FIG. 26; FIG. 27 is a diagram showing an example of an image projected from the projection unit of FIG. 26; FIG. 27 is a diagram showing an example of an image projected from the projection unit of FIG. 26; FIG. 26 is a diagram showing an example of an image visually recognized through the eyepiece of FIG. 25; FIG. 27 is a flow chart showing an example of left and right eye identification procedures in the left and right eye identification system of FIG. 26; FIG. FIG. 27 is a diagram showing a modified example of the functional blocks of the left/right eye identification system of FIG. 26; FIG. 35 is a flow chart showing an example of a left/right eye identification procedure in the left/right eye identification system of FIG. 34; FIG. FIG. 26 is a diagram showing a modified example of the schematic configuration of the right and left eye identification system of FIG. 25; FIG. 37 is a diagram showing an example of functional blocks of the left/right eye identification system of FIG. 36; FIG. 38 is a flow chart showing an example of a left/right eye identification procedure in the left/right eye identification system of FIG. 37; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. The following description is a specific example of the present invention, and the present invention is not limited to the following aspects. In addition, the present invention is not limited to the arrangement, dimensions, dimensional ratios, etc. of each component shown in each drawing. Further, in the following description, directions such as up and down, left and right, and front and back are directions viewed from the side of medical staff unless otherwise specified. As for the vertical direction, the direction from the objective lens, which will be described later, toward the surgical field is defined as downward, and the opposite direction is defined as upward. Since the patient generally undergoes surgery in the supine position, the up-down direction is the same as the vertical direction. In addition, description is given in the following order.
1. Background of the Invention 2 . First embodiment (Figs. 1 to 14)
Generation of segmentation images and identification of left and right eyes are performed by the server device,
3. An example in which brightness determination, focus determination, detection image generation, and position determination are performed by an information processing apparatus. Modified example of the first embodiment Modified example A (Figs. 15 and 16)
Example in which brightness determination, focus determination, detection image generation, and position determination are also performed by the server device (example in which the information processing device functions as an interface)
Modification B (Figs. 17-19)
Example in which the server device is omitted Modification C (Figs. 20 to 22)
In the first embodiment and modification A,
An example of transmitting a captured image to an information processing device through communication different from the communication between the information processing device and the server device Modification D (FIGS. 23 and 24)
4. Example in which the server device is omitted from modification C; Second embodiment (Figs. 25 to 33)
Generation of segmentation images and identification of left and right eyes are performed by the server device,
5. An example in which an information processing apparatus with a camera performs brightness determination, focus determination, detection image generation, and position determination. Modification of Second Embodiment Modification E (FIGS. 34 and 35)
Example in which brightness judgment, focus judgment, detection image generation, and position judgment are also performed by the server device (an example in which an information processing device with a camera functions as an interface)
Modification F (Figs. 36-38)
Example in which the server unit is omitted

<1. Background of the Invention>
1 and 2 show an example of an operating field image Ia (captured image) on which a marker Mc for position adjustment of the imaging region is superimposed. In the operative field image Ia shown in FIGS. 1 and 2, an eye 10 to be identified in a left/right eye identification system (hereinafter simply referred to as a "right/left eye identification system") according to an embodiment of the present invention is shown. include. The eye 10 includes an eyeball 11 and a lacrimal hill 12 . The lacrimal dunes 12 are present at the end (inner corner) of the eye 10 near the nose. The operative field image Ia shown in FIGS. 1 and 2 corresponds to the state of the operative field when, for example, a medical worker stands by the patient's head and looks through a microscope installed above the patient's head. . Therefore, as shown in FIG. 1, when an image (Detection image Ib) obtained by performing object detection (Detection) on the surgical field image Ia includes the lacrimal hill 12 and the inner corner of the eye. For example, by superimposing the frame of the detection image Ib on the operative field image Ia as a marker, the medical staff can intuitively grasp whether or not the entire eye 10 is within the imaging area. be able to. In this case, the medical staff adjusts the imaging area so that the marker is positioned within the predetermined area in the surgical field image Ia, and the entire eye 10 is within the predetermined area in the surgical field image Ia. It is possible to correctly identify whether the eye 10 reflected in the field image Ia is the left eye or the right eye in the left/right eye identification system.

However, when object detection is actually performed on the operative field image Ia, for example, as shown in FIG. . Therefore, as shown in FIG. 2, when the frame of the detection image Ib is used as a marker and displayed so as to be superimposed on the operative field image Ia, the medical staff can check whether the entire eye 10 is within the imaging region. It can be difficult to grasp intuitively. In this case, even if the medical staff adjusts the imaging area so that the marker is positioned within the predetermined area in the surgical field image Ia, part of the eye 10 is outside the predetermined area in the surgical field image Ia. There is a possibility that the left/right eye identification system cannot correctly identify whether the eye 10 reflected in the operative field image Ia is the left eye or the right eye. Therefore, the present invention proposes a technique for ensuring that the detection image Ib includes elements that facilitate identification of the left and right eyes (for example, the lacrimal hills 12 and the inner corner of the eye).

<2. First Embodiment>
[composition]
FIG. 3 shows an example of a schematic configuration of the left/right eye identification system 100 according to the first embodiment of the present invention. The left/right eye identification system 100 corresponds to a specific example of the "information processing system" of the present invention. FIG. 4 shows an example of functional blocks of the left/right eye identification system 100. As shown in FIG. FIG. 3 illustrates an ophthalmic surgical microscope system 200 for magnifying and visualizing an operative field by medical personnel, together with a left/right eye identification system 100 . In the present embodiment, the operative field is a region of the patient's 400 face that is exposed in the opening 610 of the drape 600 placed over the patient's 400 face. The operative field includes the eye on which the surgery is performed. A patient 400 is lying on his back on an operating table 300 , his body is covered with a sheet 500 , and his face is covered with a drape 600 .

(Microscope system 200 for ophthalmic surgery)
The ophthalmic surgical microscope system 200 includes, for example, a post 210 , a first arm 220 , a second arm 230 , a drive device 240 , a medical personnel microscope 250 and a foot switch 260 .

Support 210 supports the entire ophthalmic surgical microscope system 200 . One end of the first arm 220 is connected to the upper end of the support 210 . One end of the second arm 230 is connected to the other end of the first arm 220 . A driving device 240 is connected to the other end of the second arm 230 . Medical personnel microscope 250 is suspended by drive 240 . The camera 120 used in the left/right eye identification system 100 is attached to the microscope 250 for medical personnel. The foot switch 260 is used to perform various operations with the foot of a medical worker or the like. The driving device 240 vertically and horizontally three-dimensionally moves the medical staff microscope 250 and the camera 120 according to the operation by the medical staff.

A microscope 250 for medical personnel has a lens barrel section 251 that accommodates various optical systems, drive systems, and the like, and a pair of left and right eyepiece sections 252 . The medical staff looks into the eyepiece 252 and observes the surgical field with both eyes through the lens barrel 251 . The lens barrel section 251 includes, for example, an objective lens, an illumination optical system, an observation optical system, and a photographing optical system. The observation optical system is an optical system connected to the objective lens, and the photographing optical system is an optical system connected to the camera 120 .

The illumination optical system illuminates the surgical field through the objective lens. The observation optical system is used to observe the surgical field illuminated by the illumination optical system with the medical staff microscope 250 through the objective lens. A pair of left and right observation optical systems are provided. The right observation optical system is connected to the right eyepiece 252 , and the left observation optical system is connected to the left eyepiece 252 . The right observation optical system is provided with, for example, a beam splitter, and part of the light that is guided from the surgical field to the right observation optical system is guided to the imaging optical system by the beam splitter.

(Left/right eye identification system 100)
The left/right eye identification system 100 includes, for example, an information processing device 110, a camera 120, and a server device . Information processing device 110 and server device 130 are connected via network 140 . Information processing device 110 is configured to be able to communicate with server device 130 via network 140 . Server device 130 is configured to be able to communicate with information processing device 110 via network 140 . Information processing device 110 and camera 120 are connected via network 140 . Information processing device 110 is configured to be able to communicate with camera 120 via network 140 . Camera 120 is configured to be able to communicate with information processing apparatus 110 via network 140 .

The network 140 is, for example, a network that performs communication using a communication protocol (TCP/IP) that is standardly used on the Internet. Network 140 may be, for example, a secure network that communicates using its own communication protocol. Network 140 is, for example, the Internet, an intranet, or a local area network. The connection between the network 140 and the information processing device 110, the server device 130, or the camera 120 may be, for example, a wired LAN (Local Area Network) such as Ethernet, a wireless LAN such as Wi-Fi, or a mobile phone. It may be a line or the like.

The information processing apparatus 110 includes, for example, a control unit 111 , a memory 112 , a display unit 113 , an input unit 114 and a network IF (Interface) 115 . In this embodiment, the control unit 111 corresponds to a specific example of the "acquisition unit" of the present invention. The display section 113 corresponds to a specific example of the "display section" of the present invention. The control unit 111 includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and the like, and executes, for example, a web browser program stored in the memory 112, an operating system, and the like (not shown). The control unit 111 further executes a program 112A stored in the memory 112, for example. The program 112A corresponds to one specific example of the "information processing program" of the present invention. The program 112A causes the controller 111 to execute a series of procedures for identifying the type of eye (left eye or right eye) included in the surgical field. A series of procedures that the program 112A causes the control unit 111 to execute will be described in detail later.

The memory 112 stores programs executed by the control unit 111 (for example, a web browser program and an operating system). The memory 112 is configured by, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), an auxiliary storage device (such as a hard disk), and the like. Memory 112 also stores program 112A.

The display unit 113 is, for example, a display device such as a liquid crystal panel or an organic EL (Electro Luminescence) panel. Display unit 113 displays an image based on the video signal from control unit 111 on display surface 110A. Input unit 114 receives an instruction from the outside (for example, a user) and outputs the received instruction to control unit 111 . The input unit 114 may be, for example, a mechanical input interface including buttons and dials, or a voice input interface including a microphone. The input unit 114 may be, for example, a touch panel provided on the display surface 110A of the information processing device 110 . Network IF 115 is a communication interface for communicating with server device 130 via network 140 .

The camera 120 includes, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary metal-oxide-semiconductor) image sensor. The camera 120 , for example, performs imaging under the control of the information processing device 110 and outputs the resulting image (operative field image Ia) to the information processing device 110 via the network 140 .

The server device 130 includes, for example, a control unit 131, a segmentation learning model 132, a left/right eye identification learning model 133, a memory 134, a network IF 135, and the like. The left/right eye identification learning model 133 corresponds to a specific example of the "identification section" of the present invention. The control unit 131 includes a CPU, a GPU, and the like, and executes, for example, a web server program stored in the memory 134, an operating system, and the like. Further, the control unit 131 transmits, for example, the identification result obtained by the left/right eye identification learning model 133 to the information processing apparatus 110 via the network IF 135 .

Network IF 135 is a communication interface for communicating with information processing apparatus 110 via network 140 . The memory 134 stores programs executed by the control unit 131 (for example, a web server program and an operating system). The memory 134 is composed of, for example, a RAM, a ROM, an auxiliary storage device (such as a hard disk), and the like. The memory 134 further stores a plurality of image data 134A, a plurality of image data 134B and a plurality of surgical data 134C.

Each image data 134A is one of two types of learning data input to the segmentation learning model 132 and one of two types of learning data input to the left/right eye identification learning model 133. is one. Each image data 134A is, for example, an operating field image Ia obtained by the camera 120 as shown in FIGS. Each image data 134B is one of two types of learning data input to the segmentation learning model 132. For example, as shown in FIG. It is a Segmentation image attached. Note that the image data 134B shown in FIG. 5 is composed of pixels labeled as the eye region Ra and pixels labeled as the peripheral region Rb. Each surgical data 134C is patient surgical data, and includes, for example, patient name, patient ID, disease name, surgery date and time, operated eye (left eye, right eye), surgical procedure, doctor in charge, and the like.

In the learning process, the segmentation learning model 132 performs machine learning using a set of image data 134A and 134B as learning data. As a result of machine learning, the segmentation learning model 132 generates and stores information about the correspondence between the image data 134A and the image data 134B as a learning result. In the conversion process, the segmentation learning model 132, when the surgical field image Ia obtained by the camera 120 is input, uses the learning result to convert the surgical field image Ia into the segmentation image Ic (see FIG. 6).

In the learning process, the left/right eye identification learning model 133 performs machine learning using the image data 134A and the eye type (left eye or right eye) included in the image data 134A as learning data. As a result of machine learning, the left/right eye identification learning model 133 generates and stores information on the correspondence relationship between the image data 134A and the type of eye as a learning result. In the identification process, when the surgical field image Ia is obtained, the left/right eye identification learning model 133 uses the learning result to identify the eye type (left eye or right eye) included in the surgical field image Ia.

[motion]
Next, an example of the operation of left/right eye identification system 100 according to the present embodiment will be described.

(machine learning)
The user instructs the segmentation learning model 132 to start learning via the input unit 114 . The Segmentation learning model 132 then performs a learning process. Specifically, the segmentation learning model 132 performs machine learning using a plurality of sets of image data 134A and image data 134B stored in the memory 134 as learning data. As a result of machine learning, the segmentation learning model 132 generates and stores information about the correspondence between the image data 134A and the image data 134B as a learning result. In this way, machine learning by the Segmentation learning model 132 ends.

Next, the user instructs the left/right eye identification learning model 133 to start learning via the input unit 114 . The left/right eye identification learning model 133 then executes a learning process. More specifically, the right/left eye identification learning model 133 receives thumbnails and identifiers of the plurality of image data 134A stored in the memory 134 and an input request for the type of eye (left eye, right eye) for each image data 134A. Send to processing device 110 . If there is no problem such as communication speed, the left/right eye identification learning model 133 may transmit the original image data 134A to the information processing apparatus 110 instead of the thumbnails. When information processing apparatus 110 receives an input request for the type of eye (left eye, right eye) for each image data 134A along with thumbnails and identifiers of a plurality of image data 134A from left/right eye identification learning model 133, information processing apparatus 110 receives a request to input the type of eye (left eye, right eye) for each image data 134A. A thumbnail of the image data 134A is displayed on the display unit 113, and a display prompting the user to input the type of eye (left eye, right eye) is displayed on the display unit 113. FIG. The user inputs the eye type (left eye, right eye) to the input section 114 for each image data 134A according to the display on the display section 113 . The input unit 114 transmits the eye type (left eye, right eye) for each image data 134A input by the user to the left/right eye identification learning model 133 together with the identifier of the image data 134A. When left/right eye identification learning model 133 receives the type of eye (left eye, right eye) for each image data 134A from information processing device 110 together with the identifier of image data 134A, the type of eye (left eye, right eye) is received. , and the identifier of the image data 134A is generated and stored. In this way, the machine learning by the left/right eye identification learning model 133 ends.

(Left and right eye identification)
FIG. 7 is a flowchart showing an example of left/right eye identification procedures in left/right eye identification system 100 . First, the user instructs the start of left/right eye identification via the input unit 114 . The user instructs to start the program 112A via the input unit 114, for example. Then, the information processing device 110 (control unit 111 ) reads the program 112 A from the memory 112 and loads it into the control unit 111 . The control unit 111 loaded with the program 112A executes a series of procedures for identifying left and right eyes. Specifically, first, control unit 111 outputs an imaging instruction to camera 120 . The camera 120 takes an image of the inside of the opening of the drape 600 in accordance with the imaging instruction, and transmits the resulting surgical field image Ia to the information processing apparatus 110 .

The information processing device 110 (control unit 111) performs brightness determination on the received surgical field image Ia. For example, the information processing device 110 (control unit 111) converts the operative field image Ia into HSV, and if the V value of each pixel obtained by the conversion is outside a predetermined range, the operative field image Ia is bright. The display unit 113 displays that the brightness is insufficient or the brightness is too high. The information processing device 110 (control unit 111) performs focus determination, for example, when the obtained V value of each pixel is within a predetermined range. For example, the information processing apparatus 110 (control unit 111) performs Laplacian differentiation on the operative field image Ia, and when the score obtained thereby is below a predetermined threshold, the operative field image Ia is out of focus. The presence is displayed on the display unit 113 . The information processing device 110 (control unit 111) transmits the operative field image Ia to the server device 130, for example, when the derived score is equal to or greater than a predetermined threshold.

When the server device 130 receives the surgical field image Ia from the information processing device 110, the Segmentation learning model 132 executes the conversion process. Specifically, the segmentation learning model 132 uses the learning result to convert the surgical field image Ia into the segmentation image Ic (see FIG. 6). The segmentation learning model 132 transmits the segmentation image Ic obtained by conversion to the information processing device 110 . The information processing device 110 (control unit 111) performs object detection on the received segmentation image Ic. For example, as shown in FIG. 8, the information processing device 110 (control unit 111) cuts out a rectangular region including the eye region Ra (for example, a rectangular region in contact with the eye region Ra) from the segmentation image Ic. . An image obtained in this manner is referred to as a Detection image Id.

The information processing device 110 (control unit 111) uses the position information of the obtained detection image Id in the operating field image Ia as the position information of the region including the eye to be identified in the operating field image Ia. The information processing apparatus 110 (control unit 111), for example, takes the position information of the outer edge portion of the obtained detection image Id in the surgical field image Ia as the position information of the marker Mc. That is, the information processing apparatus 110 (control unit 111) acquires position information of a region including the eye to be identified in the surgical field image Ia by performing object detection on the segmentation image Ic.

The information processing apparatus 110 (control unit 111) may use the position information of the obtained detection image Id in the operating field image Ia as the position information of the eye to be identified in the operating field image Ia. The information processing apparatus 110 (control unit 111) may use, for example, the position information of the outer edge portion of the eye area Ra of the obtained detection image Id in the operating field image Ia as the position information of the marker Mc, which will be described later. In this case, the information processing apparatus 110 (control unit 111) acquires position information of the eye to be identified in the surgical field image Ia by performing object detection on the segmentation image Ic.

After generating the marker Mc based on the acquired position information, the information processing apparatus 110 (control unit 111) displays an image (marker image Ie) in which the generated marker Mc is superimposed on the surgical field image Ia. A signal is generated and output to the display unit 113 . For example, as shown in FIG. 9, the display unit 113 displays an image (marker image Ie) in which the marker Mc is superimposed on the surgical field image Ia on the display surface 110A based on the video signal from the control unit 111. .

The left/right eye identification system 100 periodically performs a series of procedures from the imaging instruction to the marker Mc display described above. At this time, the medical staff finely adjusts the imaging area of the camera 120 while referring to the image displayed on the display surface 110A. The information processing apparatus 110 (the control unit 111) determines whether the position of the marker Mc protrudes from the predetermined area in the surgical field image Ia each time, for example, the above-described series of procedures from the imaging instruction to the display of the marker Mc is executed. determine whether As a result, when the position of the marker Mc protrudes from the predetermined area in the surgical field image Ia, the information processing apparatus 110 (control unit 111), for example, determines whether the position of the marker Mc is in the predetermined area in the surgical field image Ia. A video signal is generated for displaying an image including information indicating that the image is protruding from the display unit 113 and is output to the display unit 113 . The display unit 113 displays, on the display surface 110A, an image including information indicating that the position of the marker Mc is out of a predetermined area in the surgical field image Ia, and the surgical field image Ia. For example, as shown in FIG. 10, the display unit 113 displays on the display surface 110A an image including a status ST such as "the position of the microscope is shifted", a marker Mc, and an operating field image Ia. . Note that the information processing apparatus 110 (control unit 111) uses a method other than image display on the display surface 110A (for example, sound, patrol lamp lighting, etc.) to move the position of the marker Mc out of the predetermined area in the surgical field image Ia. You may want to inform your healthcare provider that you are

On the other hand, when the position of the marker Mc is within the predetermined area in the operative field image Ia, the information processing device 110 (control unit 111) determines that the position of the marker Mc is within the predetermined area in the operative field image Ia. A video signal for displaying an image indicating that there is an image is generated and output to the display unit 113 . The display unit 113, for example, based on the video signal from the control unit 111, includes information indicating that the position of the marker Mc is within a predetermined area in the operative field image Ia, and the operative field image Ia. An image is displayed on the display surface 110A. For example, as shown in FIG. 11, the display unit 113 displays an image including a status ST such as "microscope position OK", a marker Mc, and an operating field image Ia on the display surface 110A. Note that the information processing apparatus 110 (control unit 111) uses a method other than image display on the display surface 110A (for example, sound, patrol lamp lighting, etc.) to ensure that the position of the marker Mc is within a predetermined area in the operating field image Ia. You may want to let your healthcare provider know.

The information processing device 110 (control unit 111), when the position of the marker Mc is within a predetermined region in the operative field image Ia, sets the left and right eye identification learning model 133 in the server device 130 to the left and right eye Output an identification indication. When the left/right eye identification learning model 133 receives the left/right eye identification instruction, the left/right eye identification learning model 133 performs the identification process. Specifically, the left/right eye identification learning model 133 uses the learning result to identify the eye included in the detection image Ib. More specifically, the left/right eye identification learning model 133 uses the learning result to identify the eye type (left eye, right eye) included in the detection image Ib. The left/right eye identification learning model 133 transmits the identification result DR to the information processing device 110 . The control unit 111 generates a video signal for displaying an image including the status ST, the identification result DR, the marker Mc, and the surgical field image Ia on the display surface 110A, and outputs the video signal to the display unit 113 . For example, as shown in FIG. 12, the display unit 113 displays an image including the status ST, the identification result DR, the marker Mc, and the operative field image Ia based on the video signal from the control unit 111. 110A. For example, as shown in FIG. 12, the display unit 113 may display the identification result DR superimposed on the surgical field image Ia, or, for example, as shown in FIG. It may be displayed in an area different from the wild image Ia.

By looking at the identification result DR displayed on the display surface 110A, the medical staff can determine whether the eye exposed in the opening 610 of the drape 600 is the right eye or the left eye, and whether it is the eye to be operated on. can be confirmed. In this way, left and right eye identification is performed.

The information processing device 110 (control unit 111) further reads, for example, surgical data 134C corresponding to the patient 400 from a plurality of surgical data 134C stored in the memory 134, The operated eye is compared with the identification result DR. The information processing device 110 (control unit 111) outputs a video signal for displaying an image including the comparison result CR, the status ST, the identification result DR, the marker Mc, and the surgical field image Ia on the display surface 110A. It is generated and output to display unit 113 . For example, as shown in FIG. 14, the display unit 113 displays the comparison result CR, the status ST, the identification result DR, the marker Mc, and the operative field image Ia based on the video signal from the control unit 111. The image containing the image is displayed on the display surface 110A. Note that FIG. 14 exemplifies display when the operated eye included in the read surgery data 134C matches the identification result DR as the comparison result CR.

[effect]
Next, the effects of the left/right eye identification system 100 according to the present embodiment will be described.

Eye surgery is commonly performed to treat eye diseases such as cataracts. When performing eye surgery, medical personnel cover the patient's face with a drape and expose the eye being operated on through an opening in the drape. When the patient's face is draped, it can be difficult to distinguish whether the eye exposed through the drape is the left or right eye. Therefore, when the non-operating eye is erroneously exposed, the medical staff may misunderstand that the operating eye is exposed.

On the other hand, in the present embodiment, by performing object detection on the Segmentation image Ic obtained from the operative field image Ia including the eye to be identified, the eye to be identified or the eye including the eye to be identified is detected. Positional information of the region in the surgical field image Ia is acquired. Furthermore, in the present embodiment, an image (marker image Ie) in which a marker Mc generated based on the acquired position information is superimposed on the surgical field image Ia is displayed. This allows the marker Mc to reliably include an element (for example, the lacrimal gland 12 and the inner corner of the eye) that facilitates identification of the left and right eyes. As a result, the medical staff adjusts the imaging area of the camera 120 so that the marker Mc is positioned within a predetermined area in the operative field image Ia, so that the eye captured in the operative field image Ia is the left eye. or the right eye can be correctly identified in the left/right eye identification system 100 . Therefore, it is possible to identify eye types that are difficult to identify.

Further, in the present embodiment, the identification result DR obtained by the left/right eye identification learning model 133 is displayed on the display surface 110A of the display unit 113 . Further, in the present embodiment, the identification result DR by the left/right eye identification learning model 133 is displayed on the display surface 110A of the display unit 113 together with the operative field image Ia. As a result, it is possible to perform reliable confirmation work without overlooking.

Further, in the present embodiment, by performing object detection on the segmentation image Ic, after the detection image Id is clipped from the segmentation image Ic, the position information of the clipped detection image Id in the surgical field image Ia is , position information of the eye to be identified, or a region including the eye to be identified, in the surgical field image Ia. As a result, compared to the case where object detection is simply performed on the surgical field image Ia, and the position information of the eye to be identified in the surgical field image Ia is derived based on the resulting detection image. , the marker Mc can certainly include an element (for example, the lacrimal gland 12 or the inner corner of the eye) that facilitates identification of the left and right eyes. As a result, the medical staff adjusts the imaging area of the camera 120 so that the marker Mc is positioned within a predetermined area in the operative field image Ia, so that the eye captured in the operative field image Ia is the left eye. or the right eye can be correctly identified in the left/right eye identification system 100 . Therefore, it is possible to identify eye types that are difficult to identify.

In addition, in the present embodiment, when the marker Mc protrudes from the predetermined area in the surgical field image Ia, information indicating that the marker Mc protrudes from the predetermined area (for example, status ST) is included. An image is displayed on the display surface 110A of the display unit 113 . Accordingly, it is possible to reliably notify the medical staff that fine adjustment of the imaging area of the camera 120 is required.

Further, in the present embodiment, after the surgical field image Ia obtained when the marker Mc is within a predetermined region in the surgical field image Ia is transmitted to the server device 130, the left and right eye identification results are sent to the server device. A network IF 115 that receives data from 130 is provided in the information processing apparatus 110 . As a result, a device (server device 130 ) different from the information processing device 110 can handle heavy-load processing performed by the segmentation learning model 132 , the left/right eye identification learning model 133 , and the like. As a result, for example, each information processing device 110 installed in each operating room can share one server device 130. Therefore, compared to the case where the right and left eye identification system 100 is installed in each operating room, it is possible to identify right and left eyes. The installation cost of system 100 can be reduced.

Further, in the present embodiment, the network IF 115 receives the surgical field image Ia transmitted from the camera 120 . As a result, it is not necessary to provide a network IF separate from the network IF 115 for communication with the camera 120, so that an inexpensive device can be used as the information processing device 110. FIG. As a result, the cost of the left/right eye identification system 100 can be reduced.

<3. Modification of First Embodiment>
Next, a modification of the left/right eye identification system 100 according to the first embodiment will be described.

[[Modification A]]
FIG. 15 shows a modified example of functional blocks of the left/right eye identification system 100 according to the above embodiment. Information processing apparatus 110 according to this modification stores program 112B in memory 112 instead of program 112A. The server device 130 according to this modification further stores a program 134D in the memory 134. FIG. The program 134D corresponds to one specific example of the "identification program" of the present invention.

The program 112</b>B causes the control unit 111 to execute a procedure for causing the information processing device 110 to function as a user interface in the left/right eye identification system 100 . The program 134D causes the controller 131 to execute a series of procedures for identifying the type of eye (left eye or right eye) included in the operative field.

(Left and right eye identification)
FIG. 16 is a flow chart showing an example of the left/right eye identification procedure in the left/right eye identification system 100 according to this modification. First, the user instructs the start of left/right eye identification via the input unit 114 . The user instructs the activation of the programs 112B and 134D via the input unit 114, for example. Then, the information processing device 110 (control unit 111 ) reads the program 112 B from the memory 112 and loads it into the control unit 111 . The control unit 111 loaded with the program 112B executes a procedure for causing the information processing device 110 to function as a user interface in the left/right eye identification system 100 . Server device 130 (control unit 131 ) reads program 134 D from memory 134 and loads it into control unit 131 . The control unit 131 loaded with the program 134D executes a series of procedures for identifying left and right eyes.

Specifically, first, control unit 111 outputs an imaging instruction to camera 120 . The camera 120 takes an image of the inside of the opening of the drape 600 in accordance with the imaging instruction, and transmits the resulting surgical field image Ia to the information processing apparatus 110 .

The information processing device 110 (control unit 111 ) transmits the received surgical field image Ia to the server device 130 . The server device 130 (control unit 131) performs brightness determination on the received surgical field image Ia. For example, the server device 130 (control unit 131) converts the operative field image Ia into HSV, and if the V value of each pixel obtained by the conversion is outside a predetermined range, the operative field image Ia is A signal is transmitted to the information processing device 110 to cause the display unit 113 to display that the brightness is insufficient or too bright. Based on the signal from the server device 130, the information processing device 110 displays on the display unit 113 that the operative field image Ia is insufficiently bright or too bright.

The server device 130 (control unit 131) performs focus determination, for example, when the obtained V value of each pixel is within a predetermined range. The server device 130 (control unit 131) performs, for example, Laplacian differentiation on the operative field image Ia, and if the resulting score is below a predetermined threshold, the operative field image Ia is out of focus. A signal is transmitted to the information processing apparatus 110 to cause the display unit 113 to display the fact. Based on the signal from the server device 130, the information processing device 110 displays on the display unit 113 that the surgical field image Ia is out of focus. The server device 130 (control unit 131) inputs the surgical field image Ia to the segmentation learning model 132, for example, when the derived score is equal to or greater than a predetermined threshold.

The Segmentation learning model 132 executes a conversion process when the surgical field image Ia is input. Specifically, the segmentation learning model 132 uses the learning result to convert the surgical field image Ia into the segmentation image Ic (see FIG. 6). The segmentation learning model 132 inputs the segmentation image Ic obtained by conversion to the control unit 131 . The control unit 131 performs object detection on the input segmentation image Ic. For example, as shown in FIG. 8, the control unit 131 cuts out a rectangular region including the eye region Ra (for example, a rectangular region in contact with the eye region Ra) from the segmentation image Ic. An image obtained in this manner is referred to as a Detection image Id.

The control unit 131 uses the position information of the obtained detection image Id in the operating field image Ia as the position information of the region including the eye to be identified in the operating field image Ia. The control unit 131 uses, for example, the position information of the outer edge portion of the obtained detection image Id in the surgical field image Ia as the position information of the marker Mc. That is, the control unit 131 acquires the position information of the region including the eye to be identified in the surgical field image Ia by performing object detection on the segmentation image Ic.

Note that the control unit 131 may use the position information of the obtained detection image Id in the operating field image Ia as the position information of the eye to be identified in the operating field image Ia. For example, the control unit 131 may use the position information of the outer edge portion of the eye area Ra of the obtained detection image Id in the surgical field image Ia as the position information of the marker Mc. In this case, the control unit 131 acquires the position information of the eye to be identified in the surgical field image Ia by performing object detection on the segmentation image Ic.

After generating the marker Mc based on the acquired position information, the control unit 131 transmits to the information processing apparatus 110 an image (marker image Ie) in which the generated marker Mc is superimposed on the surgical field image Ia. The information processing device 110 (control unit 111 ) generates a video signal for displaying an image including the received marker image Ie, and outputs the video signal to the display unit 113 . For example, as shown in FIG. 9, based on the video signal from the control unit 111, the display unit 113 displays an image in which the marker Mc is superimposed on the operative field image Ia on the display surface 110A.

The left/right eye identification system 100 periodically performs a series of procedures from the imaging instruction to the marker Mc display described above. At this time, the medical staff finely adjusts the imaging area of the camera 120 while referring to the image displayed on the display surface 110A. For example, every time the server device 130 (control unit 131) executes the above-described series of procedures from the imaging instruction to the display of the marker Mc, whether or not the position of the marker Mc protrudes from the predetermined region in the surgical field image Ia. judge. As a result, when the position of the marker Mc protrudes from the predetermined area in the operating field image Ia, the server apparatus 130 (control unit 131), for example, Information indicating the protruding portion is transmitted to the information processing device 110 . Information processing device 110 (control unit 111 ) generates a video signal for displaying an image including the received information, and outputs the video signal to display unit 113 . For example, based on the video signal from the control unit 111, the display unit 113 displays information indicating that the position of the marker Mc in the operative field image Ia protrudes from a predetermined region, the marker Mc, and the operative field image Ia. and is displayed on the display surface 110A. For example, as shown in FIG. 10, the display unit 113 displays on the display surface 110A an image including a status ST such as "the position of the microscope is shifted", a marker Mc, and an operating field image Ia. . Note that the information processing apparatus 110 (control unit 111) uses a method other than image display on the display surface 110A (for example, sound, patrol lamp lighting, etc.) to move the position of the marker Mc out of the predetermined area in the surgical field image Ia. You may want to inform your healthcare provider that you are

On the other hand, when the position of the marker Mc is within the predetermined area in the operative field image Ia, the server apparatus 130 (control unit 131) determines that the position of the marker Mc is within the predetermined area in the operative field image Ia. Information indicating that is transmitted to the information processing apparatus 110 . Information processing device 110 (control unit 111 ) generates a video signal for displaying an image including the received information, and outputs the video signal to display unit 113 . For example, based on the video signal from the control unit 111, the display unit 113 displays information indicating that the position of the marker Mc is within a predetermined region in the operative field image Ia, the marker Mc, and the operative field image. An image including Ia is displayed on the display surface 110A. Note that the information processing apparatus 110 (control unit 111) uses a method other than image display on the display surface 110A (for example, sound, patrol lamp lighting, etc.) to ensure that the position of the marker Mc is within a predetermined area in the operating field image Ia. You may want to let your healthcare provider know.

The server device 130 (control unit 131) outputs a left/right eye identification instruction to the left/right eye identification learning model 133 when the position of the marker Mc is within a predetermined area in the surgical field image Ia. When the left/right eye identification learning model 133 receives the left/right eye identification instruction, the left/right eye identification learning model 133 performs the identification process. Specifically, the left/right eye identification learning model 133 uses the learning result to identify the eye included in the detection image Ib. More specifically, the left/right eye identification learning model 133 uses the learning result to identify the eye type (left eye, right eye) included in the detection image Ib. The left/right eye identification learning model 133 transmits the identification result to the control unit 131 . Control unit 131 transmits identification result DR to information processing apparatus 110 . The control unit 111 generates a video signal for displaying an image including the status ST, the identification result DR, the marker Mc, and the surgical field image Ia on the display surface 110A, and outputs the video signal to the display unit 113 . For example, as shown in FIG. 12, the display unit 113 displays an image including the status ST, the identification result DR, the marker Mc, and the operative field image Ia based on the video signal from the control unit 111. 110A. For example, as shown in FIG. 12, the display unit 113 may display the identification result DR superimposed on the surgical field image Ia, or, for example, as shown in FIG. It may be displayed in an area different from the wild image Ia.

By looking at the identification result DR displayed on the display surface 110A, the medical staff can determine whether the eye exposed in the opening 610 of the drape 600 is the right eye or the left eye, and whether it is the eye to be operated on. can be confirmed. In this way, left and right eye identification is performed.

The information processing device 110 (control unit 111) further reads, for example, surgical data 134C corresponding to the patient 400 from a plurality of surgical data 134C stored in the memory 134, The subject eye is compared with the identification result DR, a video signal for displaying an image including the comparison result CR on the display surface 110A is generated, and output to the display unit 113. FIG. For example, as shown in FIG. 14, the display unit 113 displays an image including the comparison result CR, the identification result DR, the status ST, and the surgical field image Ia based on the video signal from the control unit 111. It is displayed on the surface 110A. Note that FIG. 14 exemplifies display when the operated eye included in the read surgery data 134C matches the identification result DR as the comparison result CR.

In this modification, in the server device 130, object detection is performed on the segmentation image Ic obtained from the operative field image Ia including the eye to be identified. is acquired in the surgical field image Ia. Furthermore, in this modification, an image (marker image Ie) in which the marker Mc generated based on the acquired position information is superimposed on the surgical field image Ia is transmitted to the information processing apparatus 110 . Accordingly, since the marker image Ie can be displayed in the information processing device 110, it is possible to reliably include elements (for example, the lacrimal hills 12 and the inner corner of the eye) that facilitate identification of the left and right eyes in the marker Mc. can. As a result, the medical staff adjusts the imaging area of the camera 120 so that the marker Mc is positioned within a predetermined area in the operative field image Ia, so that the eye captured in the operative field image Ia is the left eye. or the right eye can be correctly identified in the left/right eye identification system 100 . Therefore, it is possible to identify eye types that are difficult to identify.

Further, in this modification, by performing object detection on the segmentation image Ic, after the detection image Id is clipped from the segmentation image Ic, the position information of the clipped detection image Id in the surgical field image Ia is: This is the positional information of the eye to be identified or the region containing the eye to be identified in the surgical field image Ia. As a result, compared to the case where object detection is simply performed on the surgical field image Ia, and the position information of the eye to be identified in the surgical field image Ia is derived based on the resulting detection image. , the marker Mc can certainly include an element (for example, the lacrimal gland 12 or the inner corner of the eye) that facilitates identification of the left and right eyes. As a result, the medical staff adjusts the imaging area of the camera 120 so that the marker Mc is positioned within a predetermined area in the operative field image Ia, so that the eye captured in the operative field image Ia is the left eye. or the right eye can be correctly identified in the left/right eye identification system 100 . Therefore, it is possible to identify eye types that are difficult to identify.

[[Modification B]]
In the above embodiment, for example, as shown in FIG. 17, the server device 130 may be omitted and the information processing device 110 may have the functions of the server device 130 . At this time, the information processing apparatus 110 includes, for example, a Segmentation learning model 132 and a left/right eye identification learning model 133, as shown in FIG. 134C is stored in memory 112 .

(Left and right eye identification)
FIG. 19 is a flow chart showing an example of the left/right eye identification procedure in the left/right eye identification system 100 according to this modification. First, the user instructs the start of left/right eye identification via the input unit 114 . The user instructs to start the program 112A via the input unit 114, for example. Then, the information processing device 110 (control unit 111 ) reads the program 112 A from the memory 112 and loads it into the control unit 111 . The control unit 111 loaded with the program 112A executes a series of procedures for identifying left and right eyes.

Specifically, first, control unit 111 outputs an imaging instruction to camera 120 . The camera 120 takes an image of the inside of the opening of the drape 600 in accordance with the imaging instruction, and transmits the resulting surgical field image Ia to the information processing apparatus 110 .

The control unit 111 performs brightness determination on the received surgical field image Ia. For example, the control unit 111 converts the operative field image Ia into HSV, and if the V value of each pixel obtained by the conversion is outside a predetermined range, the operative field image Ia is insufficiently bright or too bright. The presence is displayed on the display unit 113 .

For example, when the obtained V value of each pixel is within a predetermined range, the control unit 111 performs focus determination. For example, the control unit 111 performs Laplacian differentiation on the operative field image Ia, and if the resulting score is below a predetermined threshold, the display unit 113 indicates that the operative field image Ia is out of focus. to display. For example, when the derived score is equal to or greater than a predetermined threshold, the control unit 111 inputs the operative field image Ia to the segmentation learning model 132 .

The Segmentation learning model 132 executes a conversion process when the surgical field image Ia is input. Specifically, the segmentation learning model 132 uses the learning result to convert the surgical field image Ia into the segmentation image Ic (see FIG. 6). The segmentation learning model 132 inputs the segmentation image Ic obtained by conversion to the control unit 111 . The control unit 111 performs object detection on the input segmentation image Ic. For example, as shown in FIG. 8, the control unit 111 cuts out a rectangular region including the eye region Ra (for example, a rectangular region in contact with the eye region Ra) from the segmentation image Ic. An image obtained in this manner is referred to as a Detection image Id.

The control unit 111 uses the position information of the obtained detection image Id in the operating field image Ia as the position information of the region including the eye to be identified in the operating field image Ia. The control unit 111 uses, for example, the position information of the outer edge portion of the obtained detection image Id in the surgical field image Ia as the position information of the marker Mc. That is, the control unit 111 acquires the position information of the region including the eye to be identified in the surgical field image Ia by performing object detection on the segmentation image Ic.

Note that the control unit 111 may use the position information of the obtained detection image Id in the operating field image Ia as the position information of the eye to be identified in the operating field image Ia. For example, the control unit 111 may use the position information of the outer edge portion of the eye area Ra of the obtained detection image Id in the surgical field image Ia as the position information of the marker Mc. In this case, the control unit 111 acquires the position information of the eye to be identified in the surgical field image Ia by performing object detection on the segmentation image Ic.

After generating the marker Mc based on the acquired position information, the control unit 111 generates a video signal for displaying an image in which the generated marker Mc is superimposed on the surgical field image Ia, and outputs the video signal to the display unit 113. . For example, as shown in FIG. 9, based on the video signal from the control unit 111, the display unit 113 displays an image in which the marker Mc is superimposed on the operative field image Ia on the display surface 110A.

The left/right eye identification system 100 periodically performs a series of procedures from the imaging instruction to the marker Mc display described above. At this time, the medical staff finely adjusts the imaging area of the camera 120 while referring to the image displayed on the display surface 110A. For example, the control unit 111 determines whether or not the position of the marker Mc protrudes from a predetermined area in the surgical field image Ia each time the above-described series of procedures from the imaging instruction to the marker Mc display are executed. As a result, when the position of the marker Mc protrudes from the predetermined region in the surgical field image Ia, the control unit 111 detects that the position of the marker Mc protrudes from the predetermined region in the surgical field image Ia, for example. A video signal for displaying an image including meaningful information is generated and output to the display unit 113 . For example, based on the video signal from the control unit 111, the display unit 113 displays an image including the operative field image Ia and information indicating that the position of the marker Mc is out of a predetermined region in the operative field image Ia. is displayed on the display surface 110A. For example, as shown in FIG. 10, the display unit 113 displays on the display surface 110A an image including a status ST such as "the position of the microscope is shifted", a marker Mc, and an operating field image Ia. . Note that the control unit 111 detects that the position of the marker Mc in the operative field image Ia is out of the predetermined region by a method other than displaying the image on the display surface 110A (for example, by sound, or by turning on a patrol lamp). You can let people know.

On the other hand, when the position of the marker Mc is within the predetermined area in the operating field image Ia, the control unit 111 displays an image indicating that the position of the marker Mc is within the predetermined area in the operating field image Ia. A video signal for display is generated and output to the display unit 113 . For example, based on the video signal from the control unit 111, the display unit 113 displays information indicating that the position of the marker Mc is within a predetermined region in the operative field image Ia, the marker Mc, and the operative field image. An image including Ia is displayed on the display surface 110A. For example, as shown in FIG. 11, the display unit 113 displays an image including a status ST such as "microscope position OK", a marker Mc, and an operating field image Ia on the display surface 110A. Note that the control unit 111 notifies the medical staff that the position of the marker Mc is within a predetermined region in the surgical field image Ia by a method other than displaying the image on the display surface 110A (for example, sound, lighting of a patrol lamp, etc.). may inform

The control unit 111 outputs a left/right eye identification instruction to the left/right eye identification learning model 133 when the position of the marker Mc is within a predetermined area in the surgical field image Ia. When the left/right eye identification learning model 133 receives the left/right eye identification instruction, the left/right eye identification learning model 133 performs the identification process. Specifically, the left/right eye identification learning model 133 uses the learning result to identify the eye included in the detection image Ib. More specifically, the left/right eye identification learning model 133 uses the learning result to identify the eye type (left eye, right eye) included in the detection image Ib. The left/right eye identification learning model 133 inputs the identification result DR to the control unit 111 . The control unit 111 generates a video signal for displaying an image including the status ST, the identification result DR, the marker Mc, and the surgical field image Ia on the display surface 110A, and outputs the video signal to the display unit 113 . For example, as shown in FIG. 12, the display unit 113 displays an image including the status ST, the identification result DR, the marker Mc, and the operative field image Ia based on the video signal from the control unit 111. 110A. For example, as shown in FIG. 12, the display unit 113 may display the identification result DR superimposed on the surgical field image Ia, or, for example, as shown in FIG. It may be displayed in an area different from the wild image Ia.

By looking at the identification result DR displayed on the display surface 110A, the medical staff can determine whether the eye exposed in the opening 610 of the drape 600 is the right eye or the left eye, and whether it is the eye to be operated on. can be confirmed. In this way, left and right eye identification is performed.

Further, the control unit 111 further reads, for example, the surgery data 134C corresponding to the patient 400 from among the plurality of surgery data 134C stored in the memory 112, and the operated eye and the identification result included in the read surgery data 134C. DR is compared, and a video signal for displaying the comparison result CR on the display surface 110A is generated and output to the display unit 113. FIG. For example, as shown in FIG. 14, the display unit 113 displays an image including the comparison result CR, the identification result DR, the status ST, and the surgical field image Ia based on the video signal from the control unit 111. It is displayed on the surface 110A. Note that FIG. 14 exemplifies display when the operated eye included in the read surgery data 134C matches the identification result DR as the comparison result CR.

In this modification, the information processing device 110 has the functions of the server device 130 according to the above embodiment. Thus, by simply preparing the information processing device 110 and the camera 120 and connecting them to the network 140, it is possible to identify the left and right eyes. Therefore, the trouble of installing the left/right eye identification system 100 can be reduced by the amount that the server device 130 is not required.

[[Modification C]]
In the above embodiment, for example, as shown in FIGS. 20 and 21, communication between the information processing device 110 and the camera 120 may be performed via a network 150 different from the network 140. FIG. Further, in Modification A described above, for example, as shown in FIGS. 20 and 22 , communication between the information processing apparatus 110 and the camera 120 may be performed via a network 150 different from the network 140 . At this time, the network 150 may be, for example, short-range wireless communication such as Bluetooth (registered trademark). Even with such a configuration, the same effects as those of the above-described embodiment are obtained.

[[Modification D]]
In the above embodiments, for example, as shown in FIGS. 23 and 24, the information processing device 110 has the functions of the server device 130, and communication between the information processing device 110 and the camera 120 is performed via the network 150. may be done via At this time, the information processing apparatus 110 includes a segmentation learning model 132 and a left/right eye identification learning model 133, and stores a plurality of image data 134A, a plurality of image data 134B, and a plurality of surgical data 134C in the memory 112. The information processing apparatus 110 also has a network IF 116 that communicates with the camera 120 via the network 150 .

In this modification, when network 150 is short-range wireless communication such as Bluetooth (registered trademark), establishing communication between information processing apparatus 110 and camera 120 is easier than in the above embodiment.

<4. Second Embodiment>
[composition]
FIG. 25 shows an example of a schematic configuration of a left/right eye identification system 700 according to the second embodiment of the present invention. The left/right eye identification system 700 corresponds to a specific example of the "information processing system" of the present invention. FIG. 26 shows an example of functional blocks of the left/right eye identification system 700. As shown in FIG. FIG. 25 illustrates an ophthalmic surgical microscope system 200 for magnifying and visualizing an operative field by medical personnel, together with a left/right eye identification system 700 .

(Left/right eye identification system 700)
The left/right eye identification system 700 includes, for example, an information processing device 160 with a camera and a server device 130 . Camera-equipped information processing device 160 and server device 130 are connected via network 140 . Camera-equipped information processing device 160 is configured to be able to communicate with server device 130 via network 140 . Server device 130 is configured to be able to communicate with camera-equipped information processing device 160 via network 140 .

The camera-equipped information processing device 160 includes, for example, a control unit 111, a memory 112, an input unit 114, a network IF 115, a projection unit 117, and a camera 120. FIG. The projection unit 117 is connected to the eyepiece unit 252 and causes image light generated based on the image signal from the control unit 111 to enter the eyepiece unit 252 .

The image light generated by the projection unit 117 may be, for example, a luminous flux that generates an image including the marker Mc and the status ST as shown in FIGS. 27 and 28 . The image light generated by the projection unit 117 may be, for example, a luminous flux that generates an image including the marker Mc, the status ST, and the identification result DR as shown in FIGS. 29 and 30 . At this time, in the image light generated by the projection unit 117, for example, the identification result DR may be superimposed on the marker Mc as shown in FIG. The result DR may be placed at a different location than the marker Mc. The image light generated by the projection unit 117 may be, for example, a luminous flux that generates an image including the marker Mc, the status ST, the identification result DR, and the comparison result CR, as shown in FIG. .

The image light incident on the eyepiece 252 and the light guided from the surgical field to the observation optical system are synthesized in the eyepiece 252 and enter the eyes of the medical staff. That is, the eyepiece 252 displays the status ST, the identification result DR or the comparison result CR, and the marker Mc together with the image of the surgical field formed by the observation optical system. As a result, the medical staff visually recognizes an image as shown in FIG. 32, for example. In FIG. 32, the marker Mc is superimposed on the image of the operative field, and the status ST, the identification result DR, and the comparison result CR are displayed separately from the image of the operative field (that is, on the image of the operative field). A case where the images are displayed without being superimposed is exemplified. Note that the status ST, the identification result DR, and the comparison result CR may be displayed superimposed on the image of the surgical field.

(Left and right eye identification)
FIG. 33 is a flow chart showing an example of left/right eye identification procedure in left/right eye identification system 700 . First, the user instructs the start of left/right eye identification via the input unit 114 . The user instructs to start the program 112A via the input unit 114, for example. Then, the camera-equipped information processing apparatus 160 (control unit 111 ) reads the program 112 A from the memory 112 and loads it into the control unit 111 . The control unit 111 loaded with the program 112A executes a series of procedures for identifying left and right eyes. Specifically, first, control unit 111 outputs an imaging instruction to camera 120 . The camera 120 takes an image of the inside of the opening of the drape 600 in accordance with the imaging instruction, and outputs the resulting surgical field image Ia to the control unit 111 .

The control unit 111 performs brightness determination on the surgical field image Ia input from the camera 120 . For example, the control unit 111 converts the operative field image Ia into HSV, and if the V value of each pixel obtained by the conversion is outside a predetermined range, the operative field image Ia is insufficiently bright or too bright. The presence is displayed on the display unit 113 . For example, when the obtained V value of each pixel is within a predetermined range, the control unit 111 performs focus determination. For example, the control unit 111 performs Laplacian differentiation on the operative field image Ia, and if the resulting score is below a predetermined threshold, the display unit 113 indicates that the operative field image Ia is out of focus. to display. The information processing device 110 (control unit 111) transmits the operative field image Ia to the server device 130, for example, when the derived score is equal to or greater than a predetermined threshold.

When the server device 130 receives the surgical field image Ia from the camera-equipped information processing device 160, the Segmentation learning model 132 executes the conversion process. Specifically, the segmentation learning model 132 uses the learning result to convert the surgical field image Ia into the segmentation image Ic (see FIG. 6). The segmentation learning model 132 transmits the segmentation image Ic obtained by conversion to the information processing device 160 with camera. The camera-equipped information processing device 160 (control unit 111) performs object detection on the received segmentation image Ic. For example, as shown in FIG. 8, the camera-equipped information processing apparatus 160 (control unit 111) extracts a rectangular area including the eye area Ra from the segmentation image Ic (for example, a rectangular area in contact with the eye area Ra). cut out. An image obtained in this manner is referred to as a Detection image Id.

The camera-equipped information processing apparatus 160 (control unit 111) sets the position information of the obtained detection image Id in the operating field image Ia as the position information of the region including the eye to be identified in the operating field image Ia. The camera-equipped information processing apparatus 160 (control unit 111), for example, takes the position information of the outer edge portion of the obtained detection image Id in the operating field image Ia as the position information of the marker Mc. That is, the camera-equipped information processing apparatus 160 (control unit 111) acquires the position information of the region including the eye to be identified in the surgical field image Ia by performing object detection on the segmentation image Ic.

The camera-equipped information processing apparatus 160 (control unit 111) may use the position information of the obtained detection image Id in the operating field image Ia as the position information of the eye to be identified in the operating field image Ia. The camera-equipped information processing apparatus 160 (control unit 111) may use, for example, the position information of the outer edge portion of the eye area Ra of the obtained detection image Id in the surgical field image Ia as the position information of the marker Mc. In this case, the camera-equipped information processing apparatus 160 (control unit 111) acquires position information of the eye to be identified in the surgical field image Ia by performing object detection on the segmentation image Ic.

After generating the marker Mc based on the acquired position information, the camera-equipped information processing device 160 (control unit 111) generates a video signal for generating image light including the generated marker Mc, and outputs the signal to the projection unit 117. Output. The projection unit 117 generates image light including the marker Mc based on the image signal from the control unit 111 and emits it to the eyepiece unit 252 . As a result, the image light incident on the eyepiece 252 and the light guided from the surgical field to the observation optical system are synthesized in the eyepiece 252 and enter the eyes of the medical staff. As a result, the medical staff visually recognizes an image in which the marker Mc is superimposed on the image of the surgical field. That is, the eyepiece 252 displays the marker Mc superimposed on the image of the surgical field formed by the observation optical system.

The left/right eye identification system 700 periodically performs a series of procedures from the imaging instruction to the marker Mc display described above. At this time, the medical staff finely adjusts the imaging area of the camera 120 while referring to the image of the visual field (operative field) seen through the eyepiece 252 . For example, every time the camera-equipped information processing apparatus 160 (control unit 111) executes the above-described series of procedures from the imaging instruction to the display of the marker Mc, the position of the marker Mc protrudes from the predetermined area in the surgical field image Ia. determine whether or not there is As a result, when the position of the marker Mc in the operating field image Ia protrudes from the predetermined area, the information processing apparatus 160 with camera (the control unit 111), for example, determines that the position of the marker Mc in the operating field image Ia is a predetermined area. A video signal is generated for generating video light including information indicating that it is protruding from the area of , and is output to the projection unit 117 . Based on the video signal from the control unit 111, the projection unit 117 generates video light including the marker Mc and information indicating that the position of the marker Mc is out of a predetermined area in the surgical field image Ia. , to the eyepiece 252 . As a result, the image light incident on the eyepiece 252 and the light guided from the surgical field to the observation optical system are synthesized in the eyepiece 252 and enter the eyes of the medical staff. As a result, the medical staff visually recognizes an image in which the image shown in FIG. 27 is superimposed on the image of the operative field, for example. Note that the camera-equipped information processing apparatus 160 (control unit 111) performs the surgical operation by a method other than displaying the marker Mc in the field of view (operative field) that can be seen through the eyepiece unit 252 (for example, sound, patrol lamp lighting, etc.). The medical staff may be informed that the position of the marker Mc is out of the predetermined area in the field image Ia.

On the other hand, when the position of the marker Mc is within the predetermined area in the operative field image Ia, the information processing apparatus 160 with camera (control unit 111) determines that the position of the marker Mc is within the predetermined area in the operative field image Ia. A video signal is generated for generating video light containing information indicating that it is inside, and is output to the projection unit 117 . Based on the video signal from the control unit 111, the projection unit 117 generates video light including the marker Mc and information indicating that the position of the marker Mc is within a predetermined area in the surgical field image Ia. and emitted to the eyepiece 252 . As a result, the image light incident on the eyepiece 252 and the light guided from the surgical field to the observation optical system are synthesized in the eyepiece 252 and enter the eyes of the medical staff. As a result, the medical staff visually recognizes an image in which the image shown in FIG. 28 is superimposed on the image of the operative field, for example. Note that the camera-equipped information processing apparatus 160 (control unit 111) performs the surgical operation by a method other than displaying the marker Mc in the field of view (operative field) that can be seen through the eyepiece unit 252 (for example, sound, patrol lamp lighting, etc.). The medical staff may be notified that the position of the marker Mc is within a predetermined area in the field image Ia.

When the position of the marker Mc is within a predetermined area in the operative field image Ia, the camera-equipped information processing device 160 (control unit 111) instructs the left and right eye identification learning model 133 in the server device 130 to: Output left and right eye identification instructions. When the left/right eye identification learning model 133 receives the left/right eye identification instruction, the left/right eye identification learning model 133 performs the identification process. Specifically, the left/right eye identification learning model 133 uses the learning result to identify the eye included in the detection image Ib. More specifically, the left/right eye identification learning model 133 uses the learning result to identify the eye type (left eye, right eye) included in the detection image Ib. The left/right eye identification learning model 133 transmits the identification result DR to the information processing device 160 with camera. Control unit 111 generates a video signal for generating video light including status ST, identification result DR, and marker Mc, and outputs the video signal to projection unit 117 . The projection unit 117 generates image light including the status ST, the identification result DR, and the marker Mc based on the image signal from the control unit 111 and emits it to the eyepiece unit 252 . As a result, the image light incident on the eyepiece 252 and the light guided from the surgical field to the observation optical system are synthesized in the eyepiece 252 and enter the eyes of the medical staff. As a result, the medical staff visually recognizes an image in which, for example, the image shown in FIG. 29 or 30 is superimposed on the image of the operative field.

By looking at the identification result DR in the image of the visual field (operative field) seen through the eyepiece 252, the medical staff can determine whether the eye exposed in the opening 610 of the drape 600 is the right eye or the left eye. As a result, it is possible to confirm whether or not the eye is an eye to be operated on. In this way, left and right eye identification is performed.

The camera-equipped information processing device 160 (the control unit 111) further reads, for example, the surgical data 134C corresponding to the patient 400 from among the plurality of surgical data 134C stored in the memory 134, and stores the read surgical data 134C. The included eye to be operated is compared with the identification result DR. Camera-equipped information processing device 160 (control unit 111) generates a video signal for generating video light including the comparison result CR, status ST, identification result DR, and marker Mc, and sends it to projection unit 117. Output. The projection unit 117 generates image light including the comparison result CR, the status ST, the identification result DR, and the marker Mc based on the image signal from the control unit 111 and emits it to the eyepiece unit 252 . As a result, the image light incident on the eyepiece 252 and the light guided from the surgical field to the observation optical system are synthesized in the eyepiece 252 and enter the eyes of the medical staff. As a result, for example, as shown in FIG. 32, the medical staff can view the comparison result CR, the identification result DR, the status ST, and the marker Mc through the eyepiece 252 in the visual field (operative field). visible in the image of

[effect]
Next, the effect of left/right eye identification system 700 according to the present embodiment will be described.

In the present embodiment, by performing object detection on the segmentation image Ic obtained from the operative field image Ia including the eye to be identified, the eye to be identified or the area including the eye to be identified is detected. , the position information in the operating field image Ia is acquired. Furthermore, in the present embodiment, the marker Mc generated based on the acquired position information is superimposed and displayed on the image of the surgical field formed by the observation optical system. This allows the marker Mc to reliably include an element (for example, the lacrimal gland 12 and the inner corner of the eye) that facilitates identification of the left and right eyes. As a result, the medical staff adjusts the imaging region of the camera 120 so that the marker Mc is positioned within a predetermined region in the image of the operative field formed by the observation optical system, thereby capturing the image in the operative field image Ia. The left/right eye identification system 100 can correctly identify whether the eye being viewed is the left eye or the right eye. Therefore, it is possible to identify eye types that are difficult to identify.

Further, in the present embodiment, the identification result DR is superimposed and displayed on the image of the surgical field formed by the observation optical system in the microscope for medical staff 250 . Further, in the present embodiment, in microscope 250 for medical staff, together with marker Mc, identification result DR is displayed superimposed on the image of the surgical field formed by the observation optical system. As a result, it is possible to perform reliable confirmation work without overlooking.

Further, in the present embodiment, by performing object detection on the segmentation image Ic, after the detection image Id is clipped from the segmentation image Ic, the position information of the clipped detection image Id in the surgical field image Ia is , position information of the eye to be identified, or a region including the eye to be identified, in the surgical field image Ia. As a result, compared to the case where object detection is simply performed on the surgical field image Ia, and the position information of the eye to be identified in the surgical field image Ia is derived based on the resulting detection image. , the marker Mc can certainly include an element (for example, the lacrimal gland 12 or the inner corner of the eye) that facilitates identification of the left and right eyes. As a result, the medical staff adjusts the imaging region of the camera 120 so that the marker Mc is positioned within a predetermined region in the image of the operative field formed by the observation optical system, thereby capturing the image in the operative field image Ia. The left/right eye identification system 100 can correctly identify whether the eye being viewed is the left eye or the right eye. Therefore, it is possible to identify eye types that are difficult to identify.

Further, in the present embodiment, when the marker Mc protrudes from the predetermined region in the image of the surgical field formed by the observation optical system, the marker Mc and the marker Mc protrude from the predetermined region. (for example, status ST) is superimposed on the image of the operative field formed by the observation optical system. Accordingly, it is possible to reliably notify the medical staff that fine adjustment of the imaging area of the camera 120 is required.

Further, in the present embodiment, after the surgical field image Ia obtained when the marker Mc is within a predetermined region in the surgical field image Ia is transmitted to the server device 130, the left and right eye identification results are sent to the server device. A network IF 115 that receives data from 130 is provided in the camera-equipped information processing apparatus 160 . As a result, a device (server device 130) other than the camera-equipped information processing device 160 can handle heavy-load processing performed by the segmentation learning model 132, the left/right eye identification learning model 133, and the like. As a result, for example, each camera-equipped information processing device 160 installed in each operating room can share one server device 130. Therefore, compared to the case where the right and left eye identification system 700 is installed in each operating room, The installation cost of the eye identification system 700 can be reduced.

<5. Modification of Second Embodiment>
Next, a modification of the left/right eye identification system 700 according to the second embodiment will be described.

[[Modification E]]
FIG. 34 shows a modification of the functional blocks of the left/right eye identification system 700 according to the second embodiment. The camera-equipped information processing apparatus 160 according to this modified example stores a program 112B in the memory 112 instead of the program 112A. The server device 130 according to this modification further stores a program 134D in the memory 134. FIG.

The program 112B causes the control unit 111 to execute a procedure for causing the camera-equipped information processing device 160 to function as a user interface in the left/right eye identification system 100 . The program 134D causes the controller 131 to execute a series of procedures for identifying the type of eye (left eye or right eye) included in the operative field.

(Left and right eye identification)
FIG. 35 is a flow chart showing an example of the left/right eye identification procedure in the left/right eye identification system 700 according to this modification. First, the user instructs the start of left/right eye identification via the input unit 114 . The user instructs the activation of the programs 112B and 134D via the input unit 114, for example. Then, the camera-equipped information processing apparatus 160 (control unit 111 ) reads the program 112 B from the memory 112 and loads it into the control unit 111 . The control unit 111 loaded with the program 112B executes a procedure for causing the camera-equipped information processing device 160 to function as a user interface in the left/right eye identification system 700 . Server device 130 (control unit 131 ) reads program 134 D from memory 134 and loads it into control unit 131 . The control unit 131 loaded with the program 134D executes a series of procedures for identifying left and right eyes.

Specifically, the camera-equipped information processing apparatus 160 (control unit 111 ) first outputs an imaging instruction to the camera 120 . The camera 120 takes an image of the inside of the opening of the drape 600 in accordance with the imaging instruction, and outputs the resulting surgical field image Ia to the control unit 111 .

The camera-equipped information processing device 160 (control unit 111 ) transmits the operative field image Ia input from the camera 120 to the server device 130 . The server device 130 (control unit 131) performs brightness determination on the received surgical field image Ia. The server device 130 (control unit 131), for example, derives the average value of the brightness of each pixel of the surgical field image Ia, and if the derived average value is below a predetermined threshold, the surgical field image Ia is A signal is transmitted to the camera-equipped information processing apparatus 160 to cause the display unit 113 to display the shortage. The camera-equipped information processing device 160 displays on the display unit 113 that the surgical field image Ia is insufficient in brightness based on the signal from the server device 130 .

The server device 130 (control unit 131) performs focus determination, for example, when the derived average value is within a predetermined range. The server device 130 (control unit 131) performs, for example, Laplacian differentiation on the operative field image Ia, and if the resulting score is below a predetermined threshold, the operative field image Ia is out of focus. A signal for displaying the fact on the display unit 113 is transmitted to the camera-equipped information processing apparatus 160 . The camera-equipped information processing apparatus 160 displays on the display unit 113 that the surgical field image Ia is out of focus based on the signal from the server apparatus 130 . The server device 130 (control unit 131) inputs the surgical field image Ia to the segmentation learning model 132, for example, when the derived score is equal to or greater than a predetermined threshold.

The Segmentation learning model 132 executes a conversion process when the surgical field image Ia is input. Specifically, the segmentation learning model 132 uses the learning result to convert the surgical field image Ia into the segmentation image Ic (see FIG. 6). The segmentation learning model 132 inputs the segmentation image Ic obtained by conversion to the control unit 131 . The control unit 131 performs object detection on the input segmentation image Ic. For example, as shown in FIG. 10, the control unit 131 cuts out a rectangular region including the eye region Ra (for example, a rectangular region in contact with the eye region Ra) from the segmentation image Ic. An image obtained in this manner is referred to as a Detection image Id.

The server device 130 (control unit 131) sets the position information of the obtained detection image Id in the operating field image Ia as the position information of the region including the eye to be identified in the operating field image Ia. The server device 130 (control unit 131) uses, for example, the position information of the outer edge portion of the obtained detection image Id in the surgical field image Ia as the position information of the marker Mc. In other words, the server device 130 (control unit 131) acquires the position information of the region including the eye to be identified in the surgical field image Ia by performing object detection on the segmentation image Ic.

The server device 130 (control unit 131) may use the position information of the obtained detection image Id in the operating field image Ia as the position information of the eye to be identified in the operating field image Ia. The server device 130 (control unit 131) may use, for example, the position information of the outer edge portion of the eye area Ra of the obtained detection image Id in the operating field image Ia as the position information of the marker Mc. In this case, the server device 130 (control unit 131) acquires the position information of the eye to be identified in the surgical field image Ia by performing object detection on the segmentation image Ic.

The server device 130 (control unit 131) transmits the acquired position information to the information processing device 160 with camera. After generating the marker Mc based on the position information from the server device 130 , the camera-equipped information processing device 160 generates a video signal for generating video light including the generated marker Mc, and outputs the video signal to the projection unit 117 . do. The projection unit 117 generates image light including the marker Mc based on the image signal from the control unit 111 and emits it to the eyepiece unit 252 . As a result, the image light incident on the eyepiece 252 and the light guided from the surgical field to the observation optical system are synthesized in the eyepiece 252 and enter the eyes of the medical staff. As a result, the medical staff visually recognizes an image in which the marker Mc is superimposed on the image of the operative field, for example. At this time, the position and size of the marker Mc are defined by the video signal incident on the projection section 117 from the control section 111 . That is, the eyepiece 252 displays the marker Mc superimposed on the image of the surgical field formed by the observation optical system.

The left/right eye identification system 700 periodically performs a series of procedures from the imaging instruction to the marker Mc display described above. At this time, the medical staff finely adjusts the imaging area of the camera 120 while referring to the image of the visual field (operative field) seen through the eyepiece 252 . For example, every time the server device 130 (control unit 131) executes the above-described series of procedures from the imaging instruction to the display of the marker Mc, whether or not the position of the marker Mc protrudes from the predetermined region in the surgical field image Ia. judge. As a result, when the position of the marker Mc protrudes from the predetermined area in the operating field image Ia, the server device 130 (control unit 131), for example, Information indicating protruding is transmitted to the camera-equipped information processing apparatus 160 . Camera-equipped information processing device 160 (control unit 111 ) generates a video signal for displaying an image including the received information, and outputs the video signal to display unit 113 . For example, based on the video signal from the control unit 111, the display unit 113 displays video light including the marker Mc and information indicating that the position of the marker Mc is out of a predetermined area in the surgical field image Ia. It is generated and emitted to the eyepiece 252 . As a result, the image light incident on the eyepiece 252 and the light guided from the surgical field to the observation optical system are synthesized in the eyepiece 252 and enter the eyes of the medical staff. As a result, the medical staff visually recognizes an image in which, for example, the image shown in FIG. 27 is superimposed on the image of the surgical field. Note that the camera-equipped information processing apparatus 160 (control unit 111) performs the surgical operation by a method other than displaying the marker Mc in the field of view (operative field) that can be seen through the eyepiece unit 252 (for example, sound, patrol lamp lighting, etc.). The medical staff may be informed that the position of the marker Mc is out of the predetermined area in the field image Ia.

On the other hand, when the position of the marker Mc is within the predetermined area in the operative field image Ia, the server apparatus 130 (control unit 131) determines that the position of the marker Mc is within the predetermined area in the operative field image Ia. Information indicating that is transmitted to the information processing apparatus 160 with a camera. Camera-equipped information processing device 160 (control unit 111 ) generates a video signal for generating video light including the received information, and outputs the video signal to projection unit 117 . Based on the video signal from the control unit 111, the projection unit 117 generates video light including the marker Mc and information indicating that the position of the marker Mc is within a predetermined area in the surgical field image Ia. and emitted to the eyepiece 252 . As a result, the image light incident on the eyepiece 252 and the light guided from the surgical field to the observation optical system are synthesized in the eyepiece 252 and enter the eyes of the medical staff. As a result, the medical staff visually recognizes an image in which, for example, the image shown in FIG. 28 is superimposed on the image of the surgical field. Note that the camera-equipped information processing apparatus 160 (control unit 111) performs the surgical operation by a method other than displaying the marker Mc in the field of view (operative field) that can be seen through the eyepiece unit 252 (for example, sound, patrol lamp lighting, etc.). The medical staff may be notified that the position of the marker Mc is within a predetermined area in the field image Ia.

The server device 130 (control unit 131) outputs a left/right eye identification instruction to the left/right eye identification learning model 133 when the position of the marker Mc is within a predetermined area in the surgical field image Ia. When the left/right eye identification learning model 133 receives the left/right eye identification instruction, the left/right eye identification learning model 133 performs the identification process. Specifically, the left/right eye identification learning model 133 uses the learning result to identify the eye included in the detection image Ib. More specifically, the left/right eye identification learning model 133 uses the learning result to identify the eye type (left eye, right eye) included in the detection image Ib. The left/right eye identification learning model 133 transmits the identification result DR to the control unit 131 . The server device 130 (control unit 131) transmits the identification result DR to the information processing device 160 with camera. Camera-equipped information processing device 160 (control unit 111 ) generates a video signal for generating video light including status ST, identification result DR, and marker Mc, and outputs the video signal to projection unit 117 . The projection unit 117 generates image light including the status ST, the identification result DR, and the marker Mc based on the image signal from the control unit 111 and emits it to the eyepiece unit 252 . As a result, the image light incident on the eyepiece 252 and the light guided from the surgical field to the observation optical system are synthesized in the eyepiece 252 and enter the eyes of the medical staff. As a result, the medical staff visually recognizes an image in which, for example, the image shown in FIG. 29 or 30 is superimposed on the image of the surgical field.

By looking at the identification result DR in the image of the visual field (operative field) seen through the eyepiece 252, the medical staff can determine whether the eye exposed in the opening 610 of the drape 600 is the right eye or the left eye. As a result, it is possible to confirm whether or not the eye is an eye to be operated on. In this way, left and right eye identification is performed.

Server device 130 (control unit 131) further reads, for example, surgical data 134C corresponding to patient 400 from among a plurality of surgical data 134C stored in memory 134, and surgical data 134C included in read surgical data 134C. The eye is compared with the identification result DR, and the comparison result CR is transmitted to the camera-equipped information processing device 160 . Camera-equipped information processing device 160 (control unit 111) generates a video signal for generating video light including status ST, comparison result CR, identification result DR, and marker Mc, and outputs the video signal to projection unit 117. do. The projection unit 117 generates image light including the status ST, the comparison result CR, the identification result DR, and the marker Mc based on the image signal from the control unit 111 and emits it to the eyepiece unit 252 . As a result, the image light incident on the eyepiece 252 and the light guided from the surgical field to the observation optical system are synthesized in the eyepiece 252 and enter the eyes of the medical staff. As a result, the medical staff visually recognizes an image in which the image shown in FIG. 31 is superimposed on the image of the operative field, for example.

In this modification, in the server device 130, object detection is performed on the segmentation image Ic obtained from the operative field image Ia including the eye to be identified. is acquired in the surgical field image Ia. Furthermore, in this modified example, the acquired position information is transmitted to the information processing apparatus 160 with a camera. As a result, in the camera-equipped information processing apparatus 160, the marker Mc based on the positional information can be superimposed and displayed on the image of the surgical field. For example, the lacrimal mound 12 and inner corner of the eye) can be reliably included. As a result, the medical staff adjusts the imaging region of the camera 120 so that the marker Mc is positioned within a predetermined region in the image of the operative field formed by the observation optical system, thereby capturing the image in the operative field image Ia. The left/right eye identification system 100 can correctly identify whether the eye being viewed is the left eye or the right eye. Therefore, it is possible to identify eye types that are difficult to identify.

Further, in this modification, by performing object detection on the segmentation image Ic, after the detection image Id is clipped from the segmentation image Ic, the position information of the clipped detection image Id in the surgical field image Ia is: This is the positional information of the eye to be identified or the region containing the eye to be identified in the surgical field image Ia. As a result, compared to the case where object detection is simply performed on the surgical field image Ia, and the position information of the eye to be identified in the surgical field image Ia is derived based on the resulting detection image. , the marker Mc can certainly include an element (for example, the lacrimal gland 12 or the inner corner of the eye) that facilitates identification of the left and right eyes. As a result, the medical staff adjusts the imaging area of the camera 120 so that the marker Mc is positioned within a predetermined area in the operative field image Ia, so that the eye captured in the operative field image Ia is the left eye. or the right eye can be correctly identified in the left/right eye identification system 100 . Therefore, it is possible to identify eye types that are difficult to identify.

[[Modification F]]
In the second embodiment, for example, as shown in FIG. 36, the server device 130 may be omitted and the camera-equipped information processing device 160 may have the functions of the server device 130 . At this time, the camera-equipped information processing apparatus 160, for example, as shown in FIG. Operation data 134C is stored in memory 112. FIG.

(Left and right eye identification)
FIG. 38 is a flow chart showing an example of the left/right eye identification procedure in the left/right eye identification system 700 according to this modification. First, the user instructs the start of left/right eye identification via the input unit 114 . The user instructs to start the program 112A via the input unit 114, for example. Then, the camera-equipped information processing apparatus 160 (control unit 111 ) reads the program 112 A from the memory 112 and loads it into the control unit 111 . The control unit 111 loaded with the program 112A executes a series of procedures for identifying left and right eyes.

Specifically, first, control unit 111 outputs an imaging instruction to camera 120 . The camera 120 takes an image of the inside of the opening of the drape 600 in accordance with the imaging instruction, and transmits the resulting surgical field image Ia to the control unit 111 .

The control unit 111 performs brightness determination on the received surgical field image Ia. For example, the control unit 111 derives the average value of the brightness of each pixel of the surgical field image Ia, and if the derived average value is below a predetermined threshold, it indicates that the surgical field image Ia is insufficient in brightness. Displayed on the display unit 113 .

For example, when the derived average value is equal to or greater than a predetermined threshold value, the control unit 111 performs focus determination. For example, the control unit 111 performs Laplacian differentiation on the operative field image Ia, and if the resulting score is below a predetermined threshold, the display unit 113 indicates that the operative field image Ia is out of focus. to display. For example, when the derived score is equal to or greater than a predetermined threshold, the control unit 111 inputs the operative field image Ia to the segmentation learning model 132 .

The Segmentation learning model 132 executes a conversion process when the surgical field image Ia is input. Specifically, the segmentation learning model 132 uses the learning result to convert the surgical field image Ia into the segmentation image Ic (see FIG. 6). The segmentation learning model 132 inputs the segmentation image Ic obtained by conversion to the control unit 131 . The control unit 111 performs object detection on the input segmentation image Ic. For example, as shown in FIG. 8, the control unit 111 cuts out a rectangular region including the eye region Ra (for example, a rectangular region in contact with the eye region Ra) from the segmentation image Ic. An image obtained in this manner is referred to as a Detection image Id.

The control unit 111 uses the position information of the obtained detection image Id in the operating field image Ia as the position information of the region including the eye to be identified in the operating field image Ia. The control unit 111 uses, for example, the position information of the outer edge portion of the obtained detection image Id in the surgical field image Ia as the position information of the marker Mc. That is, the control unit 111 acquires the position information of the region including the eye to be identified in the surgical field image Ia by performing object detection on the segmentation image Ic.

Note that the control unit 111 may use the position information of the obtained detection image Id in the operating field image Ia as the position information of the eye to be identified in the operating field image Ia. For example, the control unit 111 may use the position information of the outer edge portion of the eye area Ra of the obtained detection image Id in the surgical field image Ia as the position information of the marker Mc. In this case, the control unit 111 acquires the position information of the eye to be identified in the surgical field image Ia by performing object detection on the segmentation image Ic.

After generating the marker Mc based on the acquired position information, the control unit 111 generates a video signal for generating image light including the generated marker Mc, and outputs the video signal to the projection unit 117 . The projection unit 117 generates image light including the marker Mc based on the image signal from the control unit 111 and emits it to the eyepiece unit 252 . As a result, the image light incident on the eyepiece 252 and the light guided from the surgical field to the observation optical system are synthesized in the eyepiece 252 and enter the eyes of the medical staff. As a result, the medical staff visually recognizes an image in which the marker Mc is superimposed on the image of the operative field, for example. At this time, the position and size of the marker Mc are defined by the video signal incident on the projection section 117 from the control section 111 . That is, the eyepiece 252 displays the marker Mc superimposed on the image of the surgical field formed by the observation optical system.

The left/right eye identification system 100 periodically performs a series of procedures from the imaging instruction to the marker Mc display described above. At this time, the medical staff finely adjusts the imaging area of the camera 120 while referring to the image of the visual field (operative field) seen through the eyepiece 252 . For example, the control unit 111 determines whether or not the position of the marker Mc protrudes from a predetermined area in the surgical field image Ia each time the above-described series of procedures from the imaging instruction to the marker Mc display are executed. As a result, when the position of the marker Mc protrudes from the predetermined region in the surgical field image Ia, the control unit 111 detects that the position of the marker Mc protrudes from the predetermined region in the surgical field image Ia, for example. Image light including meaningful information is generated and emitted to the eyepiece 252 . As a result, the image light incident on the eyepiece 252 and the light guided from the surgical field to the observation optical system are synthesized in the eyepiece 252 and enter the eyes of the medical staff. As a result, the medical staff visually recognizes an image in which the image shown in FIG. 27 is superimposed on the image of the operative field, for example. Note that the control unit 111 can display the position of the marker Mc in the surgical field image Ia by a method other than displaying the marker Mc in the field of view (operative field) visible through the eyepiece unit 252 (for example, sound, patrol lamp lighting, etc.). is out of the defined area.

On the other hand, when the position of the marker Mc is within the predetermined area in the operative field image Ia, the control unit 111 outputs information indicating that the position of the marker Mc is within the predetermined area in the operative field image Ia. A video signal for generating video light including the light is generated and output to the projection unit 117 . Based on the video signal from the control unit 111, the projection unit 117 generates video light including the marker Mc and information indicating that the position of the marker Mc is within a predetermined area in the surgical field image Ia. and emitted to the eyepiece 252 . As a result, the image light incident on the eyepiece 252 and the light guided from the surgical field to the observation optical system are synthesized in the eyepiece 252 and enter the eyes of the medical staff. As a result, the medical staff visually recognizes an image in which, for example, the image shown in FIG. 28 is superimposed on the image of the surgical field. Note that the camera-equipped information processing apparatus 160 (control unit 111) performs the surgical operation by a method other than displaying the marker Mc in the field of view (operative field) that can be seen through the eyepiece unit 252 (for example, sound, patrol lamp lighting, etc.). The medical staff may be notified that the position of the marker Mc is within a predetermined area in the field image Ia.

The control unit 111 outputs a left/right eye identification instruction to the left/right eye identification learning model 133 when the position of the marker Mc is within a predetermined area in the surgical field image Ia. When the left/right eye identification learning model 133 receives the left/right eye identification instruction, the left/right eye identification learning model 133 performs the identification process. Specifically, the left/right eye identification learning model 133 uses the learning result to identify the eye included in the detection image Ib. More specifically, the left/right eye identification learning model 133 uses the learning result to identify the eye type (left eye, right eye) included in the detection image Ib. The left/right eye identification learning model 133 inputs the identification result DR to the control unit 111 . The control unit 111 generates a video signal for generating video light including the status ST, the identification result DR, and the marker Mc, and outputs the video signal to the projection unit 117 . The projection unit 117 generates image light including the status ST, the identification result DR, and the marker Mc based on the image signal from the control unit 111 and emits it to the eyepiece unit 252 . As a result, the image light incident on the eyepiece 252 and the light guided from the surgical field to the observation optical system are synthesized in the eyepiece 252 and enter the eyes of the medical staff. As a result, the medical staff visually recognizes an image in which, for example, the image shown in FIG. 29 or 30 is superimposed on the image of the operative field.

By looking at the identification result DR in the image of the visual field (operative field) seen through the eyepiece 252, the medical staff can determine whether the eye exposed in the opening 610 of the drape 600 is the right eye or the left eye. As a result, it is possible to confirm whether or not the eye is an eye to be operated on. In this way, left and right eye identification is performed.

Further, the control unit 111 further reads, for example, the surgery data 134C corresponding to the patient 400 from among the plurality of surgery data 134C stored in the memory 112, and the operated eye and the identification result included in the read surgery data 134C. Compare with DR. Control unit 111 generates a video signal for generating video light including status ST, comparison result CR, identification result DR, and marker Mc, and outputs the video signal to projection unit 117 . The projection unit 117 generates image light including the status ST, the comparison result CR, the identification result DR, and the marker Mc based on the image signal from the control unit 111 and emits it to the eyepiece unit 252 . As a result, the image light incident on the eyepiece 252 and the light guided from the surgical field to the observation optical system are synthesized in the eyepiece 252 and enter the eyes of the medical staff. As a result, the medical staff visually recognizes an image in which the image shown in FIG. 31 is superimposed on the image of the operative field, for example.

In this modification, an information processing device 160 with a camera has the functions of the server device 130 according to the second embodiment. As a result, it is possible to identify the left and right eyes simply by preparing the information processing device 160 with a camera. Therefore, the trouble of installing the left/right eye identification system 700 can be reduced by the amount that the server device 130 is not required.

Although the present invention has been described above with reference to the embodiments and modifications thereof, the present invention is not limited to the embodiments and the like, and various modifications are possible. Although the present invention has been applied to eye surgery in the embodiments and the like, it can be suitably applied to fields where it is necessary to identify difficult-to-identify objects. It should be noted that the effects described in this specification are merely examples. The effects of the present invention are not limited to the effects described herein. The present invention may have advantages other than those described herein.

DESCRIPTION OF SYMBOLS 10... Eye, 11... Eyeball, 12... Lacrimal dune, 100... Right-and-left eye identification system, 110... Information processing apparatus, 110A... Display surface, 111... Control part, 112... Memory, 112A, 112B... Program, 113... Display part , 114... input unit, 115, 116... network IF, 117... projection unit, 120... camera, 130... server device, 131... control unit, 132... segmentation learning model, 133... right and left eye identification learning model, 134... memory, 134A, 134B... Image data 134C... Surgery data 134D... Program 135... Network IF 140, 150... Network 160... Information processing device with camera 200... Microscope system for ophthalmic surgery 210... Prop 220... Third DESCRIPTION OF SYMBOLS 1 arm, 230... 2nd arm, 240... Driving device, 250... Microscope for medical staff, 251... Lens tube part, 252... Eyepiece part, 260... Foot switch, 300... Operating table, 400... Patient, 500... Sheet, 600... Drape, 610... Opening, 700... Right and left eye identification system, CR... Comparison result, DR... Identification result, Ia... Operating field image, Ib... Detection image, Ic... Segmentation image, Id... Detection image, Mc... Marker, Ra... Eye area, Rb... Peripheral area, ST... Status.

Claims (4)

By detecting the eye region in the segmentation image obtained from the captured image including the eye region including the eyeball and the lacrimal gland , the position information of the region including the eye region in the captured image is acquired and acquired. an acquisition unit that uses the obtained position information as position information of a marker ;
a display unit that displays a composite image in which a marker generated based on the position information acquired by the acquisition unit is superimposed on the captured image so as to surround the eye region ,
When the marker protrudes from the captured image, the display unit further displays an image including information indicating that the marker protrudes from the captured image as the synthesized image.
Information processing system. By detecting the eye region in the segmentation image obtained from the captured image including the eye region including the eyeball and the lacrimal gland , the position information of the region including the eye region in the captured image is acquired and acquired. an acquisition unit that uses the obtained position information as position information of a marker ;
a synthesizing unit that superimposes a marker generated based on the position information acquired by the acquisition unit on an image of a field of view including the eye region formed by an optical system so as to surround the eye region ;
When the marker protrudes from the captured image, the synthesizing unit generates an image including the marker, information indicating that the marker protrudes from the captured image, and an image of the field of view. do
Information processing system. a communication unit that transmits the captured image obtained when the marker is within the captured image to an external device, and receives from the external device an identification result of the eye region included in the captured image; The information processing system according to claim 1 or 2 , further comprising: 2. The apparatus according to claim 1 , further comprising an identification unit that identifies whether the eye in the eye region included in the captured image obtained when the marker is within the captured image is a right eye or a left eye. Item 3. The information processing system according to item 2 .

Images