JP2017102847A - Information processing system, relay device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology of preventing undesired GUI-based operation on a server from a client terminal, without installing a monitoring agent on the server.SOLUTION: An information processing system includes: a screen acquisition unit 31 which acquires screen information indicating a predetermined area of a screen to be displayed as a remote desktop, on the basis of remote desktop information received from a server 10; a remote desktop transmission unit 32 for transmitting the remote desktop information to a client terminal 20; an operation information receiving unit 33 which receives operation information indicating operation on the remote desktop on the client terminal 20, from the client terminal 20; an operation monitoring unit 34 which determines whether the operation information is to be detected or not, on the basis of whether an operation position based on the operation information is included in an area similar to an image to be detected, in the screen represented by the screen information; and an operation information transmission unit 35 which transmits the operation information determined not to be detected, to the server 10.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for monitoring operations from a client terminal to a server.

  There is a desire to monitor the operation of the server from the client terminal. In response to such a demand, it is well known that a monitoring agent is mounted on a server to detect an undesirable operation. Examples of the unfavorable operation include an unauthorized operation, an erroneous operation, or an operation that should be avoided from an operational viewpoint. However, there are cases where the monitoring agent cannot be mounted on the server, such as when it is not allowed to apply an extra load on the server.

  In order to cope with such a case, generally, a method of monitoring an operation from a client terminal to a server using a relay device is known. In a general method, the relay apparatus detects that a character string input at a client terminal, a title of a window activated on a server, or the like is a registered character string. In this case, it is possible to prevent an undesired operation by registering in advance an input character string or window title that may cause an undesired operation.

  Patent Document 1 describes an example of related technology for managing access and operation from a client terminal to a server using a relay device. In this related technology, the relay device executes the first user authentication process by using the first identification information given to the user of the client terminal that requests access to the server. In addition, the relay device requests the server for the second user authentication process using the second identification information associated with the first identification information. Then, when the first and second user authentication processes are successful, the relay device records an access log and an operation log for the server from the client terminal while relaying between the client terminal and the server.

International Publication No. 2015/004743

  In recent years, servers that enable remote desktop connection are well known. However, using the general method described above or the related technology described in Patent Document 1 to monitor the operation of such a server from a client terminal has the following problems.

  A general method detects an undesired operation on a character string basis, such as an input character string on a client terminal or a window title activated on a server. For this reason, it is difficult to prevent a GUI (graphical user interface) based undesired operation with a general method. Here, the GUI-based operation is an operation using the GUI.

  For example, consider a case where it is desired to monitor an operation from a client terminal related to service management in a server. For example, it is assumed that the following GUI for service management is provided by the server. That is, when each service listed in the service management window is selected by a mouse click, a start or stop menu item is displayed. When the corresponding menu item is clicked with the mouse, the application corresponding to the selected service is started or stopped. At this time, as a general technique, it is possible to detect the activation of the service management window by registering the title of the service management window in advance. However, there is a case where it is desired not to detect activation or termination of individual services, but to detect activation of the service management window itself. However, since start and stop operations for individual services are performed by mouse clicks, general techniques cannot prevent these operations.

  For example, consider a case where it is desired to prevent a client terminal from copying or pasting a file on a server. For example, assume that the following file management GUI is provided by the server. That is, when a file icon displayed on the remote desktop or an empty space in the folder window is selected by a mouse click, a copy or paste menu item is displayed. When the corresponding menu item is clicked with the mouse, copy or paste is executed for the selected file or folder. However, since a general method cannot detect a GUI-based operation, it cannot prevent a file copy or paste operation by a mouse click.

  Patent Document 1 describes that a relay device records an operation log from a client terminal for a server, taking a file sharing server that provides a GUI as an example. However, the operation log in this related technology associates the operation contents such as “Create file” and “Copy and paste file” with the process contents such as “Created file name” and “Copy and paste file name”. To record. In other words, this related technique is considered to record a result of processing performed by the server according to GUI-based operation from the client terminal as an operation log. Therefore, with this related technology, it is difficult to detect and prevent an unfavorable operation before a process corresponding to a GUI-based operation is executed on the server.

  The present invention has been made to solve the above-described problems. That is, an object of the present invention is to provide a technique for preventing an undesirable GUI-based operation from a client terminal to a server without mounting a monitoring agent on the server.

  The relay device of the present invention receives information representing a remote desktop provided to a client terminal from a server, and based on the received information, is a screen that is image data representing a predetermined area of a screen that can be displayed as the remote desktop A screen acquisition unit for acquiring information, a remote desktop transmission unit for transmitting information representing the remote desktop to the client terminal, and operation information representing an operation performed on the display area of the remote desktop screen in the client terminal Whether the operation position based on the operation information is included in an area that satisfies the similarity condition for the image to be detected in the operation information receiving unit that receives from the client terminal and the screen represented by the screen information To determine whether or not the operation information is a detection target Comprising an operation monitoring section, the operation information determined to be not detection target by the operation monitoring unit, an operation information transmitting unit that transmits to the server, the.

  An information processing system according to the present invention includes the above-described relay device, the client terminal, and the server.

  Further, the method of the present invention is image data representing a predetermined area of a screen that can be displayed as the remote desktop based on the received information representing the remote desktop provided to the client terminal from the server. Screen information is acquired, information representing the remote desktop is transmitted to the client terminal, and operation information representing an operation performed on a display area of the screen of the remote desktop in the client terminal is received from the client terminal. Whether the operation information is the detection target based on whether the operation position based on the operation information is included in an area that satisfies the similarity condition with respect to the detection target image on the screen represented by the screen information. The operation information determined not to be detected is transmitted to the server.

  Further, according to another method of the present invention, the server transmits information representing the remote desktop to the client terminal via the relay device, and the relay device receives the remote desktop based on the information received from the server. Screen information, which is image data representing a predetermined area of the screen that can be displayed as, and transmits information representing the remote desktop to the client terminal, the client terminal based on the information received from the relay device , Displaying the remote desktop screen on a display device, and obtaining operation information representing an operation on the displayed display area of the remote desktop screen from the input device, the obtained operation information is addressed to the server via the relay device. The relay device transmits the operation on the screen represented by the screen information. Based on whether or not the operation position based on the information is included in an area satisfying the similarity condition with respect to the detection target image, it is determined whether or not the operation information is a detection target, and it is determined that the operation position is not a detection target The operation information is transmitted to the server, and the server executes processing based on the operation information received from the relay device.

  The program of the present invention is image data representing a predetermined area of a screen that can be displayed as the remote desktop based on the received information that represents the remote desktop provided to the client terminal from the server. A screen acquisition step for acquiring screen information, a remote desktop transmission step for transmitting information representing the remote desktop to the client terminal, and an operation representing an operation performed on the display area of the remote desktop screen in the client terminal In the operation information receiving step for receiving information from the client terminal and the screen represented by the screen information, whether or not the operation position based on the operation information is included in a region that satisfies the similarity condition with respect to the image to be detected The operation information is And operation monitoring step of determining whether a subject, the operation information determined not to be the detection target in the operation monitoring step, said executing the operation information transmitting step of transmitting to the server, to the computer device.

  The present invention can provide a technique for preventing an undesirable GUI-based operation from a client terminal for a server without mounting a monitoring agent on the server.

It is a block diagram which shows the structure of the information processing system as embodiment of this invention. It is a figure which shows an example of the hardware constitutions of the information processing system as embodiment of this invention. It is a flowchart explaining operation | movement of the information processing system as embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows a functional block configuration of an information processing system 1 as an embodiment of the present invention. In FIG. 1, the information processing system 1 includes a server 10, a client terminal 20, and a relay device 30. The relay device 30 is communicably connected to the server 10 and the client terminal 20 via a network. 1 shows one server 10, one client terminal 20, and one relay device 30, but the number of devices in the present embodiment is not limited.

  The server 10 is a device that provides a remote desktop connection service.

  The client terminal 20 is a device that uses the server 10 via a remote desktop connection. The client terminal 20 may be a personal computer or a mobile terminal, for example.

  The relay device 30 is a device that relays communication between the server 10 and the client terminal 20. Further, the relay device 30 includes a screen acquisition unit 31, a remote desktop transmission unit 32, an operation information reception unit 33, an operation monitoring unit 34, and an operation information transmission unit 35.

  Here, the information processing system 1 can be configured by hardware elements as shown in FIG. In FIG. 2, the server 10 includes a CPU (Central Processing Unit) 1001, a memory 1002, and a network interface 1005. The CPU 1001 reads and executes a computer program stored in the memory 1002. The memory 1002 includes a RAM (Random Access Memory), a ROM (Read Only Memory), an auxiliary storage device (such as a hard disk), and the like. The network interface 1005 is an interface connected to a network that can communicate with the relay device 30.

  2, the client terminal 20 includes a CPU 2001, a memory 2002, a display device 2003, an input device 2004, and a network interface 2005. The CPU 2001 reads and executes a computer program stored in the memory 2002. The memory 2002 includes a RAM, a ROM, an auxiliary storage device, and the like. The display device 2003 is configured by a device that displays information, such as a liquid crystal display. The input device 2004 may include a pointing device such as a mouse or a touch panel, a keyboard, or operation buttons. A network interface 2005 is an interface connected to a network that can communicate with the relay device 30.

  In FIG. 2, the relay device 30 includes a CPU 3001, a memory 3002, and a network interface 3005. The CPU 3001 reads and executes a computer program stored in the memory 3002. The memory 3002 includes a RAM, a ROM, an auxiliary storage device, and the like. The network interface 3005 is an interface connected to a network that can communicate with the server 10 and the client terminal 20. In this case, the functional blocks of the screen acquisition unit 31, the remote desktop transmission unit 32, the operation information reception unit 33, and the operation information transmission unit 35 are configured as follows. That is, each of these functional blocks includes a network interface 3005 and a CPU 3001 that reads and executes a computer program stored in the memory 3002. The operation monitoring unit 34 includes a CPU 3001 that reads and executes a computer program stored in the memory 3002.

  Note that the hardware configuration of each device and each functional block constituting the information processing system 1 is not limited to the above-described configuration.

  Next, functions of each device will be described.

  The server 10 transmits information representing the remote desktop to the client terminal 20 via the relay device 30. Here, the information representing the remote desktop includes screen information and audio information output as a result of being executed by the application in the server 10 based on the operation of the client terminal 20. Note that the screen information included in the information representing the remote desktop may be information representing a difference from the previous screen information. For example, when receiving a remote desktop connection request from the client terminal 20, the server 10 transmits information representing the remote desktop. Further, the server 10 transmits information representing the remote desktop according to the update of the configuration of the remote desktop. In addition, when the server 10 receives operation information from the client terminal 20 via the relay device 30, the server 10 performs processing according to the operation information.

  The client terminal 20 displays the remote desktop screen on the display device 2003 based on the information representing the remote desktop received from the server 10 via the relay device 30. At this time, the client terminal 20 displays the remote desktop screen by applying the display size (for example, resolution, display range, etc.) set in the own device. Then, the client terminal 20 acquires operation information representing an operation input via the input device 2004 in the display area of the remote desktop screen. Then, the client terminal 20 transmits the acquired operation information to the server 10 via the relay device 30.

  Here, when a GUI-based operation is performed, the operation information includes information indicating an operation position. The operation position represents a position designated as an operation target in the display area of the remote desktop screen. The operation position may be expressed as coordinates in a screen coordinate system applied to the display area of the remote desktop screen.

  For example, when a click operation with the mouse as the input device 2004 is performed, the operation position may be the position of the mouse pointer at the time of the click. For example, when a tap operation with a finger or a pen or the like is performed on the touch panel as the input device 2004, the operation position may be a position where contact by a tap is detected. For example, when a key or button is pressed by a keyboard or operation button as the input device 2004, the operation position may be the position of a display element selected on the screen at the time of pressing. Note that the display element selected at the time of pressing may be an element selected by a mouse click operation, a touch panel tap operation, an arrow key or a tab key of a keyboard or an operation button, etc. before the pressing operation. Good.

  Note that the number of operation positions included in the operation information is not necessarily one. For example, when the operation is dragging with a mouse, the operation information may include information representing a plurality of operation positions representing the drag start point, end point, locus, and the like.

  Further, the operation information may include information indicating the operation content. For example, when an operation with the mouse as the input device 2004 is performed, the operation content may be information representing a distinction such as left click, right click, and drag. Further, for example, when a touch operation on the touch panel as the input device 2004 is performed, the operation content may be information indicating a distinction such as a tap, a flick, and a swipe. Further, for example, when a key or button is pressed by a keyboard or operation button as the input device 2004, the operation content is information representing characters or symbols associated with the pressed key or button. Also good.

  The relay device 30 relays communication between the server 10 and the client terminal 20 and monitors operations on the remote desktop. Below, the detail of each functional block of the relay apparatus 30 is demonstrated.

  The screen acquisition unit 31 receives information representing the remote desktop from the server 10. Then, the screen acquisition unit 31 acquires screen information based on the information received from the server 10. Here, the screen information is image data representing a predetermined area of a remote desktop screen that can be displayed based on information received from the server 10. The predetermined area is preferably the same as or larger than the display area of the screen displayed as the remote desktop on the client terminal 20. For example, the predetermined area may be the entire area of the remote desktop screen that can be displayed. The screen information is used by the operation monitoring unit 34 described later.

  The remote desktop transmission unit 32 transmits information representing the remote desktop to the client terminal 20.

  The operation information receiving unit 33 receives, from the client terminal 20, operation information representing an operation performed on the remote desktop screen display area in the client terminal 20.

  The operation monitoring unit 34 analyzes whether or not the operation position based on the operation information is included in an area satisfying the similarity condition with respect to the detection target image on the screen represented by the screen information. Then, the operation monitoring unit 34 determines whether the operation information is a detection target based on the analysis result. In other words, the operation monitoring unit 34 detects an operation performed in a region that satisfies the similarity condition for the detection target image as the detection target operation.

  Here, the detection target image is an image including an operation position when the detection target operation is performed. The image to be detected may be an image stored in advance in the memory 3002. Alternatively, the detection target image may be an image input from the outside via a network, an input device, a portable storage medium, or the like.

  Further, the similarity condition may be a condition based on a degree of similarity representing a degree of similarity between two images. For example, when a similarity indicating a larger value as two images are similar is employed, the similarity condition may be that the similarity is equal to or greater than a threshold value. For calculating the similarity, a known image similarity calculation algorithm for calculating the similarity between two images may be employed. Hereinafter, a region that satisfies the similarity condition for the detection target image is also referred to as a detection target similar region.

  Specifically, the operation monitoring unit 34 determines that the operation information is the detection target when the detection target similar region is included in the screen represented by the screen information and the operation position is included in the region.

  For example, the operation monitoring unit 34 may store a control panel image for performing computer settings, a directory service management window image, a server function management console image, and the like as images to be detected. Good. In this case, the operation monitoring unit 34 can detect operations on the windows and consoles represented by these images. Further, the operation monitoring unit 34 may store the setting window and menu item images for changing the display format of such windows and consoles as detection target images. Thereby, the operation monitoring unit 34 can detect an operation for changing a display format for a window or a console in which an operation desired to be detected is performed, and can prevent the display format from being changed. Thereby, the operation monitoring unit 34 more reliably detects the detection target image.

  Further, for example, the operation monitoring unit 34 may store a menu list displayed by right-clicking on a file or folder icon or an empty space in a folder window as an image to be detected. For example, the operation monitoring unit 34 may store an image of an area including items such as copy, cut, and paste in the menu list as the detection target image. Accordingly, the operation monitoring unit 34 can detect copy, cut, and paste operations for files and folders.

  Note that the coordinate system representing the operation position included in the operation information received from the client terminal 20 may be different from the coordinate system on the screen represented by the screen information acquired by the own device. This is presumably because the display size (resolution, display range, etc.) of the remote desktop screen has been changed in the client terminal 20. In such a case, the operation monitoring unit 34 may perform the above-described processing after converting the operation position included in the operation information into a coordinate system on the screen represented by the screen information.

  In addition, the operation monitoring unit 34 detects whether the operation information is a detection target based on the operation content included in the operation information in addition to whether or not the operation position is included in the detection target similar region on the screen represented by the screen information. It may be determined whether or not there is. For example, in the screen represented by the screen information, the operation monitoring unit 34 determines that the operation information is a detection target when the operation position is included in the detection target similar region and the operation content is a left click. Good. In this case, the operation monitoring unit 34 may store the operation content of the detection target for each detection target image.

  For example, when the operation monitoring unit 34 determines that the operation information is a detection target, the operation monitoring unit 34 may perform a predetermined block process. The predetermined block process may be, for example, a process of notifying the client terminal 20 of information indicating that the corresponding operation is not possible without transmitting the operation information to the server 10. The predetermined block process may be a process of notifying the client terminal 20 of information indicating that the subsequent operation is blocked without transmitting the subsequent operation information to the server 10. . Specifically, the operation monitoring unit 34 may generate image information representing notification contents. Then, the operation monitoring unit 34 may control the remote desktop transmission unit 32 so as to transmit the generated image information of the notification content to the client terminal 20 together with information instructing to superimpose and display the image information on the remote desktop screen. .

  For example, the operation monitoring unit 34 may store such block processing in association with the detection target image. In this case, when the operation monitoring unit 34 determines that the operation information is a detection target, the operation monitoring unit 34 executes a block process associated with the detection target image used for the determination.

  The operation information transmission unit 35 transmits operation information that is determined not to be detected by the operation monitoring unit 34 to the server 10. That is, the operation information transmission unit 35 does not transmit the operation information determined to be detected by the operation monitoring unit 34 to the server 10.

  Further, when the operation monitoring unit 34 determines that the operation information is not a detection target, the operation information transmission unit 35 may not transmit the operation information received from the client terminal 20 to the server 10 thereafter.

  The operation of the information processing system 1 configured as described above will be described with reference to FIG. In FIG. 3, the left diagram shows the operation of the client terminal 20, the right diagram shows the operation of the server 10, and the center diagram shows the operation of the relay device 30. In addition, it is assumed that the relay device 30 stores an image to be detected in the memory 1002 in advance. In FIG. 3, it is assumed that a remote desktop connection via the relay device 30 has already been established between the server 10 and the client terminal 20.

  First, the server 10 transmits information representing a remote desktop to the client terminal 20 via the relay device 30 (step S1).

Next, in the relay device 30, the screen acquisition unit 31 receives information representing the remote desktop from the server 10. Then, the screen acquisition unit 31 acquires screen information based on the received information (step S2). As described above, the screen information is image data representing a predetermined area of the screen that can be displayed on the remote desktop. Next, in the relay device 30, the remote desktop transmission unit 32 sends information representing the remote desktop to the client terminal 20. (Step S3).

  Next, the client terminal 20 displays a remote desktop screen on the display device 2003 based on the received information (step S4). As described above, at this time, the client terminal 20 displays the remote desktop screen with the display size based on the setting of the own device.

  Next, the client terminal 20 acquires operation information for the display area of the remote desktop screen via the input device 2004 (step S5).

  Next, the client terminal 20 transmits the input operation information to the server 10 via the relay device 30 (step S6).

  Next, in the relay device 30, the operation information receiving unit 33 receives operation information from the client terminal 20. Then, the operation monitoring unit 34 determines whether or not the received operation information is GUI-based operation information (step S7). This can be determined by, for example, whether or not the operation position is included in the operation information.

  Here, in the case of GUI-based operation information, the operation monitoring unit 34 determines whether the detection target similar area is included in the screen represented by the screen information, and whether the operation position is included in the detection target similar area. To analyze. And the operation monitoring part 34 judges whether operation information is a detection target based on an analysis result (step S8).

  Here, when the operation information is a detection target, the operation monitoring unit 34 executes a predetermined block process (step S9).

  For example, the operation monitoring unit 34 may generate image information indicating that the operation is not possible as described above. Then, the operation monitoring unit 34 controls the remote desktop transmission unit 32 to transmit the generated image information to the client terminal 20 together with information for instructing to superimpose and display the generated image information on the screen of the remote desktop.

  On the other hand, the case where it is determined in step S7 that the operation information is not GUI-based or the operation information is determined not to be detected in step S8 will be described. In this case, the operation information transmission unit 35 transmits the operation information to the server 10 (step S10).

  And the server 10 performs the process based on the received operation information (step S11).

  In step S8, as described above, the operation monitoring unit 34 further determines whether the operation content is a detection target in addition to whether or not the operation position is included in the detection target similar region in the screen represented by the screen information. You may analyze whether or not. Then, the operation monitoring unit 34 may determine that the operation information is the detection target when the operation position is included in the detection target similar region and the operation content is the detection target. In this case, the operation monitoring unit 34 may determine that the operation information is not the detection target when the operation position is not included in the detection target similar region or the operation content is not the detection target.

  Further, the relay device 30 may omit the execution of step S9 when the block process is not particularly defined. Even in this case, the operation information transmitting unit 35 does not transmit the operation information determined to be the detection target to the server 10, and thus the server 10 does not execute the processing based on the detection target operation.

  Above, description of operation | movement of the information processing system 1 is complete | finished.

  Next, effects of the embodiment of the present invention will be described.

  The information processing system as an embodiment of the present invention can prevent an undesirable GUI-based operation from a client terminal to a server without mounting a monitoring agent on the server.

  The reason will be described. In the present embodiment, the relay device relays communication between the server and a client terminal connected to the server by remote desktop. In the relay device, the screen acquisition unit receives information representing the remote desktop from the server, and acquires screen information that is image data representing a predetermined area of the screen that can be displayed as the remote desktop based on the received information. . Then, the remote desktop transmission unit transmits information representing the remote desktop to the client terminal. The operation information receiving unit receives operation information representing an operation performed on the display area of the remote desktop screen from the client terminal from the client terminal. Then, the operation monitoring unit determines whether or not the operation information is a detection target based on whether or not the operation position is included in the detection target similar region on the screen represented by the screen information. This is because the operation information transmission unit transmits the operation information determined not to be detected to the server and does not transmit the operation information determined to be detected to the server.

  As described above, in this embodiment, the relay apparatus always acquires and stores screen information that is image data representing a predetermined area of a screen that can be displayed on the remote desktop. Thereby, this embodiment can specify the operation position on the screen represented by the screen information based on the operation position included in the operation information, regardless of how the remote desktop display size is set in the client terminal. . For this reason, this Embodiment can analyze whether the operation position is included in the detection target similar region in the screen represented by the screen information. In particular, when the screen information represents the entire area of the screen that can be displayed on the remote desktop, the present embodiment can detect the detection target similar region more reliably and accurately. As a result, this embodiment can set the operation of the detection target with finer granularity than the case of performing detection based on the input character string or window title without installing the monitoring agent on the server, Such an operation can be prevented.

  As described above, this embodiment can prevent a GUI-based undesired operation even when fraud is not detected in the connection itself from the client terminal to the server.

  Note that, in the present embodiment, the description has focused on an example in which there is one client terminal connected to the server by remote desktop. The present embodiment is not limited to this, and can also be applied to a case where a plurality of client terminals are connected to a server by remote desktop connection. In this case, each functional block in the relay apparatus is configured to hold information for each client terminal and operate for each client terminal.

  Further, in the present embodiment, the operation monitoring unit is described as transmitting operation information that is not GUI-based to the server as it is. Not limited to this, the operation monitoring unit may determine whether operation information that is not GUI-based is also a detection target based on a known method based on a character string or the like.

  Further, in the present embodiment, the description has focused on an example in which each functional block of the relay device is realized by a CPU that executes a computer program stored in a storage device or a ROM. However, the present invention is not limited to this, and some, all, or a combination of each functional block may be realized by dedicated hardware.

  In the present embodiment, the functional blocks of the relay device may be realized by being distributed among a plurality of devices.

  In each embodiment of the present invention described above, the operation of the relay apparatus described with reference to the flowchart may be stored in a storage device (storage medium) of a computer as a computer program of the present invention. Then, the computer program may be read and executed by the CPU. In such a case, the present invention is constituted by the code of the computer program or a storage medium.

  Further, the present invention is not limited to this embodiment, and can be implemented in various modes.

DESCRIPTION OF SYMBOLS 1 Information processing system 10 Server 20 Client terminal 30 Relay device 31 Screen acquisition part 32 Remote desktop transmission part 33 Operation information reception part 34 Operation monitoring part 35 Operation information transmission part 1001, 2001, 3001 CPU
1002, 2002, 3002 Memory 1005, 2005, 3005 Network interface 2003 Display device 2004 Input device

Claims (7)

A screen acquisition unit that receives information representing a remote desktop provided to a client terminal from a server, and acquires screen information that is image data representing a predetermined area of a screen that can be displayed as the remote desktop based on the received information When,
A remote desktop sending unit for sending information representing the remote desktop to the client terminal;
An operation information receiving unit that receives, from the client terminal, operation information representing an operation performed on the display area of the remote desktop screen in the client terminal;
Whether or not the operation information is a detection target based on whether or not the operation position based on the operation information is included in an area that satisfies a similarity condition with respect to the detection target image on the screen represented by the screen information An operation monitoring unit for determining whether or not
An operation information transmitting unit that transmits operation information that is determined not to be detected by the operation monitoring unit to the server;
A relay device comprising:   The relay apparatus according to claim 1, wherein the operation monitoring unit performs predetermined block processing when determining that the operation information is a detection target.   The relay apparatus according to claim 2, wherein when the operation monitoring unit determines that the operation information is a detection target, the operation monitoring unit performs the block processing associated with the detection target image used for the determination. The relay device according to any one of claims 1 to 3,
The client terminal;
The server;
Information processing system with Receives information representing the remote desktop provided to the client terminal from the server,
Based on the received information, obtain screen information that is image data representing a predetermined area of the screen that can be displayed as the remote desktop,
Sending information representing the remote desktop to the client terminal;
Operation information representing an operation performed on the display area of the remote desktop screen in the client terminal is received from the client terminal;
Whether or not the operation information is a detection target based on whether or not the operation position based on the operation information is included in an area that satisfies a similarity condition with respect to the detection target image on the screen represented by the screen information Determine whether
A method of transmitting operation information determined not to be detected to the server. The server
Send information representing the remote desktop to the client terminal via the relay device,
The relay device is
Based on the information received from the server, obtain screen information that is image data representing a predetermined area of the screen that can be displayed as the remote desktop;
Sending information representing the remote desktop to the client terminal;
The client terminal is
Based on the information received from the relay device, display the remote desktop screen on a display device,
When the operation information representing the operation for the display area of the displayed remote desktop screen is acquired from the input device, the acquired operation information is transmitted to the server via the relay device,
The relay device is
Whether or not the operation information is a detection target based on whether or not the operation position based on the operation information is included in an area that satisfies a similarity condition with respect to the detection target image on the screen represented by the screen information Determine whether
The operation information determined not to be detected is sent to the server,
The server
A method of executing processing based on the operation information received from the relay device. A screen acquisition step of receiving information representing a remote desktop provided to the client terminal from the server, and acquiring screen information that is image data representing a predetermined area of the screen that can be displayed as the remote desktop based on the received information When,
A remote desktop transmission step of transmitting information representing the remote desktop to the client terminal;
An operation information receiving step for receiving, from the client terminal, operation information representing an operation performed on the display area of the remote desktop screen in the client terminal;
Whether or not the operation information is a detection target based on whether or not the operation position based on the operation information is included in an area that satisfies a similarity condition with respect to the detection target image on the screen represented by the screen information An operation monitoring step for determining whether or not
An operation information transmission step of transmitting operation information determined not to be detected in the operation monitoring step to the server;
That causes a computer device to execute the program.

Images