JP2012090077A - Portable terminal, and method for operating processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a portable terminal and a method for operating a processing apparatus which enable an intuitive operation of a processing apparatus such as an image forming apparatus.SOLUTION: The portable terminal having a display section includes: a position identification section for identifying the position of the terminal; a direction identification section for identifying the direction of the terminal; and a control section for identifying a predetermined processing apparatus present within a predetermined distance range of the terminal and in the specified direction of the terminal according to position information of one or more processing apparatuses stored beforehand, and displaying on the display section an operation screen for remote operation of the predetermined processing apparatus. An imaging section is further provided to effect imaging in the specified direction of the terminal, and the control section synthesizes the video captured by the imaging section with the operation screen.

Description

  The present invention relates to a method for operating a portable terminal and a processing apparatus, and more particularly, to a portable terminal having a function for specifying a position and a direction, and a method for operating a processing apparatus using the portable terminal.

  An image forming apparatus (MFP: Multi Function Peripheral) equipped with a copy function, a print function, a scan function, etc. is widespread, and a company or the like connects a plurality of image forming apparatuses to a network and forms an image selected by a user. Printed material is output from the device.

  In such a system, when the image forming apparatus to be processed by the user is used by another user, the process is waited until the job of the other user is completed. Therefore, it is necessary to check which image forming apparatus is processing a job. Conventionally, the user has to move to a place where the panel of the image forming apparatus can be seen and check the display on the panel. Further, in order to select the function of the image forming apparatus, select a file to be processed, or set processing conditions, the user had to visit the image forming apparatus and operate the panel. .

  As a method for eliminating such complexity, there is a method for remotely checking and operating the state of the image forming apparatus using a WEB browser, such as PageScope WebConnection. In this method, a connection URL (Uniform Resource Locator) is used. If the IP (Internet Protocol) address of the image forming apparatus is necessary, and the user wants to know the status of a plurality of image forming apparatuses at a time, it is necessary to access and view each image forming apparatus. The state confirmation and operation of the image forming apparatus cannot be performed easily.

  As another method, Patent Document 1 below is an image forming processing system that forms an image with an image forming apparatus based on data transmitted from a portable terminal device that can communicate with a server device wirelessly. The server device, in response to a request from the portable terminal device, retrieves an image forming device on the network in the vicinity of the portable terminal device, and the location of the image forming device found by the retrieving unit A generating unit that generates data viewable by the portable terminal for displaying a map image indicating a position, and identification information for identifying the data generated by the generating unit are notified to the portable terminal. In response to the access request to the data indicated by the notification means and the identification information, the data generated by the generation means is directed to the portable terminal device And an image forming processing system configured to form an image by an image forming apparatus designated in the portable terminal device from among the image forming apparatuses shown in the map image. Yes.

Japanese Patent Laid-Open No. 2004-2342189

  The method of Patent Document 1 is a method of displaying a map image showing the location of an image forming apparatus on a portable terminal, and grasping the positional relationship between the portable terminal that transmits data and the surrounding image forming apparatuses. However, in this method, since the positional relationship between the mobile terminal and the image forming apparatus must be grasped on a two-dimensional map, it is difficult to know in which direction the image forming apparatus is located. Therefore, there is a problem in that it is difficult to check the status of the image forming apparatus and perform operations.

  The present invention has been made in view of the above problems, and a main object thereof is to provide a portable terminal capable of intuitively operating a processing apparatus such as an image forming apparatus and a method of operating the processing apparatus. It is in.

  In order to achieve the above object, the present invention provides a portable terminal including a display unit, a position specifying unit that specifies the position of the terminal, a direction specifying unit that specifies the direction of the terminal, Based on position information of a plurality of processing devices, a predetermined processing device existing in a predetermined distance range from the own terminal and in a specific direction of the own terminal is specified, and the predetermined processing device is remotely operated A control unit that displays an operation screen on the display unit.

  The present invention also provides a method for operating the processing device in a system in which a mobile terminal including a display unit, one or a plurality of processing devices, and a server are connected via a communication network. Or a first step of storing a device management table describing position information of a plurality of processing devices, and the portable terminal identifies the position of the own terminal, notifies the server of the position information of the own terminal, and the server Referring to the device management table, the processing device existing within a predetermined distance range from the portable terminal is identified and notified to the portable terminal, the portable terminal identifies the direction of the own terminal, and the predetermined A second step of identifying a predetermined processing device existing in a specific direction of the own terminal from among the processing devices existing within the distance range, and the portable terminal for remotely operating the predetermined processing device A third step of displaying the drawing plane on the display unit, in which to run.

  The present invention also provides a method for operating the processing device in a system in which a mobile terminal including a display unit, one or a plurality of processing devices, and a server are connected via a communication network. Or a first step of storing a device management table describing position information of a plurality of processing devices, and the portable terminal specifies the position and direction of the own terminal and notifies the server of the position information and direction information of the own terminal Then, the server refers to the device management table and identifies a predetermined processing device existing within a predetermined distance range from the mobile terminal and in a specific direction of the mobile terminal and notifies the mobile terminal And a third step in which the portable terminal displays an operation screen for remotely operating the predetermined processing device on the display unit.

  The present invention is also a method for operating the processing device in a system in which a mobile terminal including a display unit and one or more processing devices are connected via a communication network, wherein the mobile terminal includes the one or more processing devices. A first step of storing a device management table in which the position information of the processing device is stored, and specifying the position and direction of the own terminal, referring to the device management table, within a predetermined distance range from the own terminal, and Executing a second step of specifying a predetermined processing device existing in a specific direction of the terminal, and a third step of displaying an operation screen for remotely operating the predetermined processing device on the display unit It is.

  According to the operation method of the portable terminal and the processing apparatus of the present invention, it is possible to intuitively operate a processing apparatus such as an image forming apparatus.

  The reason is that the display unit of the mobile terminal selects functions, selects files, and sets processing conditions for processing devices that exist around the mobile terminal and in a specific direction of the mobile terminal. This is because an operation screen is displayed, and the processing device can be remotely operated on the operation screen.

  As a result, the user does not need to go to the front of the processing apparatus, and can also operate the processing apparatus arranged at a position that cannot be physically seen, such as the other side of the wall. Further, the operation can be performed without knowing the IP address or URL of the processing apparatus, and the operability can be remarkably improved.

  Further, since the processing device can be operated on the screen of the portable terminal, it is possible to reduce a risk that a confidential document, a password, and the like can be peeped from the side as in the case of operating from the panel of the processing device.

It is a figure which shows the structural example of the control system which concerns on 1st Example of this invention. It is a block diagram which shows the structure of AR server which concerns on 1st Example of this invention. It is a block diagram which shows the structure of the portable terminal which concerns on 1st Example of this invention. 1 is a block diagram illustrating a configuration of an image forming apparatus according to a first exemplary embodiment of the present invention. It is a figure which shows the image of AR (augmented reality) application with the portable terminal which concerns on 1st Example of this invention. It is a figure which shows an example of the screen (status display screen) displayed on the portable terminal which concerns on 1st Example of this invention. It is a figure which shows an example of the screen (operation screen) displayed on the portable terminal which concerns on 1st Example of this invention. It is a figure which shows the other example of the screen (operation screen) displayed on the portable terminal which concerns on 1st Example of this invention. It is a figure which shows the other example of the screen (operation screen) displayed on the portable terminal which concerns on 1st Example of this invention. It is a sequence diagram which shows operation | movement of the control system which concerns on 1st Example of this invention. FIG. 3 is a flowchart illustrating a remote operation method for the image forming apparatus according to the first exemplary embodiment of the present invention. It is an example of the apparatus management table memorize | stored in AR server which concerns on 1st Example of this invention. It is an image figure which shows the method of specifying the position from electric field strength. It is a figure which shows the method of calculating the position which synthesize | combines status information. It is a figure which shows the image of AR (augmented reality) application with the portable terminal which concerns on 2nd Example of this invention. It is a figure which shows an example of the screen (operation screen) displayed on the portable terminal which concerns on 2nd Example of this invention. It is a figure which shows the other example of the screen (operation screen) displayed on the portable terminal which concerns on 2nd Example of this invention. It is a figure which shows the other example of the screen (operation screen) displayed on the portable terminal which concerns on 2nd Example of this invention. It is a figure which shows the other example of the screen (operation screen) displayed on the portable terminal which concerns on 2nd Example of this invention. It is an example of the data management table memorize | stored in AR server which concerns on 2nd Example of this invention. It is a sequence diagram which shows operation | movement of the control system which concerns on 2nd Example of this invention.

  As shown in the background art, in a system in which a plurality of image forming apparatuses are connected to a network such as a company, it is required to easily operate the image forming apparatus installed in an appropriate place. A method for displaying an image forming apparatus on a map has been proposed. However, this method has a problem that information to be displayed is limited, and it is difficult to know where the image forming apparatus displayed on the two-dimensional map is actually located.

  Therefore, in one embodiment of the present invention, a processing device that exists in a specific direction (viewing angle of the camera) of the mobile terminal is specified from the processing devices such as an image forming device around the mobile terminal, and the image is captured by the camera. An operation screen for operating the specified processing device is synthesized and displayed on the live view video so that the processing device can be remotely operated intuitively.

  Although there is an augmented reality technology that displays information that is uploaded by a user linked to position data and displayed through a smartphone camera, such as a Sekai camera, a processing device as in the present invention exists. There is nothing to display the state of the system in real time or to operate the processing device.

  In order to describe the above-described embodiment of the present invention in more detail, an operation method of the portable terminal and the processing apparatus according to the first example of the present invention will be described with reference to FIGS. FIG. 1 is a diagram illustrating a configuration example of a control system according to the present embodiment, and FIGS. 2 to 4 are block diagrams illustrating configurations of an AR server, a portable terminal, and an image forming apparatus, respectively. FIG. 5 is a diagram illustrating an image of an augmented reality application on the mobile terminal, and FIGS. 6 to 9 are diagrams illustrating examples of screens displayed on the mobile terminal. FIG. 10 is a sequence diagram showing the operation of the control system of this embodiment, and FIG. 11 is a flowchart showing the remote operation method of the image forming apparatus of this embodiment. 12 is an example of a device management table stored in the AR server, FIG. 13 is an image diagram showing a method of specifying the position of the mobile terminal from the electric field strength, and FIG. 14 is a position where the status information is synthesized. It is a figure which shows the method of calculating.

  As shown in FIG. 1, a control system 10 according to the present embodiment includes an augmented reality (AR) server 20, a mobile terminal 30 such as a smartphone, a mobile phone, or a PDA (Personal Digital Assistants), and a mobile terminal 30. It is configured with a remotely operated processing device (in this embodiment, it is assumed to be an image forming device 40 such as an MFP), and these are connected to a network such as a LAN (Local Area Network) or a WAN (Wide Area Network) ( The portable terminal 30 is connected to a network via a wireless router or a wireless base station. Hereinafter, the configuration of each apparatus will be described in detail.

[AR server]
As shown in FIG. 2, the AR server 20 includes a control unit 21, a storage unit 22, a communication I / F unit 23, and the like.

  The control unit 21 includes a CPU (Central Processing Unit) and a memory such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and controls the entire AR server 20. Further, the control unit 21 receives information (referred to as status information) indicating the state of the apparatus from the image forming apparatus 40 and information (referred to as function information) such as functions that can be executed by the apparatus and conditions that can be set. In addition to acquiring the position information from the mobile terminal 30, the image forming apparatus 40 existing around the mobile terminal 30 (within a predetermined distance range from the mobile terminal 30) is specified, and a list of the image forming apparatuses 40 ( Device information processing unit 21a for transmitting status information and function information (collectively referred to as device information) and the like to the mobile terminal 30, and obtaining instruction information from the mobile terminal 30; It also functions as an apparatus control unit 21b that causes the image forming apparatus 40 to execute a function instructed according to the instruction information.

  The storage unit 22 is configured by an HDD (Hard Disk Drive) or the like, and includes an apparatus management table (to be described later), user operation authority for each image forming apparatus 40, PDL (Pages) such as PCL (Printer Control Language) and PS (Post Script). Save actual data described in Description Language).

  The communication I / F unit 23 is an interface such as a NIC (Network Interface Card) or a modem, and performs communication with the portable terminal 30 and the image forming apparatus 40 based on a standard such as Ethernet (registered trademark).

[Mobile devices]
As shown in FIG. 3, the portable terminal 30 includes a control unit 31, a storage unit 32, a communication I / F unit 33, a display unit 34, an operation unit 35, a position specifying unit 36, a direction specifying unit 37, an image pickup unit 38, and the like. Composed.

  The control unit 31 includes a CPU and a memory such as a RAM and a ROM, and controls the mobile terminal 30 as a whole. In addition, the control unit 31 transmits the position information of the own terminal to the AR server 20, acquires a peripheral MFP list of the image forming apparatus 40 around the own terminal from the AR server 20, and device information of the image forming apparatus 40 ( The apparatus information acquisition unit 31a for acquiring status information and function information) and the image forming apparatus 40 to be remotely operated are specified from the peripheral MFP list, and the status information is displayed on the display unit 34 (or combined with the imaging screen). Thus, the state of the image forming apparatus 40 can be confirmed, the operation screen created based on the function information is displayed on the display unit 34 (or combined with the imaging screen), and the image forming apparatus 40 can be remotely operated. It also functions as the management unit 31b. The functions as the device information acquisition unit 31a and the device management unit 31b may be realized as hardware, or a program that causes the control unit 31 to function as the device information acquisition unit 31a and the device management unit 31b (referred to as an AR application). .).

  The storage unit 32 is configured by an HDD or the like, and stores a peripheral MFP list and device information acquired from the AR server 20, electronic data to be processed by the image forming device 40, and the like.

  The communication I / F unit 33 is an interface such as a NIC or a modem, and is connected to a network via a wireless router or a wireless base station, and communicates with the AR server 20 and the image forming apparatus 40.

  The display unit 34 is an LCD (Liquid Crystal Display), an organic EL (electroluminescence) display, or the like, and displays a screen based on status information (a status display screen described later), a screen based on function information (an operation screen described later), and the like. .

  The operation unit 35 is a hard key or a touch panel on the display unit 34, accepts operations on the displayed status display screen and operation screen, and enables various instructions to the image forming apparatus 40.

  The position specifying unit 36 specifies the position (coordinates) of the terminal itself using GPS (Global Positioning System). In addition, the direction specifying unit 37 specifies the direction of the terminal itself using a self-supporting positioning technique that combines a gyro and an acceleration sensor. Note that the image forming apparatus 40 whose position is specified transmits electromagnetic waves (radio waves defined by standards such as wireless LAN, Wi-Fi (Wireless Fidelity), Bluetooth, etc.), and the mobile terminal 30 receives the electromagnetic waves. If possible, as shown in FIG. 13, the portable terminal 30 measures the electric field strength of electromagnetic waves transmitted from a plurality (preferably three or more) of the image forming apparatuses 40, and based on the value of the electric field strength, The position (coordinates) and direction of the terminal with respect to the image forming apparatus 40 may be specified. Further, the image captured by the image capturing unit 38 is analyzed, and the image forming apparatus 40 is identified by recognizing the color, shape, pattern, etc., and the position and direction of the terminal itself are identified from the position of the image forming apparatus. Good. Further, the image forming apparatus 40 is identified by recognizing a bar code or the like attached to the image forming apparatus 40 or by recognizing a tag such as RFID (Radio Frequency Identification). You may identify the position and direction of an own terminal from a position.

  The imaging unit 38 is a CCD (Charge Coupled Devices) camera, a CMOS (Complementary Metal Oxide Semiconductor) camera, or the like, and acquires an image of the image forming apparatus 40 and the like to obtain a live view video.

  In FIG. 3, the imaging unit 38 is provided in the portable terminal 30, but the imaging unit 38 is omitted when it is not necessary to synthesize a status display screen or an operation screen with the live view video captured by the imaging unit 38. You can also.

[Image forming apparatus]
As shown in FIG. 4, the image forming apparatus 40 includes a control unit 41, a storage unit 42, a communication I / F unit 43, a display unit 44, an operation unit 45, a scanner unit 46, a printing unit 47, and the like.

  The control unit 41 includes a CPU and a memory such as a RAM and a ROM, and controls the entire image forming apparatus 40. The control unit 41 also functions as a data analysis unit that analyzes actual data described in PCL or PDL, and an image processing unit that generates image data by rasterizing (bitmap development) the actual data based on the analysis result. .

  The storage unit 42 is configured by an HDD or the like, and stores actual data, image data, various setting information, and the like.

  The communication I / F unit 43 is an interface such as a NIC or a modem, and performs communication with the AR server 20 and the mobile terminal 30 based on a standard such as Ethernet (registered trademark).

  The display unit 44 is an LCD, an organic EL display, or the like, and displays various screens for realizing functions such as copying, scanning, printing, and FAX.

  The operation unit 45 is a hard key or a touch panel on the display unit 44, and performs various instructions for functions such as copying, scanning, printing, and FAX.

  The scanner unit 46 is a part that optically reads image data from a document sheet on a document table. The light source scans the document, the image sensor such as a CCD that converts light reflected from the document into an electrical signal, An A / D converter or the like for A / D converting the signal is used.

  The printing unit 47 transfers the image data to a sheet. Specifically, the photosensitive drum charged by the charging device is irradiated with light corresponding to the image from the exposure device to form an electrostatic latent image, and the toner charged by the developing device is attached and developed. The image is primarily transferred to a transfer belt, secondarily transferred from the transfer belt to a paper medium, and a fixing device fixes the toner image onto the paper medium. Further, processing such as folding, bookbinding, and stapling is performed as necessary.

  In the present embodiment, the control unit 21 of the AR server 20 is configured to identify the image forming apparatus 40 around the mobile terminal 30, but the location information of the image forming apparatus 40 is stored in the mobile terminal 30. The control unit 31 of the portable terminal 30 may be configured to identify the image forming apparatus 40 that exists around the terminal itself and in a specific direction. In this embodiment, the AR server 20 obtains status information and function information from the image forming apparatus 40, but the portable terminal 30 may obtain these information directly from the image forming apparatus 40. In this embodiment, the AR server 20 is configured to instruct the image forming apparatus 40 to perform processing. However, the portable terminal 30 may be configured to directly instruct the image forming apparatus 40 to perform processing. Thus, when the mobile terminal 30 has the function of the control unit 21 of the AR server 20, the AR server 20 can be omitted. In this embodiment, the image forming apparatus 40 is controlled. However, any apparatus that can be remotely operated by the portable terminal 30 such as a computer apparatus can be similarly controlled.

  Next, the basic operation of the control system 10 having the above configuration will be described with reference to the sequence diagram of FIG. In the following description, information about each image forming apparatus 40 (registered name, product name, IP address, connection port, SSL (Secure Socket Layer) communication availability, etc.) on the network, and position information of each image forming apparatus 40 (Latitude, longitude, altitude, etc.) are stored in advance in the storage unit 22 of the AR server 20 as the device management table shown in FIG. The position information of the device management table may be registered by the user. If the image forming apparatus 40 is equipped with a position specifying function such as GPS, the position information transmitted from the image forming apparatus 40 is used. Registration may be performed, or the portable terminal 30 may be brought near the image forming apparatus 40 and registered using position information transmitted from the portable terminal 30.

  The AR server 20 periodically accesses the image forming apparatus 40 connected to the network (or the image forming apparatus 40 side periodically accesses the AR server 20), and device information (status of each image forming apparatus 40) Information and function information) is acquired and stored in the storage unit 22 or the like. Although the access interval is arbitrary, since the state of the image forming apparatus 40 changes every moment, it is desirable to shorten the access interval after the mobile terminal 30 activates the AR application. In addition, here, the status information and the function information are periodically acquired. However, if the function information does not change, the function information may be acquired in advance and only the status information may be periodically acquired. .

  Next, when the user operates the operation unit 35 of the mobile terminal 30 to instruct activation of the AR application, the control unit 31 activates the AR application and displays a login screen on the display unit 34. When the user inputs an ID or password and presses the login button, the control unit 31 notifies the AR server 20 of the login, the AR server 20 performs an authentication process, and notifies the mobile terminal 30 of the result. If the authentication result is OK, the control unit 31 activates the imaging unit 38 to start imaging, and displays the live view video on the display unit 34.

  Next, the position specifying unit 36 of the mobile terminal 30 specifies the position of the terminal itself, and the device information acquisition unit 31 a transmits the specified position information to the AR server 20. For example, in the case where the position is specified using the electric field intensity of the electromagnetic wave transmitted by the image forming apparatus 40, as shown in FIG. 13, the three image forming apparatuses 40 of apparatuses A, B, and C are each concentric in the figure. When the electromagnetic field is transmitted with the electric field strength of 1 to 3 to the region shown, the electric field strength of the device A is “1”, and the electric field strengths of the devices B and C are “2”, the mobile terminal 30 is shaded Since it exists in the area, the position (coordinates) of its own terminal is specified based on the position (coordinates) of each image forming apparatus 40.

  When the device information processing unit 21a of the AR server 20 acquires the position information from the mobile terminal 30, the device information processing unit 21a refers to the device management table stored in advance in the storage unit 22 or the like, and surrounds the mobile terminal 30 (the distance from the mobile terminal 30 is The image forming apparatus 40 existing within a predetermined range is extracted, and the extracted list of image forming apparatuses 40 (peripheral MFP list) and status information of the image forming apparatus 40 are transmitted to the portable terminal 30. The device information acquisition unit 31a of the portable terminal 30 stores the received peripheral MFP list and status information in the storage unit 32 or the like.

  Next, the position specifying unit 36 and the direction specifying unit 37 of the mobile terminal 30 specify the position and direction of the own terminal, and the device management unit 31b includes the surroundings of the mobile terminal 30 among the image forming apparatuses 40 in the peripheral MFP list. In addition, the image forming apparatus 40 existing in a specific direction is specified. Here, the peripheral MFP list is acquired from the AR server 20, and the image forming apparatus 40 that exists around the own terminal and in a specific direction is specified from the list. Information and direction information may be transmitted, and the AR server 20 may identify the image forming apparatus 40 that exists around the mobile terminal 30 and in a specific direction. Here, after acquiring the peripheral MFP list, the position specifying unit 36 is used to specify the position of the own terminal again. However, when the position of the portable terminal 30 is not moved (only the direction is rotated), Only the direction of the terminal itself may be specified using the direction specifying unit 37.

  Then, the apparatus management unit 31b calculates the position of the identified image forming apparatus 40 on the live view video based on the position and direction of the own terminal, the position information of the image forming apparatus 40, the angle of view of the imaging unit 38, and the like. The status information of the image forming apparatus 40 is displayed at the position. When the direction of the portable terminal 30 changes and the image forming apparatus 40 deviates from the specific direction, the display of the status information is erased. When the new image forming apparatus 40 enters the specific direction, the image forming apparatus The operation of displaying 40 status information is repeated. That is, when the portable terminal 30 is rotated 360 degrees, the status information of the image forming apparatus 40 existing in a specific direction is sequentially displayed.

  The display form of the status information is not particularly limited, and the status information may be displayed alone. When the image forming apparatus 40 is imaged by the imaging unit 38, the status information is displayed on the live view video of the image forming apparatus 40. The status information may be combined and displayed. FIG. 5 is a diagram showing a situation where the periphery is being imaged by the mobile terminal 30, and a live view image captured by the imaging unit 38 is displayed on the display unit 34 and connected to a network as necessary. A frame or the like showing the arrangement of the image forming apparatus 40 and the peripheral range of the portable terminal 30 is displayed. FIG. 6 is an enlarged view (only main components) of the display unit 34 of the portable terminal 30 when the image forming apparatus 40 is imaged. The image of the image forming apparatus 40 imaged by the image capturing unit 38. Includes a frame with status information (such as “standby” characters indicating that the image forming apparatus 40 is waiting for a job or “printing” characters indicating that a job is being executed). A screen (referred to as a status display screen) is synthesized. As described above, by combining the status information with the live view video of the image forming apparatus 40 and displaying it, the state of the image forming apparatus 40 can be intuitively grasped.

  The status information is not limited to the above, and the number of remaining jobs, the estimated job end time, etc. may be displayed, or detailed information such as user information that registered the job may be displayed. Further, an error state, a recovery method from the error, guidance indicating an operation procedure, and the like may be displayed. Further, when displaying the status information, the display size of the status information is changed according to the perspective according to the distance between the image forming apparatus 40 and the portable terminal 30, or the display color, transparency, and size are changed according to the state of the apparatus and the number of remaining jobs. The animation effect or the like may be changed, or items to be displayed may be changed according to user authority, system settings, or the like. In addition, instead of displaying the status for each device, the status may be displayed for each job so that job control such as job execution cancellation, deletion, and re-execution can be performed.

In addition, in the display form having the configuration of FIG. 6, in order to display status information on the image of the image forming apparatus 40, the x coordinate of the arrangement position of the status information and the center of the image forming apparatus 40 on the screen are used. It is sufficient that the x coordinate coincides with each other. As shown in FIG. 14A, the x-coordinate of the image forming apparatus 40 wk the screen display size (the size in the X direction) of the portable terminal 30, and as shown in FIG. When the specific direction (upward here) is the reference direction, the clockwise direction of the image forming apparatus 40 is αm, and the angle of view of the camera in the horizontal direction (more precisely, the shooting range angle within the displayable range) is set. Assuming that αc is the direction of the mobile terminal 30 from the reference direction (the center of the imaging range angle (αr) of the camera) is αc,
x = wk × (αm− (αc−αr / 2)) / αr
It becomes.

Further, as shown in FIG. 14C, αm is assumed that the coordinates of the portable terminal 30 are (xk, yk) and the coordinates of the image forming apparatus 40 are (xm, ym).
αm = 180− (tan ⁻¹ ((ym−yk) / (xm−xk)) × (360 / 2π))
It becomes.

  Therefore, if x is obtained according to the above equation and the x coordinate of the display position of the status information is set to this value, the status information can be synthesized at the center of the image of the image forming apparatus 40.

  Here, if the status information indicates that the image forming apparatus 40 is on standby, processing using the image forming apparatus 40 is possible, so that the image forming apparatus 40 can be remotely operated according to the following procedure. To do.

  Specifically, the user touches a predetermined operation (for example, touches an area of the status display screen, presses a specific operation button of the portable terminal 30, or presses a position corresponding to the operation panel of the image forming apparatus 40. The control unit 31 of the portable terminal 30 confirms the operation authority of the login user with the AR server 20 as necessary. The operation unit 21 of the AR server 20 acquires the operation authority of the login user from the storage unit 22 or the like, and transmits it to the mobile terminal 30.

  Next, the device management unit 31b of the portable terminal 30 acquires the function information of the image forming device 40 from the AR server 20, creates the operation information according to the function information and the user authority as necessary, and displays the display unit. 34. For example, as shown in FIG. 7, on the live view video, an operation screen for instructing functions (here, copy, scan, print, fax, MFP management) that can be executed by the login user for the image forming apparatus 40 (In this case, the function selection screen) is combined so that the image forming apparatus 40 can be remotely operated.

  The configuration of the function selection screen is arbitrary. For example, the function selection screen may have the same configuration as that of the operation screen displayed on the display unit 44 of the image forming apparatus 40, or all functions provided in the image forming apparatus 40 are displayed. Functions that cannot be executed may not be selected (for example, displayed in gray). Here, the function selection screen is combined with the live view video of the image forming apparatus 40 that performs remote operation. However, after specifying the image forming apparatus 40 by an operation such as touching an area of the status display screen, Since the live view video and the function selection screen of the image forming apparatus 40 do not necessarily have to be displayed in correspondence with each other, only the function selection screen may be displayed or any live view video (for example, the image forming apparatus 40). The function selection screen may be combined with a live view video that has not been captured.

  When the user selects a predetermined function (here, the print function) on the operation screen (function selection screen), the device management unit 31b of the mobile terminal 30 displays a list of file storage locations (for example, the AR server 20 and the mobile terminal). 30 and the image forming apparatus 40), and when a desired storage location (for example, the storage unit 32 of the portable terminal 30) is selected, a list of files stored in advance in the storage location is obtained, An operation screen for selection is created and displayed on the display unit 34. For example, as shown in FIG. 8, an operation screen (in this case, a file selection screen) describing the file name and file attributes is synthesized on the live view video, and the selection of the file to be printed by the image forming apparatus 40 is performed remotely. Enable operation.

  The configuration of the file selection screen is also arbitrary. For example, all files stored in the storage unit may be displayed, only files with a predetermined attribute may be displayed, or files created by the login user Alternatively, only files for which the login user has authority to use may be displayed. Also here, the file selection screen is combined with the live view video of the image forming apparatus 40 that performs remote operation. However, after specifying the image forming apparatus 40, the live view video and the file selection screen are not necessarily associated with each other. Therefore, only the file selection screen may be displayed, or the file selection screen may be combined with an arbitrary live view video.

  Next, when the user selects a file on the operation screen (file selection screen), the device management unit 31b of the mobile terminal 30 creates an operation screen corresponding to the function information and displays it on the display unit 34. For example, as shown in FIG. 9, an operation screen (in this case, a condition setting screen) for setting conditions (here, the number of copies, color, paper size, page designation, etc.) for the selected function is synthesized on the live view video. The condition setting for the selected function can be remotely operated.

  The configuration of the condition setting screen is also arbitrary. For example, the condition setting screen may have the same configuration as the operation screen displayed on the display unit 44 of the image forming apparatus 40, or all conditions that can be set for the selected function are displayed. Items that cannot be set by the login user may be displayed in gray. Also, here, the condition setting screen is combined with the live view video of the image forming apparatus 40 that performs remote operation. However, after specifying the image forming apparatus 40, the live view video and the condition setting screen are not necessarily associated with each other. Therefore, only the condition setting screen may be displayed, or the condition setting screen may be combined with an arbitrary live view video.

  Then, after setting the print conditions on the condition setting screen and pressing the OK button, the device management unit 31b of the mobile terminal 30 transmits the selected file, the selected function, and setting information to the AR server 20. Specifically, the position on the display unit 34 of the function selection screen, the file selection screen, and the condition setting screen is stored in advance, and the function, file, and condition corresponding to the position touched on the screen by the user are specified. Instruction information is created, and the instruction information is transmitted to the AR server 20. The AR server 20 generates a PCL file from the selected file and transfers it to the image forming apparatus 40, and the image forming apparatus 40 executes print processing based on the PCL file. In addition, when the PCL file is transferred to the image forming apparatus 40, the device management unit 31b of the portable terminal 30 displays a print acceptance completion screen on the display unit 34.

  Thereafter, when the user gives an instruction to end the AR application, the control unit 31 of the mobile terminal 30 notifies the AR server 20 of logout, and the AR server 20 executes logout processing and also controls the image forming apparatus 40 to be controlled. The device information acquisition interval of the mobile terminal 30 is restored, and the control unit 31 of the mobile terminal 30 ends the AR application.

  In the above description, after the status display screen is displayed, the operation screen (function selection screen, file selection screen, condition setting screen, etc.) is displayed. However, the status display screen may be omitted. In this case, when the image forming device 40 is imaged by the imaging unit 38, the function selection screen is automatically displayed, or when a predetermined operation is performed in a state where the image forming device 40 is imaged by the imaging unit 38, the function selection screen Is displayed. Further, when the direction of the portable terminal 30 changes and the image forming apparatus 40 deviates from a specific direction, the display of the function selection screen is erased. When the new image forming apparatus 40 enters the specific direction, the image formation is performed. The operation of displaying the function selection screen of the device 40 is repeated.

  In the above description, the operation screen is configured to display three screens: a function selection screen, a file selection screen, and a condition setting screen. However, one or more screens for operating the image forming apparatus 40 are displayed. It only has to be done. For example, in the case of copying, the file selection screen can be omitted, and in the case of scanning, a screen for designating the storage destination of the scanned image can be displayed instead of the file selection screen.

  Next, a method for remotely operating the image forming apparatus 40 using a screen displayed on the portable terminal 30 will be described in detail with reference to the flowchart of FIG. Note that the solid line frame in FIG. 11 indicates the operation of the mobile terminal 30, and the broken line frame indicates the operation of the AR server 20 or the image forming apparatus 40.

  First, the status information of the image forming apparatus 40 is combined with the live view video displayed on the display unit 34 of the mobile terminal 30 and displayed. When the user presses the status information display area (status display screen) (S101), the device information acquisition unit 31a of the mobile terminal 30 accesses the AR server 20, and the login server operates authority ( (Operation authority for each function such as copy, print, scan, fax transmission, management function) is acquired (S102). It should be noted that this operation authority may be added with restrictions such as the number of times of use and usage time period, or may be set for each image forming apparatus 40.

  Next, the device information acquisition unit 31a of the mobile terminal 30 acquires the function information of the image forming apparatus 40 (executability of each function, settable value of each function, etc.) from the AR server 20 (S103). The management unit 31b causes the display unit 34 to display a function selection screen as shown in FIG. 7 according to the operation authority of the login user and the function information of the image forming apparatus 40 (S104). The values that can be set for each function include, for example, color in the printing function, n in 1, double-sided, punch / staplability, color in the scanning function, resolution, etc., color in the copy function, n in 1, double-sided , Whether punching / stapling, etc., and fax transmission capability in the fax transmission function.

  Then, the device management unit 31b determines which function on the function selection screen has been pressed (S105), and executes processing according to the pressed function. Here, an example will be described in which the functions of copy, scan, and print are selected.

  First, when the copy button is pressed on the function selection screen, the apparatus management unit 31b displays a copy setting selection screen on the display unit 34 based on the function information of the image forming apparatus 40 (S106). When the user selects the copy setting (S107) and presses the start button (S108), the device management unit 31b transmits the copy setting information to the AR server 20 (S109), and the AR server 20 performs the copy setting. According to the information, the image forming apparatus 40 is instructed to execute copying (S110), and the image forming apparatus 40 executes copying (S111). Here, the copy setting information is transmitted from the portable terminal 30 to the AR server 20, and the AR server 20 instructs the image forming apparatus 40 to execute copying. However, the portable terminal 30 communicates with the image forming apparatus 40. When the function is installed, the image forming apparatus 40 may be instructed to execute copying directly from the portable terminal 30 without using the AR server 20.

  When the scan button is pressed on the function selection screen, the apparatus management unit 31b displays the scan setting selection screen on the display unit 34 based on the function information of the image forming apparatus 40 (S112). When the user selects the scan setting (S113), the apparatus management unit 31b displays a scan file storage destination selection screen on the display unit 34 (S114). When the user selects a storage destination on the scan file storage destination selection screen (S115) and presses the start button (S116), the apparatus management unit 31b transmits the scan setting information and the storage destination information to the AR server 20 (S117). The AR server 20 instructs the image forming apparatus 40 to execute a remote scan according to the scan setting information and the storage destination information (S118), and the image forming apparatus 40 executes the scan (S119). Similarly to the above, when the mobile terminal 30 has a function of communicating with the image forming apparatus 40, the remote scan is performed directly from the mobile terminal 30 to the image forming apparatus 40 without using the AR server 20. May be indicated.

  When the storage of the storage destination is the mobile terminal 30, the AR server 20 acquires scan data from the image forming apparatus 40 (S121), and the mobile terminal 30 acquires the scan data from the AR server 20 (S122). The scan data is saved (S123). If the storage at the storage destination is a shared server, the AR server 20 acquires scan data from the image forming apparatus 40 (S124), and transfers the scan data to the shared server (S125). When the storage destination is the storage unit 42 of the image forming apparatus 40, the scan data is stored in the storage unit 42 (S126).

  When the print button is pressed on the function selection screen, the device management unit 31b displays a file selection screen on the display unit 34 (S127). When the user selects a file (S128), the apparatus management unit 31b displays a print setting selection screen on the display unit 34 based on the function information of the image forming apparatus 40 (S129). When the user selects print settings (S130), the apparatus management unit 31b determines the storage location of the selected file (S131). If the selected file is stored in the mobile terminal 30, the print setting information and the selected file are stored. Data is transmitted to the AR server 20 (S132). If the file is stored in the shared server, the device management unit 31b transmits the print setting information and the path information of the selected file to the AR server 20 (S133). The AR server 20 transmits the information from the shared server according to the path information. A file is acquired (S134). After that, the AR server 20 creates a PCL file (S135), transfers it to the image forming apparatus 40 (S136), and the image forming apparatus 40 executes printing (S137).

  As described above, the mobile terminal 30 combines and displays the status information of the image forming apparatus 40 on the live view video of the image forming apparatus 40, and then displays an operation screen such as a function selection screen, a file selection screen, and a print setting screen. In order to display (preferably, display an operation screen combined with a live view video) and to remotely control the image forming apparatus 40 on the operation screen, the image forming apparatus 40 can be operated intuitively, and the user can The convenience can be improved significantly.

  Next, a method for operating the portable terminal and the processing apparatus according to the second embodiment of the present invention will be described with reference to FIGS. FIG. 15 is a diagram illustrating an image of an augmented reality application on a mobile terminal, and FIGS. 16 to 19 are examples of a display screen of the mobile terminal. FIG. 20 is a sequence diagram showing the operation of the control system of this embodiment, and FIG. 21 is an example of a data management table stored in the AR server.

  In the first embodiment, the file selection screen is displayed and the file is selected. However, when storing electronic data (document data, image data, music data, etc.) that can be processed by the processing device, Location information can also be associated with electronic data. Therefore, in this embodiment, information (for example, a data icon) specifying electronic data registered in association with the position where the imaging unit 38 is imaging is synthesized with the live view video, and the operation screen and the data icon are used. Thus, the image forming apparatus 40 can be remotely operated more intuitively.

  Specifically, as shown in FIG. 21, information about electronic data (type of file or folder, name of file or folder, extension, etc.) and information about the storage destination of electronic data (server name, file path, protocol) , Port number, etc.) and information (latitude, longitude, altitude, etc.) regarding the registration position of electronic data (the actual position of the device storing the electronic data may be a position in any space). The described data management table is created, and the data management table is stored in the storage unit 22 of the AR server 20 or the like.

  Then, the device information processing unit 21a of the AR server 20 acquires position information from the mobile terminal 30, refers to this data management table, and exists around the mobile terminal 30 (within a predetermined distance range from the mobile terminal 30). The electronic control unit 21b performs control for identifying the electronic data to be transmitted and transmitting a list of the electronic data (referred to as a peripheral data list) to the portable terminal 30, and the device control unit 21b acquires instruction information for the electronic data from the portable terminal 30. Then, control is performed to cause the image forming apparatus 40 to execute the function instructed according to the instruction information.

  In addition, the device information acquisition unit 31a of the mobile terminal 30 performs control to transmit the position information of the own terminal to the AR server 20 and acquire a list of electronic data around the own terminal (peripheral data list) from the AR server 20. The device management unit 31b synthesizes information (such as a data icon) for specifying electronic data with the live view video so that the electronic data can be processed by an appropriate image forming apparatus 40.

  In this embodiment, the control unit 21 of the AR server 20 is configured to identify the electronic data around the mobile terminal 30, but the location information of the electronic data is stored in the mobile terminal 30, and The control unit 31 may be configured to specify electronic data existing around the terminal and in a specific direction. When the mobile terminal 30 has the function of the control unit 21 of the AR server 20, the AR server 20 is used. It can be omitted.

  Hereinafter, the operation of the control system 10 configured as described above will be described with reference to the sequence diagram of FIG. In the following description, it is assumed that the device management table shown in FIG. 12 and the data management table shown in FIG. 21 are registered in advance in the storage unit 22 of the AR server 20.

  The AR server 20 periodically accesses the image forming apparatus 40 connected to the network (or the image forming apparatus 40 side periodically accesses the AR server 20), and device information (status of each image forming apparatus 40) Information and function information). Although the access interval is arbitrary, since the state of the image forming apparatus 40 changes every moment, it is desirable to shorten the access interval after the mobile terminal 30 activates the AR application. In addition, here, the status information and the function information are periodically acquired. However, if the function information does not change, the function information may be acquired in advance and only the status information may be periodically acquired. .

  Next, when the user operates the operation unit 35 of the mobile terminal 30 to instruct activation of the AR application, the control unit 31 activates the AR application and displays a login screen on the display unit 34. When the user inputs an ID or password and presses the login button, the control unit 31 notifies the AR server 20 of the login, the AR server 20 performs an authentication process, and notifies the mobile terminal 30 of the result. If the authentication result is OK, the control unit 31 activates the imaging unit 38 to start imaging, and displays the live view video on the display unit 34.

  Next, the position specifying unit 36 of the mobile terminal 30 specifies the position of the terminal itself, and the device information acquisition unit 31 a transmits the specified position information to the AR server 20.

  When the device information processing unit 21a of the AR server 20 acquires the position information from the mobile terminal 30, the device information processing unit 21a refers to the device management table stored in advance in the storage unit 22 or the like, The image forming apparatus 40 existing within a predetermined range is extracted, and the extracted list of image forming apparatuses 40 (peripheral MFP list) and status information of the image forming apparatus 40 are transmitted to the portable terminal 30. The device information acquisition unit 31a of the portable terminal 30 stores the received peripheral MFP list and status information in the storage unit 32 or the like.

  Further, the AR server 20 refers to the data management table stored in advance, extracts electronic data existing around the mobile terminal 30 (distance from the mobile terminal 30 within a predetermined range), and lists the extracted electronic data (Peripheral data list) is transmitted to the portable terminal 30. The device information acquisition unit 31a of the portable terminal 30 stores the received peripheral data list in the storage unit 32 or the like.

  Next, the position specifying unit 36 and the direction specifying unit 37 of the mobile terminal 30 specify the position and direction of the own terminal, and the device management unit 31b includes the surroundings of the mobile terminal 30 among the image forming apparatuses 40 in the peripheral MFP list. In addition, the image forming apparatus 40 existing in a specific direction is specified. Then, the apparatus management unit 31b calculates the position of the identified image forming apparatus 40 on the live view video based on the position and direction of the own terminal, the position information of the image forming apparatus 40, the angle of view of the imaging unit 38, and the like. The status information of the image forming apparatus 40 is displayed at the position.

  In addition, the device management unit 31b of the mobile terminal 30 specifies electronic data that exists around the mobile terminal 30 and in a specific direction from among the electronic data in the peripheral data list. Then, the device management unit 31b calculates the position of the specified electronic data on the live view video based on the position and direction of the terminal itself, the position information of the electronic data, the angle of view of the imaging unit 38, and the like. A data icon for specifying the electronic data is displayed.

  When the direction of the portable terminal 30 changes and the image forming apparatus 40 and electronic data deviate from a specific direction, the status information and the data icon are deleted, and the new image forming apparatus 40 and electronic data are specified. When the direction is entered, the operation of displaying the status information and data icon of the image forming apparatus 40 is repeated.

  The display form of the status information and the data icon is not particularly limited, and the status information and the data icon may be displayed alone or may be combined with the live view video and displayed. FIG. 15 is a diagram illustrating a situation where the periphery is being imaged by the mobile terminal 30, and a data icon is combined with the live view image captured by the imaging unit 38. When displaying the data icon, the display size, display color, transparency, size, animation effect, etc. of the data icon may be changed according to the perspective according to the distance between the electronic data and the portable terminal 30.

  When the user selects electronic data by pressing a desired data icon or the like, the device management unit 31b of the mobile terminal 30 extracts the image forming apparatus 40 that can process the electronic data, and the electronic data is selected. When the user images the image forming apparatus 40 in a desired direction, the position specifying unit 36 and the direction specifying unit 37 specify the position and direction of the own terminal, and the device management unit 31b displays the live image of the captured image forming apparatus 40. A position on the view video is specified, and an operation screen for remotely operating the image forming apparatus 40 is synthesized at the position.

  FIG. 16 shows an example of an operation screen (here, a function instruction screen). Electronic data used for a function instructed on the function instruction screen is selected by dragging a data icon into the frame of the function execution screen. . FIG. 17 shows another example of the function instruction screen. When the image forming apparatus 40 that is taking an image can print and fax the selected electronic data, two function instruction screens are displayed. Then, by dragging the data icon into the frame of one of the function instruction screens, electronic data to be used for printing or FAX is selected.

  The configuration of the function instruction screen is arbitrary. For example, the function executed by the image forming apparatus 40 may be limited according to the use authority of the user who has logged into the mobile terminal 30 as in the first embodiment. Good. Also, here, after selecting the electronic data, the image forming apparatus 40 is selected by changing the position and orientation of the mobile terminal 30. For example, the function is applied to a live view image capturing the image forming apparatus 40. After the instruction screen is synthesized and the function instruction screen is selected, the electronic data may be selected by changing the position or orientation of the mobile terminal 30. In this example, the data icon is dragged onto the operation screen (function instruction screen). However, the data icon may be double-clicked while the function instruction screen is displayed. The screen may be dragged onto the data icon.

  Then, when the process is instructed on the function instruction screen (the OK button is pressed), the device management unit 31b of the mobile terminal 30 transmits instruction information instructing the process using the selected electronic data to the AR server 20, and the AR The server 20 acquires actual data corresponding to the data icon, generates a PCL file from the actual data, and transfers it to the image forming apparatus 40. The image forming apparatus 40 executes processing based on the PCL file.

  Thereafter, when the user gives an instruction to end the AR application, the control unit 31 of the mobile terminal 30 notifies the AR server 20 of logout, and the AR server 20 executes logout processing and also controls the image forming apparatus 40 to be controlled. The device information acquisition interval of the mobile terminal 30 is restored, and the control unit 31 of the mobile terminal 30 ends the AR application.

  In the above description, the function instruction screen is exemplified as the operation screen. However, as in the first embodiment, a screen for setting processing conditions may be displayed, and before the operation screen is displayed, A status display screen may be displayed. Further, in the above embodiment, the processing is performed using the electronic data in which the position is registered in advance. For example, as shown in FIG. 18, the folder stored in the processing device imaged by the imaging unit 38 is stored. After displaying and selecting the folder and the electronic data stored in the folder, the image forming apparatus 40 may be selected by changing the position and orientation of the portable terminal 30.

  As described above, the mobile terminal 30 displays an operation screen such as a function instruction screen on the live view video in which the data icons are combined (preferably, the operation screen is combined with the live view video and displayed). Since the image forming apparatus 40 can be remotely operated, the image forming apparatus 40 can be intuitively operated, and the convenience of the user can be significantly improved.

  In addition, this invention is not limited to description of the said Example, The structure and control can be changed suitably, unless it deviates from the meaning of this invention.

  For example, in the second embodiment, the operation screen is displayed when the image forming apparatus 40 is imaged. However, when electronic data is selected as shown in FIG. 19, the image forming apparatuses around the electronic data are selected. It is also possible to display an apparatus selection screen for displaying a list of 40 and execute the processing in the image forming apparatus 40 selected from the apparatus selection screen.

  In each of the above embodiments, the case where the image forming apparatus 40 is remotely controlled has been described. However, for any apparatus capable of processing electronic data, such as a computer apparatus, a projector, a television monitor, a shared server, and other portable terminals. However, the same can be applied. For example, you can display images by dragging the data icon of image data or document data to a projector or TV monitor, or drag the data icon to a computer device to perform operations such as transferring electronic data, sending e-mails, or playing music. You may make it perform.

  The present invention can be used for a mobile terminal and a method of remotely operating a processing apparatus using the mobile terminal.

DESCRIPTION OF SYMBOLS 10 Control system 20 AR server 21 Control part 21a Device information processing part 21b Device control part 22 Storage part 23 Communication I / F part 30 Portable terminal 31 Control part 31a Device information acquisition part 31b Device management part 32 Storage part 33 Communication I / F Unit 34 display unit 35 operation unit 36 position specifying unit 37 direction specifying unit 38 imaging unit 40 image forming apparatus 41a data analysis unit 41b image processing unit 42 storage unit 43 communication I / F unit 44 display unit 45 operation unit 46 scanner unit 47 printing Part

Claims (16)

A portable terminal comprising a display unit,
A location identifying unit that identifies the location of the terminal;
A direction identifying unit that identifies the direction of the terminal;
Based on the position information of one or more processing devices stored in advance, a predetermined processing device existing within a predetermined distance range from the own terminal and in a specific direction of the own terminal is specified, and the predetermined processing device is And a control unit that causes the display unit to display an operation screen for remote operation. Furthermore, an imaging unit for imaging the specific direction of the terminal is provided,
The mobile terminal according to claim 1, wherein the control unit synthesizes the operation screen with an image captured by the imaging unit.   As a result of moving the own terminal, the control unit is displayed on the display unit when the predetermined processing device does not exist within a predetermined distance range from the own terminal or in a specific direction of the own terminal. The mobile terminal according to claim 1, wherein the operation screen is erased.   The mobile terminal according to any one of claims 1 to 3, wherein the control unit displays information according to an authority of a user who logs in to the terminal on the operation screen.   The control unit acquires status information indicating a state of the predetermined processing apparatus before displaying the operation screen on the display unit, and displays the status information on the display unit. Item 5. The mobile terminal according to any one of Items 1 to 4.   The control unit identifies predetermined electronic data existing within a predetermined distance range from the own terminal and in a specific direction of the own terminal based on position information associated with one or more electronic data stored in advance. The portable terminal according to claim 1, wherein an icon for specifying the predetermined electronic data is displayed on the display unit. A method for operating the processing device in a system in which a portable terminal including a display unit, one or more processing devices, and a server are connected via a communication network,
A first step of storing, in the server, a device management table describing location information of the one or more processing devices;
The portable terminal identifies the position of the own terminal, notifies the server of the position information of the own terminal, and the server exists within a predetermined distance range from the portable terminal with reference to the device management table A processing device is identified and notified to the portable terminal, and the portable terminal identifies the direction of the own terminal and exists in a specific direction of the own terminal from among the processing devices existing within the predetermined distance range. A second step of identifying a predetermined processing device;
A method of operating a processing apparatus, wherein the portable terminal executes a third step of displaying an operation screen for remotely operating the predetermined processing apparatus on the display unit. A method for operating the processing device in a system in which a portable terminal including a display unit, one or more processing devices, and a server are connected via a communication network,
A first step of storing, in the server, a device management table describing location information of the one or more processing devices;
The portable terminal identifies the position and direction of the own terminal, notifies the server of the position information and the direction information of the own terminal, and the server refers to the device management table to determine a predetermined distance from the portable terminal. A second step of identifying a predetermined processing device within a range and in a specific direction of the mobile terminal and notifying the mobile terminal;
A method of operating a processing apparatus, wherein the portable terminal executes a third step of displaying an operation screen for remotely operating the predetermined processing apparatus on the display unit. A method for operating the processing device in a system in which a portable terminal including a display unit and one or more processing devices are connected via a communication network,
The mobile terminal is
A first step of storing a device management table describing position information of the one or more processing devices;
A second step of identifying the position and direction of the own terminal, referring to the device management table, and identifying a predetermined processing device within a predetermined distance range from the own terminal and in a specific direction of the own terminal; ,
And a third step of displaying an operation screen for remotely operating the predetermined processing device on the display unit. The portable terminal includes an imaging unit that captures the specific direction of the terminal.
The method for operating a processing apparatus according to claim 7, wherein, in the third step, the operation screen is combined with an image captured by the imaging unit.   In the third step, as a result of moving the portable terminal, when the predetermined processing device is no longer in a predetermined distance range from the portable terminal or in a specific direction of the portable terminal, the display The method of operating a processing apparatus according to claim 7, wherein the operation screen displayed on the section is deleted.   12. The processing device according to claim 7, wherein in the third step, information according to an authority of a user who logs in to the mobile terminal is displayed on the operation screen. Method of operation. In the second step, the portable terminal acquires status information indicating a state of the predetermined processing device,
The method of operating a processing apparatus according to claim 7, wherein, in the third step, the status information is displayed before the operation screen is displayed on the display unit. The operation screen includes a first screen for selecting a function to be executed by the predetermined processing device, and a second screen for selecting electronic data to be processed by the predetermined processing device,
The method of operating a processing apparatus according to any one of claims 7 to 13, wherein, in the third step, the second screen is displayed when a function is selected on the first screen. In the first step, a data management table describing position information associated with one or a plurality of electronic data is stored,
In the second step, referring to the data management table, electronic data existing within a predetermined distance range from the mobile terminal and in a specific direction of the mobile terminal is specified,
The method for operating a processing apparatus according to claim 7, wherein in the third step, an icon for specifying the predetermined electronic data is displayed on the display unit.   The method of operating a processing apparatus according to claim 15, wherein, in the third step, when the icon is selected, the operation screen corresponding to electronic data specified by the icon is displayed.

Images