CN108307084B - Information processing apparatus and information processing method - Google Patents

Abstract

Information processing apparatus and information processing method. An information processing apparatus includes a controller. If the first image related to the first device required to perform the cooperation function is specified, the controller performs control to present a guide indicating a second device capable of performing the cooperation function together with the first device.

Description

Information processing apparatus and information processing method

technical field

The present invention relates to an information processing apparatus and an information processing method.

Background technique

Japanese Unexamined Patent Application Publication Nos. 2015-177504 and 2015-223006 disclose techniques for causing a plurality of apparatuses to cooperate with each other.

However, in some cases the desired collaborative function may not be performed.

SUMMARY OF THE INVENTION

Therefore, an object of the present invention is to increase user convenience in the case of performing cooperative functions.

According to a first aspect of the present invention, there is provided an information processing apparatus including a controller. If the first image related to the first device required to perform the cooperation function is specified, the controller performs control to present a guide indicating a second device capable of performing the cooperation function together with the first device.

According to the second aspect of the present invention, if an image related to a device that cannot perform the cooperation function together with the first device is further specified, the controller performs control to present the guidance.

According to the third aspect of the present invention, if the operation of linking the first image and the image related to the device that cannot perform the cooperation function together with the first device is performed to each other, the controller performs control to present the guide.

According to the fourth aspect of the present invention, if the first image and the image related to the device that cannot perform the cooperation function together with the first device are superimposed on each other, the controller performs control to present the guide.

According to the fifth aspect of the present invention, if a partial image included in the first image is designated, the controller performs control to present the first image indicating that the cooperative function can be performed together with the function corresponding to the partial image. The guide of the two devices.

According to the sixth aspect of the present invention, as the control for presenting the guidance, the controller performs control to display a candidate list showing information on one or more second apparatuses capable of performing the cooperative function .

According to the seventh aspect of the present invention, if a second device is specified from among the one or more second devices on the candidate list, the controller performs control to display the information on using the specified second device Information on collaboration features.

According to the eighth aspect of the present invention, the controller performs control to display the cooperation function while changing the cooperation function according to the order in which the first device and the second device are designated.

According to the ninth aspect of the present invention, if the first device and the second device are designated, the controller further performs control to present a third device indicating that a cooperative function can be performed with the first device and the second device device guidance.

According to the tenth aspect of the present invention, the controller performs control to present the guidance while changing the third device according to the order in which the first device and the second device are designated.

According to an eleventh aspect of the present invention, there is provided an information processing apparatus including a controller. If the first image related to the first function required to execute the cooperative function is specified, the controller performs control to present a guide indicating a second function capable of executing the cooperative function together with the first function.

According to the twelfth aspect of the present invention, if an image related to a function that cannot perform the cooperation function together with the first function is further specified, the controller performs control to present the guidance.

According to the thirteenth aspect of the present invention, if an operation of linking a first image and an image related to a function that cannot perform the cooperation function together with the first function is performed to each other, the controller performs control to present all guide.

According to the fourteenth aspect of the present invention, if the first image and the image related to the function that cannot perform the cooperative function together with the first function are superimposed on each other, the controller performs control to present the guide .

According to the fifteenth aspect of the present invention, as the control for presenting the guidance, the controller performs control to display a candidate list showing information on one or more second functions capable of executing the cooperative function information.

According to the sixteenth aspect of the present invention, the order in which the one or more second functions are arranged in the candidate list is determined based on past usage records of the one or more second functions.

According to the seventeenth aspect of the present invention, the controller performs control to display the cooperative function while changing the cooperative function according to the order in which the first function and the second function are specified.

According to the eighteenth aspect of the present invention, if the first function and the second function are specified, the controller further performs control to present an indication that the first function and the second function can be performed together The bootstrap of the third function of the collaboration function.

According to the nineteenth aspect of the present invention, the controller performs control to present the guidance while changing the third function according to the order in which the first function and the second function are specified.

According to the twentieth aspect of the present invention, the first function and the second function are included in a pre-registered set of functions, a set of functions of one or more identified devices, a set of functions displayed on a display , or a set of functions displayed in a specific area of the screen of the display.

According to a twenty-first aspect of the present invention, there is provided an information processing method comprising: if a first image related to a first device required for performing a cooperation function is specified, performing control to present an indication that the instruction can be associated with The first device together performs the directing of the second device of the cooperative function.

According to a twenty-second aspect of the present invention, there is provided an information processing method comprising: if a first image related to a first function required for execution of a cooperative function is specified, performing control to present an indication that can interact with The first function together performs the directing of a second function of the cooperating function.

According to the first, ninth, and twenty-first aspects of the present invention, user convenience is increased in the case of performing a cooperative function.

According to the second or twelfth aspect of the present invention, the complications that may arise if the guidance is always presented can be avoided.

According to the third or thirteenth aspect of the present invention, control is performed to present guidance through an operation of linking images.

According to the fourth or fourteenth aspect of the present invention, control is performed to present guidance through an operation of superimposing images.

According to a fifth aspect of the present invention, control is performed to present a guide indicating a device capable of performing a cooperative function together with a specific function of the device.

According to the sixth or seventh aspect of the present invention, a list of apparatuses capable of performing the cooperation function is presented.

According to the eighth or seventeenth aspect of the present invention, control is performed to display the cooperative function estimated to be used.

According to a tenth aspect of the present invention, control is performed to present guidance indicating a device estimated to be used.

According to the eleventh, eighteenth, twentieth or twenty-second aspect of the present invention, in the case of specifying a function required to execute the cooperative function, user convenience is increased.

According to the fifteenth aspect of the present invention, a list of functions capable of executing the cooperative function is presented.

According to the sixteenth aspect of the present invention, it becomes easy to grasp the records of the relative use of the respective devices.

According to the nineteenth aspect of the present invention, control is performed to present guidance indicating a function estimated to be used.

Description of drawings

Exemplary embodiments of the present invention will be described in detail based on the following drawings, in which:

1 is a block diagram illustrating an apparatus system according to an exemplary embodiment of the present invention;

FIG. 2 is a block diagram illustrating an image forming apparatus according to an exemplary embodiment;

3 is a block diagram illustrating a server according to an exemplary embodiment;

4 is a block diagram illustrating a terminal device according to an exemplary embodiment;

5 is a schematic diagram showing the appearance of the image forming apparatus;

6 is a diagram showing an example of a device function management table;

7 is a diagram showing an example of a cooperation function management table;

Figure 8 is a diagram showing the device used alone;

9 is a diagram showing an example of a function display screen;

10 is a diagram showing an example of a function display screen;

11 is a diagram illustrating target devices cooperating with each other;

12 is a diagram showing an example of a function display screen;

13 is a sequence diagram showing connection processing;

14A and 14B are diagrams showing examples of device display screens;

15 is a diagram showing an example of a device display screen according to Example 1;

16 is a diagram showing an example of a device display screen according to Example 2;

17 is a diagram showing an example of a device display screen according to Example 3;

18 is a diagram showing an example of a device display screen according to Example 3;

19 is a diagram showing an example of a device display screen according to Example 3;

20 is a diagram showing an example of a device display screen according to Example 3;

21 is a diagram showing an example of a device display screen according to Example 4;

22 is a diagram showing an example of a device display screen according to Example 4;

23 is a diagram showing an example of a device display screen according to Example 5;

24 is a diagram showing an example of a device display screen according to Example 5;

25 is a diagram showing an example of a screen according to Example 5;

26 is a diagram showing an example of a device selection screen according to Example 6;

27 is a diagram showing an example of a device selection screen according to Example 7;

28 is a diagram showing an example of a device selection screen according to Example 7;

29 is a diagram showing an example of a function selection screen according to Example 7;

30 is a diagram showing an example of a screen according to Example 7;

31 is a diagram illustrating an example of a message according to Example 7;

32 is a diagram illustrating an example of a message according to Example 7;

33 is a diagram showing an example of a device selection screen according to Example 8;

34 is a diagram showing an example of a device selection screen according to Example 8;

35 is a diagram showing an example of a screen according to Example 8;

36 is a diagram showing an example of a cooperation function management table;

37A and 37B are diagrams showing an example of a device display screen and an example of a function display screen, respectively;

38A and 38B are diagrams showing an example of a device display screen and an example of a function display screen, respectively;

39 is a diagram showing an example of a device function management table;

40A and 40B are diagrams showing an example of a device display screen and an example of a function display screen, respectively;

41 is a diagram showing an example of a device function management table;

42 is a diagram showing an example of a cooperation function management table;

43A, 43B, and 43C are diagrams showing examples of screens displayed on the terminal device;

44A and 44B are diagrams showing examples of screens displayed on the terminal device; and

45A and 45B are diagrams showing examples of screens displayed on the terminal device.

Detailed ways

An apparatus system serving as an information processing system according to an exemplary embodiment of the present invention will be described with reference to FIG. 1 . FIG. 1 shows an example of a device system according to an exemplary embodiment.

The apparatus system according to the exemplary embodiment includes a plurality of apparatuses (eg, apparatuses 10 and 12 ), a server 14 which is an example of an external apparatus, and a terminal apparatus 16 which is an example of an information processing apparatus. The apparatuses 10 and 12, the server 14, and the terminal device 16 have a function of communicating with each other through a communication path N such as a network. Of course, the apparatuses 10 and 12, the server 14, and the terminal device 16 may communicate with another device through different communication paths. In the example shown in Figure 1, two devices (devices 10 and 12) are included in the device system. Three or more devices may be included in a device system. In addition, a plurality of servers 14 and a plurality of terminal devices 16 may be included in the apparatus system.

The apparatuses 10 and 12 are devices having a specific function and may be, for example, an image forming device having an image forming function, a personal computer (PC), a projector, a display device such as a liquid crystal display or a projector, a telephone, a clock, a surveillance camera, and the like. The apparatuses 10 and 12 have functions to transmit and receive data to and from another device. In the exemplary embodiment, for example, it is assumed that the apparatus 10 is an image forming apparatus. The image forming apparatus (apparatus 10 ) is an apparatus having at least one of a scan function, a print function, a copy function, and a facsimile function.

The server 14 is a device that manages the functions of each device. For example, the server 14 manages the function of each device, a cooperative function using a plurality of functions, and the like. The server 14 also has the function of sending data to and receiving data from another device.

Server 14 may manage, for each user, one or more functions available to that user. For example, the function available to the user is a function provided to the user for free or a function provided to the user for a fee and purchased by the user. The server 14 may manage, for each user, available function information (eg, function purchase history information) representing one or more functions available to the user. Of course, the server 14 does not necessarily manage functions according to whether the functions have been purchased, since there are free functions, additional update functions, and special functions managed by the administrator. The function purchase process is performed by, for example, the server 14 . Of course, the function purchase process may be performed by another device.

The terminal device 16 is a device such as a PC, a tablet PC, a smart phone, or a mobile phone, and has a function of transmitting and receiving data to and from another device. When the apparatus is used, the terminal device 16 serves as, for example, a user interface unit (UI unit).

In the device system according to the exemplary embodiment, if the first image related to the first device is designated, control is performed to present a guide indicating a second device capable of performing a cooperative function together with the first device. Alternatively, if the first image related to the first function is specified, control may be performed to present guidance indicating a second function capable of performing the cooperative function together with the first function.

Hereinafter, each device included in the apparatus system according to the exemplary embodiment will be described in detail.

The configuration of the apparatus 10 as an image forming apparatus will be described in detail with reference to FIG. 2 . Hereinafter, the apparatus 10 may be referred to as an image forming apparatus 10 . FIG. 2 shows the configuration of the image forming apparatus 10 .

The communication unit 18 is a communication interface, and has a function of transmitting data to and receiving data from another device. The communication unit 18 may be a communication interface having a wireless communication function, or may be a communication interface having a wired communication function.

The image forming unit 20 has an image forming function. Specifically, the image forming unit 20 has at least one of a scan function, a print function, a copy function, and a facsimile function. When the scan function is performed, the document is read and scan data (image data) is generated. When the print function is executed, an image is printed on a recording medium such as paper. When the copy function is performed, the document is read and printed on the recording medium. When the facsimile function is executed, image data is sent or received by facsimile. In addition, a cooperative function using a plurality of functions can be performed. For example, a scan-and-transfer function that is a combination of a scan function and a transfer (transfer) function can be performed. When the scan and transfer function is performed, a document is read, scan data (image data) is generated, and the scan data is transmitted to a destination (for example, an external device such as the terminal device 16). Of course, this collaboration function is just an example, and another collaboration function can be performed.

The memory 22 is a storage device such as a hard disk or a memory (eg, a solid state drive (SSD), etc.). The memory 22 stores, for example, information representing an image forming instruction (for example, job information), image data to be printed, scan data generated by executing a scan function, device address information representing the address of another device, data representing the address of the server 14 . Server address information, various control data, and various programs. Of course, these information and data may be stored in different storage devices or in one storage device.

The UI unit 24 is a user interface unit, and includes a display and an operation unit. The display is a display device such as a liquid crystal display. The operating unit is an input device such as a touch screen or a keyboard. Of course, a user interface (eg, a touch display or a device including a display that electronically displays a keyboard or the like) may be used that serves as both a display and an operating unit. The image forming apparatus 10 does not necessarily include the UI unit 24, and may include a hardware user interface unit (hardware UI unit) functioning as hardware instead of a display. For example, the hardware UI unit is a hardware keypad dedicated to entering numbers (eg, a numeric keypad) or a hardware keypad dedicated to indicating directions (eg, a direction indicating keypad).

The controller 26 controls the operations of the respective units of the image forming apparatus 10 .

Hereinafter, the configuration of the server 14 will be described in detail with reference to FIG. 3 . FIG. 3 shows the configuration of the server 14 .

The communication unit 28 is a communication interface, and has a function of transmitting data to and receiving data from another device. The communication unit 28 may be a communication interface having a wireless communication function, or may be a communication interface having a wired communication function.

The memory 30 is a storage device such as a hard disk or a memory (eg, SSD, etc.). The memory 30 stores, for example, device function management information 32 , cooperation function management information 34 , various data, various programs, device address information indicating the address of each device, and server address information indicating the address of the server 14 . Of course, these information and data may be stored in different storage devices or in one storage device. The device function management information 32 and the cooperative function management information 34 stored in the memory 30 may be provided to the terminal device 16 periodically or at a designated timing, and thus, the information stored in the terminal device 16 may be updated. Hereinafter, the device function management information 32 and the cooperation function management information 34 will be described.

The device function management information 32 is information for managing the function of each device, and is, for example, between device identification information for identifying the device and one or more pieces of function information indicating one or more functions of the device Correspondence information. The device identification information includes, for example, a device ID, a device name, information indicating the type of the device, the model of the device, information indicating the location of the device (device location information), and an appearance image indicating the appearance of the device. The function information includes, for example, a function ID and a function name. For example, if the image forming apparatus 10 has a scan function, a print function, a copy function, and a scan and transfer function, the device identification information of the image forming apparatus 10 and the function information indicating the scan function, the function information indicating the print function, the The function information is associated with the function information representing the scan-and-delivery function. By referring to the device function management information 32 to determine (identify) the function of each device,

For example, the devices managed by the device function management information 32 are devices (eg, devices 10 and 12 ) included in the device system. Of course, devices not included in the device system can also be managed by the device function management information 32 . For example, the server 14 may obtain information on a new device not included in the device system (information including device identification information and function information) and may newly register the information in the device function management information 32 . The information on the device may be obtained using the Internet or the like, or may be input by an administrator or the like. The server 14 may update the device function management information 32 at a certain timing, periodically, or at a timing designated by an administrator or the like. Therefore, function information indicating functions that the device before the update does not have and the device has after the update can be registered in the device function management information 32 . In addition, function information indicating functions that the device has before updating and which the device does not have after updating may be deleted from the device function management information 32 and may be registered as unavailable information. Information for updating may be obtained using the Internet or the like, or may be input by an administrator or the like.

The cooperation function management information 34 is information for managing the cooperation functions, each of which is executed by cooperation among a plurality of functions. One or more collaborative functions are performed through cooperation between multiple functions. The various cooperative functions may be performed by cooperation between multiple functions of one device (eg, device 10 or 12 ), or may be performed by cooperation between multiple functions of multiple devices (eg, devices 10 and 12 ) . A terminal device (in an exemplary embodiment, terminal device 16 ) that provides the operation instructions may be included in the apparatus to be identified, and the function of the terminal device may be used as part of the collaborative function.

Collaborative functions may be functions that are performed without the use of hardware devices. For example, a collaborative function may be a function performed through cooperation between multiple software elements. Of course, the cooperative function may be a function performed by cooperation between a function of a hardware device and a function realized by software.

For example, the cooperation function management information 34 is information that represents the correspondence between the combination of function information representing each function used in the cooperation function and the cooperation function information representing the cooperation function. For example, the collaboration function information includes a collaboration function ID and a collaboration function name. If a single function is updated, the cooperative function management information 34 is also updated according to the update. Therefore, cooperation functions using a plurality of functions that cannot cooperate with each other before the update may become available after the update, or cooperation functions that were available before the update may become unavailable after the update. The collaboration function information indicating the collaboration function that becomes available after the update is registered in the collaboration function management information 34, and the collaboration function information indicating the collaboration function that becomes unavailable after the update is deleted from the collaboration function management information 34 or registered as unavailable. use information.

In the case of causing a plurality of apparatuses to cooperate with each other, the cooperative function management information 34 is information for managing one or more cooperative functions that use the plurality of functions of the plurality of apparatuses, and is information indicating identification for the one Information on the correspondence between the combination of the device identification information of each device of the cooperation function or the cooperation function information. If the device function management information 32 is updated, the cooperation function management information 34 is also updated according to the update. Therefore, a cooperation function using a plurality of devices that cannot cooperate with each other before the update may become available, or the cooperation function available before the update may become unavailable after the update.

The cooperative function may be a function performed by cooperation between a plurality of different functions, or may be a function performed by cooperation between the same functions. Collaborative functions may be functions that are not available without collaboration. A function that is unavailable without cooperation may be a function that becomes available by combining the same function among the functions of the target device that cooperate with each other, or a function that becomes available by combining different functions among the functions of the target device that cooperate with each other available functions. For example, cooperation between a device having a printing function (printer) and a device having a scanning function (scanner) realizes a copy function as a cooperation function. That is, the cooperation between the print function and the scan function realizes the copy function. In this case, the copy function as the cooperative function is associated with the combination of the print function and the scan function. In the cooperation function management information 34, for example, cooperation function information indicating a copy function as a cooperation function is associated with a combination of device identification information for identifying a device with a print function and device identification information for identifying a device with a scan function .

The memory 30 may store available function management information. The available function management information is information for managing one or more functions available to each user, and is, for example, user identification information for identifying a user and one or more functions representing one or more functions available to the user Correspondence information between pieces of function information (which may include collaboration function information). As described above, for example, the functions available to the user are the functions provided to the user for free or the functions purchased by the user, and may be a single function or a cooperative function. For example, the user identification information is user account information such as user ID and user name. The functions available to each user are determined (identified) by referring to the available function management information. The available function management information is updated every time a function is provided to a user (eg, every time a function is provided to a user for a fee or for free).

The controller 36 controls the operation of the various elements of the server 14 . The controller 36 includes a determination unit 38 .

The determination unit 38 receives device identification information for identifying a device and determines one or more pieces of function information representing one or more functions associated with the device identification information in the device function management information 32 stored in the memory 30 . Thus, one or more functions of the device are determined (identified). For example, device identification information is transmitted from the terminal device 16 to the server 14, and then the determination unit 38 determines one or more pieces of function information representing one or more functions associated with the device identification information. For example, information about the one or more functions (eg, function information and function specification information) is transmitted from the server 14 to the terminal device 16 and displayed on the terminal device 16 . Accordingly, information on one or more functions of the device determined by the device identification information is displayed on the terminal device 16 .

Further, the determination unit 38 receives a plurality of pieces of device identification information for identifying target devices that cooperate with each other, and determines, among the cooperative function management information 34 stored in the memory 30, a representation that is associated with a combination of the plurality of pieces of device identification information. One or more pieces of collaboration function information for one or more collaboration functions. Thus, one or more cooperative functions performed through the cooperation between the functions of the target device cooperating with each other are determined (identified). For example, pieces of device identification information are sent from the terminal device 16 to the server 14, and then the determination unit 38 determines one or more pieces of cooperation function information representing one or more cooperation functions associated with the pieces of device identification information. For example, information on the one or more cooperative functions (eg, cooperative function information and cooperative function specification information) is transmitted from the server 14 to the terminal device 16 and displayed on the terminal device 16 . Accordingly, information on one or more cooperative functions performed by the plurality of devices determined by the plurality of pieces of device identification information is displayed on the terminal device 16 .

For example, if a device is identified (eg, if a device is photographed), the determination unit 38 may receive device identification information for identifying the device, and may determine in the device function management information 32 the device identification information associated with the device One or more pieces of feature information representing one or more features. Thus, if a device is identified (eg, if a device is photographed), one or more functions of the device are determined (identified). If a plurality of devices are identified (for example, if a plurality of devices are photographed), the determination unit 38 may receive a plurality of pieces of device identification information for identifying respective devices included in the plurality of devices, and may manage in the cooperation function One or more pieces of cooperative function information representing one or more cooperative functions associated with the combination of the plurality of pieces of device identification information are determined in the information 34 . Thus, if multiple devices are identified (eg, if multiple devices are photographed), one or more cooperative functions using the functions of the multiple devices are determined (identified).

The determination unit 38 may receive a plurality of pieces of function information representing respective functions used in the cooperative function, and may determine, in the cooperative function management information 34 stored in the memory 30, a piece of function information representing one or more associated with the combination of the plurality of pieces of function information. One or more pieces of collaboration function information for more collaboration functions. Thus, one or more cooperative functions performed through the cooperation between the target functions are determined (identified). For example, a plurality of pieces of function information are transmitted from the terminal device 16 to the server 14, and then the determination unit 38 determines one or more pieces of cooperation function information representing one or more cooperation functions associated with the pieces of function information. In a manner similar to that described above, information on one or more cooperative functions performed by the plurality of functions determined by the plurality of pieces of function information is displayed on the terminal device 16 .

If functions available to the user are managed, the determination unit 38 may receive user identification information for identifying the user, and may determine function information representing respective functions associated with the user identification information among the available function management information stored in the memory 30 . Thus, a set of functions available to the user is determined (identified). For example, the user identification information is transmitted from the terminal device 16 to the server 14, and the function information representing each function associated with the user identification information is determined by the determination unit 38. For example, information on the respective functions available to the user (eg, information indicating the names of the respective functions) is transmitted from the server 14 to the terminal device 16 and displayed on the terminal device 16 . Therefore, information on the respective functions available to the user determined by the user identification information is displayed on the terminal device 16 . For example, the determination unit 38 receives the device identification information and the user identification information, determines in the device function management information 32 one or more pieces of function information representing one or more functions associated with the device identification information, and also available functions One or more pieces of function information representing one or more functions associated with the user identification information are determined in the management information. Accordingly, one or more functions that the device identified by the device identification information has and are available to the user identified by the user identification information are determined.

The controller 36 may perform function purchase processing and may manage purchase history. For example, if the user purchases a paid feature, the controller 36 may apply a charging process to the user.

The controller 36 can perform functions related to image processing, such as a character recognition function, a translation function, an image processing function, and an image forming function. Of course, the controller 36 may perform functions related to processing other than image processing. When the character recognition function is performed, characters in the image are recognized and character data representing the characters are generated. When the translation function is executed, the characters in the image are translated into characters represented by a specific language, and character data representing the translated characters are generated. When performing image processing functions, images are processed. For example, the controller 36 may receive scan data generated by execution of the scan function from the image forming apparatus 10, and may perform an image processing-related function, such as a character recognition function, a translation function, or an image processing function, on the scan data. The controller 36 may receive image data from the terminal device 16 and perform various functions on the image data. For example, character data or image data generated by the controller 36 is transmitted from the server 14 to the terminal device 16 . The server 14 may be used as an external device, and the cooperative function may be a function using the functions of a plurality of devices including the server 14 .

Hereinafter, the configuration of the terminal device 16 will be described in detail with reference to FIG. 4 . FIG. 4 shows the configuration of the terminal device 16 .

The communication unit 40 is a communication interface, and has a function of transmitting data to and receiving data from another device. The communication unit 40 may be a communication interface having a wireless communication function, or may be a communication interface having a wired communication function.

The camera 42 serving as a photographing unit photographs a subject, thereby generating image data (eg, still image data or moving image data). Alternatively, instead of using the camera 42 of the terminal device 16, image data captured by an external camera connected to a communication path such as a network may be received by the communication unit 40, and may be displayed on the UI unit 46 so that the image data may be accessed by the communication unit 40. User action.

The memory 44 is a storage device such as a hard disk or a memory (eg, SSD, etc.). The memory 44 stores various programs, various data, address information of the server 14, address information of the respective devices (eg, address information of the devices 10 and 12), information on the identified devices, information on the identified target devices that cooperate with each other. information, information about the function of the identified device, and information about the collaboration function.

The UI unit 46 is a user interface unit, and includes a display and an operation unit. The display is a display device such as a liquid crystal display. The operating unit is an input device such as a touch screen, a keyboard or a mouse. Of course, a user interface (eg, a touch display or a device including a display that electronically displays a keyboard or the like) can be used that serves as both a display and an operating unit.

The controller 48 controls the operation of the various units of the terminal device 16 . The controller 48 functions as, for example, a display controller (controller) and causes the display of the UI unit 46 to display various information.

The display of UI unit 46 displays, for example, images captured by camera 42 , images related to devices identified as target devices to be used (eg, single use devices or cooperating target devices), functions related images, and the like. The device-related image may be an image (still image or moving image) representing the device captured by the camera 42, or may be an image (eg, an icon) that schematically represents the device. The data schematically representing the image of the apparatus may be stored in the server 14 and provided to the terminal device 16 from the server 14, may be pre-stored in the terminal device 16, or may be stored in another device and retrieved from the other device. A device is provided to the terminal device 16 . The image related to the function is, for example, an image representing an icon or the like of the function.

The above-described device function management information 32 may be stored in the memory 44 of the terminal device 16 . In this case, the device function management information 32 is not necessarily stored in the memory 30 of the server 14 . Furthermore, the above-mentioned cooperation function management information 34 may be stored in the memory 44 of the terminal device 16 . In this case, the cooperative function management information 34 is not necessarily stored in the memory 30 of the server 14 . The controller 48 of the terminal device 16 may include the above-described determination unit 38, which may determine one or more functions of the device by identifying the device based on the device identification information and may determine one or more cooperative functions using the plurality of functions. In this case, the server 14 does not necessarily include the determination unit 38.

If the available function management information is created, the available function management information may be stored in the memory 44 of the terminal device 16 . In this case, the available function management information is not necessarily stored in the memory 30 of the server 14 . The controller 48 of the terminal device 16 may manage the user's feature purchase history. In this case, the controller 36 of the server 14 does not necessarily have a management function therefor. The controller 48 of the terminal device 16 may determine one or more functions available to the user based on the user identification information.

Alternatively, the device function management information 32 and the cooperation function management information 34 may be stored in a device such as the device 10 or 12 , and the device such as the device 10 or 12 may include the determination unit 38 . That is, the processing by the determination unit 38 of the server 14 (for example, the processing of identifying the device, the function of identifying the function, or the processing of identifying the cooperative function) may be executed in the server 14, may be executed in the terminal device 16, or may be executed in a system such as Executed in device 10 or device 12.

In an exemplary embodiment, for example, augmented reality (AR) technology is applied to obtain device identification information and identify the device. For example, the AR technology is applied to obtain device identification information of a device used alone and identify the device, and also obtain device identification information of a target device that cooperates with each other and identify the target device. Use AR technology according to the state of the art. For example, marker-based AR technology using a marker such as a two-dimensional barcode, markerless AR technology using image recognition technology, location information AR technology using location information, and the like are used. Of course, device identification information can be obtained and the device can be identified without applying AR technology. For example, in the case of a device connected to a network, the device can be identified based on its IP address or by reading its device ID. In addition, in the case of a device or terminal device having various types of wireless communication functions based on infrared communication, visible light communication, Wireless Fidelity (Wi-Fi, registered trademark), or Bluetooth (registered trademark), the device can be obtained by obtaining The ID identifies devices that cooperate with each other using the wireless communication function, and can perform the cooperation function.

Hereinafter, the process of obtaining the device identification information will be described in detail with reference to FIG. 5 . As an example, the case of obtaining the device identification information of the image forming apparatus 10 will be described. FIG. 5 schematically shows the appearance of the image forming apparatus 10 . Here, a process of obtaining device identification information by applying the marker-based AR technology will be described. An indicia 50 such as a two-dimensional barcode is attached to the housing of the image forming apparatus 10 . The mark 50 is information obtained by encoding device identification information of the image forming apparatus 10 . The user activates the camera 42 of the terminal apparatus 16 and uses the camera 42 to photograph the mark 50 attached to the image forming apparatus 10 as the target to be used. Thus, image data representing the marker 50 is generated. For example, image data is sent from the terminal device 16 to the server 14 . In the server 14, the controller 36 performs decoding processing on the marker image represented by the image data, thereby extracting device identification information. Therefore, the image forming apparatus 10 to be used (the image forming apparatus 10 to which the photographed mark 50 is attached) is identified. The determination unit 38 of the server 14 determines the function information of the function associated with the extracted device identification information in the device function management information 32 . Therefore, the function of the image forming apparatus 10 to be used is determined (identified).

Alternatively, the controller 48 of the terminal device 16 may perform decoding processing on the image data representing the marker 50 to extract device identification information. In this case, the extracted device identification information is transmitted from the terminal device 16 to the server 14 . The determination unit 38 of the server 14 determines, in the device function management information 32 , function information indicating the function associated with the device identification information transmitted from the terminal device 16 . If the device function management information 32 is stored in the memory 44 of the terminal device 16, the controller 48 of the terminal device 16 may determine, in the device function management information 32, function information representing the function associated with the extracted device identification information.

The mark 50 may include encoded function information representing the function of the image forming apparatus 10 . In this case, by performing decoding processing on the image data representing the marker 50 , device identification information of the image forming apparatus 10 is extracted and function information representing the functions of the image forming apparatus 10 is also extracted. Therefore, the image forming apparatus 10 is determined (identified) and the function of the image forming apparatus 10 is also determined (identified). The decoding process may be performed by the server 14 or the terminal device 16 .

In the case of executing a cooperative function using the functions of a plurality of devices, the marks of target devices that cooperate with each other are photographed to obtain device identification information of the devices, thereby determining (identifying) the cooperative function.

In the case of obtaining the device identification information by applying the markerless AR technology, for example, the user photographs the entire exterior or partial appearance of the device to be used (eg, the image forming apparatus 10 ) with the camera 42 of the terminal device 16 . Of course, it is useful to obtain information for determining the device to be used, such as the device name (eg, trade name) or model, by photographing the appearance of the device. As a result of the photographing, appearance image data representing the entire appearance or part of the appearance of the device to be used is generated. For example, the appearance image data is transmitted from the terminal device 16 to the server 14 . In the server 14, the controller 36 identifies the device to be used based on the appearance image data. For example, the memory 30 of the server 14 stores appearance image correspondence information for each device, which represents the correspondence between the appearance image data representing the entire appearance or part of the appearance of the device and the device identification information of the device. The controller 36 compares, for example, the appearance image data received from the terminal device 16 with the respective pieces of appearance image data included in the appearance image correspondence information, and determines the device identification information of the device to be used based on the comparison result. For example, the controller 36 extracts the features of the appearance of the device to be used from the appearance image data received from the terminal device 16, and determines in the appearance image data group included in the appearance image correspondence information the features that represent the same or similar appearance as the appearance. appearance image data of the feature, and device identification information associated with the appearance image data is determined. Thus, the device to be used (the device photographed by the camera 42 ) is identified. As another example, if a name (eg, trade name) or model of a device is photographed and appearance image data representing the name or model is generated, the device to be used can be identified based on the name or model represented by the appearance image data. The determination unit 38 of the server 14 determines, in the device function management information 32, function information indicating each function associated with the determined device identification information. Therefore, the functions of the apparatus to be used (eg, the image forming apparatus 10 ) are determined.

Alternatively, the controller 48 of the terminal apparatus 16 may associate the appearance image data representing the entire appearance or a part of the appearance of the apparatus to be used (eg, the image forming apparatus 10 ) with the respective pieces of appearance image data included in the appearance image correspondence information A comparison is made, and the device identification information of the device to be used can be determined based on the comparison result. The appearance image correspondence information may be stored in the memory 44 of the terminal device 16 . In this case, the controller 48 of the terminal device 16 refers to the appearance image correspondence information stored in the memory 44 of the terminal device 16, thereby determining the device identification information of the device to be used. Alternatively, the controller 48 of the terminal device 16 may obtain the appearance image correspondence information from the server 14 and may refer to the appearance image correspondence information to determine the device identification information of the device to be used.

In the case of executing a cooperative function using a plurality of functions of a plurality of apparatuses, the entire appearance or a part of the appearance of each apparatus cooperating with each other is photographed to obtain apparatus identification information of the apparatus, thereby determining (identifying) the cooperative function.

In the case of obtaining the device identification information by applying the position information AR technology, for example, the position information indicating the position of the device (eg, the image forming apparatus 10 ) is obtained using a global positioning system (GPS) function. For example, each device has a GPS function and obtains device location information representing the location of the device. The terminal device 16 outputs information indicating a request to obtain device location information to the device to be used, and receives the device location information of the device from the device in response to the request. For example, the device position information is transmitted from the terminal device 16 to the server 14 . In the server 14, the controller 36 identifies the device to be used based on the device location information. For example, the memory 30 of the server 14 stores location correspondence information for each device, which indicates the correspondence between the device location information indicating the location of the device and the device identification information of the device. The controller 36 determines device identification information associated with the device location information received from the terminal device 16 in the location correspondence information. Therefore, the device to be used is determined (identified). The determination unit 38 of the server 14 determines, in the device function management information 32, function information indicating each function associated with the determined device identification information. Therefore, the function of the apparatus to be used (for example, the image forming apparatus 10 ) is determined (identified).

The controller 48 of the terminal device 16 may determine device identification information associated with the location information of the device to be used in the location correspondence information. The location correspondence information may be stored in the memory 44 of the terminal device 16 . In this case, the controller 48 of the terminal device 16 refers to the position correspondence information stored in the memory 44 of the terminal device 16, thereby determining the device identification information of the device to be used. Alternatively, the controller 48 of the terminal device 16 may obtain the location correspondence information from the server 14 and refer to the location correspondence information to determine the device identification information of the device to be used.

In the case of executing a cooperation function using a plurality of devices, device location information of devices that cooperate with each other is obtained and device identification information of the devices is determined based on the device location information. Therefore, the cooperative function is determined (identified).

Hereinafter, the device system according to the exemplary embodiment will be described in further detail.

The device function management information 32 will be described in detail with reference to FIG. 6 . FIG. 6 shows an example of a device function management table as the device function management information 32 . In the device function management table, for example, a device ID, information representing a device name (eg, device type), information (function information) representing one or more functions of the device, and an image ID are associated with each other. The device ID and the device name correspond to examples of device identification information. The image ID is an example of image identification information for identifying an image representing the device (eg, an image representing the appearance of the device or an image (eg, an icon) schematically representing the device). The device function management table does not necessarily include the image ID. For example, the device having the device ID "B" is a multi-function peripheral (MFP, an image forming apparatus having a plurality of image forming functions) and has a printing function, a scanning function, and the like. An image ID for identifying an image representing a device is associated with the device. The data representing the image of the device is stored, for example, in the memory 30 of the server 14 or another device.

For example, with AR technology, a device ID for identifying a device to be used is obtained. The determination unit 38 of the server 14 determines the name of the device, one or more functions of the device, and the image ID associated with the device ID by referring to the device function management table. Thus, the device to be used is identified. For example, information representing the name of the device and data representing the image of the device are transmitted from the server 14 to the terminal device 16 , and then they are displayed on the UI unit 46 of the terminal device 16 . The image representing the device is displayed as an image related to the device. Of course, the image captured by the camera 42 may be displayed on the UI unit 46 of the terminal device 16 . If the user specifies an image related to the device on the UI unit 46 of the terminal device 16 (eg an image captured by the camera 42 or an image schematically representing the device), information about one or more functions of the device (eg , function information or function description information) may be sent from the server 14 to the terminal device 16 and may be displayed on the UI unit 46 of the terminal device 16 .

Next, the cooperative function management information 34 will be described in detail with reference to FIG. 7 . FIG. 7 shows the cooperation function management information 34 as an example of the cooperation function management table. In the cooperation function management table, for example, combinations of device IDs, information representing names (types) of target devices that cooperate with each other, and information representing one or more cooperation functions (cooperation function information) are associated with each other. For example, a device with device ID "A" is a personal computer (PC), and a device with device ID "B" is an MFP. The cooperation between the PC (A) and the MFP (B) realizes, for example, a scan and transfer function and a print function as a cooperation function. The scan and transfer function is a function to transfer the image data generated by the scanning of the MFP (B) to the PC (A). The print function is a function of sending data (eg, image data or document data) stored in the PC (A) to the MFP (B) and printing the data through the MFP (B).

Hereinafter, the processing in the case of using the device alone will be described with reference to FIG. 8 . Figure 8 shows an example of a device used alone. For example, it is assumed that the image forming apparatus 10 is a device used alone. The image forming apparatus 10 is, for example, an MFP. The image forming apparatus 10 is a device existing in a real space. The terminal device 16 shown in FIG. 8 is a device existing in a real space, and is, for example, a mobile terminal device such as a smartphone or a mobile phone.

For example, the housing of the image forming apparatus 10 is attached with a mark 50 such as a two-dimensional barcode. In the case of using the marker-based AR technology or the markerless AR technology, the user photographs the image forming device 10 to be used with the camera 42 of the terminal device 16 (eg, smartphone). Thus, image data representing the mark 50 or appearance image data representing the appearance of the image forming apparatus 10 is generated. A device display screen 52 is displayed on the display of the UI unit 46 of the terminal apparatus 16 , and a device image 54 related to the image forming apparatus 10 is displayed on the device display screen 52 . The device image 54 is, for example, an image generated by photographing by the camera 42 (having an original size at the time of photographing or an enlarged or reduced size).

Image data generated by photographing by the camera 42 is transmitted from the terminal device 16 to the server 14 . In the server 14 , the controller 36 performs decoding processing on the image data to extract the device identification information of the image forming apparatus 10 , and thus, recognizes the image forming apparatus 10 . Alternatively, appearance image data representing the appearance of the image forming apparatus 10 may be generated, and the appearance image data may be transmitted from the terminal apparatus 16 to the server 14 . In this case, in the server 14, the controller 36 determines the device identification information of the image forming apparatus 10 by referring to the appearance image correspondence information. Thus, the image forming apparatus 10 is recognized.

The determination unit 38 of the server 14 determines (identifies) the function of the image forming apparatus 10 by referring to the device function management information 32 (eg, the device function management table shown in FIG. 6 ). This will be described in detail with reference to FIG. 6 . For example, it is assumed that the image forming apparatus 10 is "MFP(B)". The determination unit 38 determines the function associated with the MFP (B) in the device function management table shown in FIG. 6 . Therefore, the function of MFP(B) is determined. Information about the determined functions is sent from the server 14 to the terminal device 16 . Of course, the processing for identifying devices and functions may be performed by the terminal device 16 .

On the device display screen 52 , instead of the image generated by shooting by the camera 42 , a preliminary image (not an image obtained by shooting, but a schematic image (eg, an icon)) related to the identified image forming apparatus 10 or a schematic image (eg, an icon) by An image captured by an external camera can be displayed as the device image 54 .

For example, in the case of using image data obtained by photographing the device, the appearance of the device in the current state (for example, including the appearance of scratches, notes, stickers attached to the device, etc.) is reflected in the image, and thus, the user can Differences from another device of the same type are more clearly recognized visually.

In the case of using a schematic image, for example, data of the schematic image is transmitted from the server 14 to the terminal device 16 . For example, when the image forming apparatus 10 is identified, the determination unit 38 of the server 14 determines a schematic image related to the image forming apparatus 10 by referring to the device function management table (device function management information 32 ) shown in FIG. 6 . The data of the schematic image is transmitted from the server 14 to the terminal device 16 , and the schematic image is displayed on the device display screen 52 as the device image 54 . The data of the schematic image may be stored in the terminal device 16 in advance. In this case, when the image forming apparatus 10 is recognized, the device image 54 stored in the terminal device 16 is displayed on the device display screen 52 . The data of the schematic image may be stored in a device other than the server 14 and the terminal device 16 .

Further, when the device is identified, information representing the device name may be transmitted from the server 14 to the terminal device 16 , and the device name may be displayed on the device display screen 52 in the terminal device 16 . In the example shown in FIG. 8, the image forming apparatus 10 is an MFP, and its name "MFP(B)" is displayed.

After determining the function of the image forming apparatus 10, the controller 48 of the terminal apparatus 16 causes the UI unit 46 of the terminal apparatus 16 to display the function display screen 56, and causes information about the function to be displayed on the function display screen 56, as shown in FIG. 9 . As the information on the function, for example, a button image for providing an instruction to execute the function is displayed. The MFP (B) as the image forming apparatus 10 has, for example, a print function, a scan function, a copy function, and a facsimile function, and therefore, button images for providing instructions to execute these functions are displayed on the function display screen 56 . For example, when the user designates a button image representing a print function using the terminal device 16 and provides an instruction to execute the print function, execution instruction information representing the instruction to execute the print function is transmitted from the terminal device 16 to the image forming apparatus 10 . The execution instruction information includes control data for executing the print function, data such as image data to which the print function is applied, and the like. In response to the reception of the execution instruction information, the image forming apparatus 10 executes printing according to the execution instruction information.

FIG. 10 shows another example of the function display screen. The function display screen 58 is a screen displayed on the UI unit 46 of the terminal device 16 when a single device is used as shown in FIG. 8 . As described above, the device to be used (eg, the image forming apparatus 10 ) is determined and the function of the device to be used is determined. Function information representing functions associated with user identification information of a user using the target device (ie, functions available to the user) may be determined. In addition, since the functions of the device to be used are determined, it is possible to determine functions that the device to be used does not have among a set of functions to be provided. Such information can be displayed on the function display screen 58 .

On the function display screen 58 shown in FIG. 10 , a button image 60 representing function A, a button image 62 representing function B, and a button image 64 representing function C are displayed as examples of function information. The function A is a function of the apparatus to be used (eg, the recognized image forming apparatus 10 ) and is a function available to the user. Function B is a function of the device to be used and is not available to the user. By providing the function B to the user, the user becomes able to use the function B. If the function B is a paid function, by purchasing the function B, the user becomes able to use the function B. If the function B is a free function, the user becomes able to use the function B by providing the function B free of charge to the user. The function C is a function that the device to be used does not have, that is, a function that is not compatible with the device to be used. The controller 48 of the terminal device 16 may change the display form of the button image according to whether the function represented by the button image is the function of the device to be used. Further, the controller 48 may change the display form of the button image according to whether the function represented by the button image is a function available to the user. For example, the controller 48 may change the color or shape of each button image. In the example shown in FIG. 10, button images 60, 62 and 64 are displayed in different colors. For example, a button image representing a function that the device to be used has and is available to the user (eg, the button image 60 representing function A) is displayed in blue. Button images representing functions that are available to the device to be used and that are not available to the user (eg, button image 62 representing function B) are displayed in yellow. A button image representing a function that the device to be used does not have (for example, the button image 64 representing function C) is displayed in gray. Alternatively, the controller 48 may change the shape of the button images 60, 62 and 64, or may change the font of the function display name. Of course, the display form can be changed according to another method. Therefore, users can identify the availability of individual functions with enhanced visibility.

For example, if the user designates the button image 60 representing the function A with the terminal device 16 and provides an instruction to execute the function A, execution instruction information representing the instruction to execute the function A is transmitted from the terminal device 16 to the target device to be used. The execution instruction information includes control data for executing the function A, image data to be subjected to the processing of the function A, and the like. In response to receiving the execution instruction information, the target device executes the function A according to the execution instruction information. For example, if the target device is the image forming apparatus 10 and if the function A is the scan and transfer function, the image forming unit 20 of the image forming apparatus 10 performs the scan function to generate scan data (image data). Then, the scan data is transmitted from the image forming apparatus 10 to the set destination (for example, the terminal apparatus 16).

If the user designates the button image 62 representing the function B using the terminal device 16 and provides an instruction to execute the function B, the providing process is executed. If the providing process is performed by the server 14 , the terminal device 16 accesses the server 14 . Therefore, as information enabling the user to use the function B, a screen (eg, a website) for providing the function B is displayed on the UI unit 46 of the terminal device 16 . The user becomes able to use the function B by performing the provisioning process on this screen. For example, the terminal device 16 stores a program of a web browser. Using this web browser, the user can access the server 14 from the terminal device 16 . When the user accesses the server 14 using a web browser, a function providing screen (eg, a website) is displayed on the UI unit 46 of the terminal device 16, and functions are provided to the user through the website. Of course, the providing process may be performed by a server other than the server 14 . Alternatively, as information enabling the user to use the function B, a usage permission request screen (eg, website) for requesting the administrator or the like to use the function B may be displayed on the UI unit 46 of the terminal device 16 . If the user requests permission to use the function B from the administrator or the like through the use permission request screen and if the permission is obtained, the user can use the function B.

Hereinafter, the processing in the case of using the cooperation function will be described with reference to FIG. 11 . FIG. 11 shows an example of target devices cooperating with each other. For example, it is assumed that the image forming apparatus 10 and a projector as the device 12 (may be referred to as the projector 12 hereinafter) are used as target devices. The image forming apparatus 10, the projector 12, and the terminal apparatus 16 are devices that exist in a real space.

For example, indicia 50 , such as a two-dimensional barcode, is attached to the housing of image forming apparatus 10 , and indicia 66 , such as a two-dimensional barcode, is attached to the housing of projector 12 . The mark 66 is information obtained by encoding the device identification information of the projector 12 . If marker-based AR technology or markerless AR technology is used, the user photographs the image forming device 10 and the projector 12 as target devices cooperating with each other with the camera 42 of the terminal device 16 (eg, smartphone). In the example shown in FIG. 11 , the image forming apparatus 10 and the projector 12 are photographed together in a state where both of the image forming apparatus 10 and the projector 12 are within the field of view of the camera 42 . Thus, image data representing markers 50 and 66 are generated. The device display screen 68 is displayed on the display of the UI unit 46 of the terminal device 16 . On the device display screen 68 , a device image 70 related to the image forming apparatus 10 and a device image 72 related to the projector 12 are displayed. The device images 70 and 72 are images generated by, for example, photographing by the camera 42 (having the original size at the time of photographing or an enlarged or reduced size).

Image data generated by photographing by the camera 42 is transmitted from the terminal device 16 to the server 14 . In the server 14 , the controller 36 performs decoding processing on the image data to extract the device identification information of the image forming apparatus 10 and the device identification information of the projector 12 , and thus, the image forming apparatus 10 and the projector 12 are identified. Alternatively, appearance image data representing the appearance of the image forming apparatus 10 and the projector 12 may be generated and transmitted from the terminal apparatus 16 to the server 14 . In this case, in the server 14 , the controller 36 determines the device identification information of the image forming apparatus 10 and the device identification information of the projector 12 by referring to the appearance image correspondence information. Thus, the image forming apparatus 10 and the projector 12 are recognized.

The determination unit 38 of the server 14 determines (identifies) use of one or more of the functions of the image forming apparatus 10 and the functions of the projector 12 by referring to the cooperation function management information 34 (for example, the cooperation function management table shown in FIG. 7 ). Collaboration features. This will be described in detail with reference to FIG. 7 . For example, it is assumed that the image forming apparatus 10 is an MFP (B) and the projector 12 is a projector (C). The determination unit 38 determines the cooperation function associated with the combination of the MFP (B) and the projector (C) in the cooperation function management table shown in FIG. 7 . Therefore, the cooperative function performed by the cooperation between the MFP (B) and the projector (C) is determined. Information about the determined cooperation function is sent from the server 14 to the terminal device 16 . Of course, the processing for identifying the device and the cooperation function may be performed by the terminal device 16 .

On the device display screen 68 , in place of the image generated by photographing by the camera 42 , a preliminary image (eg, a schematic image such as an icon) related to the identified image forming apparatus 10 or an image generated by photographing by an external camera may be is displayed as device image 70 . Also, a preliminary image related to the identified projector 12 or an image generated by photographing by an external camera may be displayed as the device image 72 . As described above, the data of the schematic image may be transmitted from the server 14 to the terminal device 16, may be pre-stored in the terminal device 16, or may be stored in another device.

When the device is identified, information representing the device name may be transmitted from the server 14 to the terminal device 16 , and the device name may be displayed on the device display screen 68 in the terminal device 16 . In the example shown in FIG. 11 , the name “MFP (B)” of the image forming apparatus 10 and the name “Projector (C)” of the projector 12 are displayed.

If a plurality of devices are photographed, the determination unit 38 of the server 14 may determine the function of each device by referring to the device function management information 32 . In the example shown in FIG. 11 , the determination unit 38 can determine the function of the image forming apparatus 10 and the function of the projector 12 . Information about the determined functions can be sent from the server 14 to the terminal device 16 .

After determining the cooperation function, the controller 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display the function display screen 74 and causes information about the cooperation function to be displayed on the function display screen 74 as shown in FIG. 12 . As the information on the cooperative function, for example, a button image for providing an instruction to execute the cooperative function is displayed. The cooperation between the MFP (B) and the projector (C) enables the execution of a cooperative function of projecting by the projector (C) an image generated by scanning by the MFP (B) and printing by the MFP (B) the projector (C) Collaborative capabilities of projected images. Button images for providing instructions to execute these cooperative functions are displayed on the function display screen 74 . For example, when the user designates a button image using the terminal device 16 and provides an instruction to execute the cooperative function, execution instruction information indicating the instruction to execute the cooperative function is transmitted from the terminal device 16 to the image forming apparatus 10 and the projector 12 . In response to the reception of the execution instruction information, the image forming apparatus 10 and the projector 12 execute the cooperative function designated by the user.

Target devices cooperating with each other can be specified by a user operation. For example, as a result of photographing the image forming apparatus 10 and the projector 12 with the camera 42, as shown in FIG. 11, a device image 70 related to the image forming apparatus 10 and a device image 72 related to the projector 12 are displayed on the terminal device 16 on the UI unit 46. The device-related image may be an image obtained by photographing by the camera 42, or may be a preliminary image (eg, a schematic image such as an icon) related to the identified device. When the user designates the device images 70 and 72 on the device display screen 68, the image forming apparatus 10 and the projector 12 are designated as target devices that cooperate with each other. For example, when the user designates the device image 70 , the marker-based AR technology or the markerless AR technology is applied to the device image 70 , thereby determining (identifying) the image forming apparatus 10 . Also, when the user designates the device image 72 , the marker-based AR technology or the markerless AR technology is applied to the device image 72 , thereby determining (identifying) the projector 12 . Accordingly, the cooperative function performed by the image forming apparatus 10 and the projector 12 is determined, and information on the cooperative function is displayed on the UI unit 46 of the terminal apparatus 16 .

As another example, a user may touch device image 70 on device display screen 68 with, for example, his/her finger, and may move the finger to device image 72 to designate device images 70 and 72 and connect image forming apparatus 10 and projector 12 Target devices designated as cooperating with each other. The order in which the user touches the device images 70 and 72 or the direction of movement of the finger may be reversed from the above example. Of course, a screen pointing unit other than a finger moving on the device display screen 68, such as a pen, may be used. The user can link the device images 70 and 72 to each other to designate the device images 70 and 72 , and can thereby designate the image forming apparatus 10 and the projector 12 as target devices that cooperate with each other. The user can superimpose the device images 70 and 72 on each other to designate the device images 70 and 72 , and can thereby designate the image forming apparatus 10 and the projector 12 as target devices that cooperate with each other. In addition, target devices cooperating with each other may be specified by drawing a circle thereon, or target devices may be specified by specifying a device image related to the device within a predetermined period. In the case of releasing the collaboration, the user can designate the target device to be released on the device display screen 68, or can press the collaboration releasing button. If an image of a device that is not a cooperating target device is on the device display screen 68, the user may designate the device on the device display screen 68 to remove the device from the cooperating target devices with each other. The device to be released can be specified by performing a predetermined operation (eg, drawing a cross on it).

Target devices that cooperate with each other can be photographed individually. For example, target devices cooperating with each other are identified by performing photographing by the camera 42 a plurality of times. If the photographing by the camera 42 is performed a plurality of times, the device identification information of the device identified in each photographing operation is stored in the memory of the server 14 or the terminal device 16 . For example, the image forming apparatus 10 is photographed with the image forming apparatus 10 within the field of view of the camera 42 , and then the projector 12 is photographed with the projector 12 within the field of view of the camera 42 . Thus, image data representing the image forming apparatus 10 and image data representing the projector 12 are generated. By applying marker-based AR technology or markerless AR technology to each piece of image data, the image forming apparatus 10 and the projector 12 are determined (identified), and the cooperative function using the functions of the image forming apparatus 10 and the projector 12 is determined (identified). . For example, the image forming apparatus 10 and the projector 12 , which are target devices cooperating with each other, are not always close to each other within the field of view of the camera 42 . The viewing angle of the camera 42 can be changed, or the field of view can be increased or decreased. If these operations are not enough, shooting may be performed multiple times to identify target devices that cooperate with each other.

For another example, the target device for cooperation may be preset as a basic cooperation device. For example, it is assumed that the image forming apparatus 10 is preset as a basic cooperation device. The device identification information of the basic cooperation device may be stored in the memory of the server 14 or the terminal device 16 in advance. Alternatively, the user may specify the basic cooperation means using the terminal device 16 . If the basic cooperation device is set, the user uses the camera 42 of the terminal device 16 to photograph a target device other than the basic cooperation device. Accordingly, target devices that cooperate with each other are determined (identified), and one or more cooperation functions using the functions of the basic cooperation device and the photographed device are determined (identified).

In the examples shown in FIGS. 11 and 12 , the respective cooperative functions are functions using hardware devices. Alternatively, the collaboration function may be a function using a function implemented by software (application). For example, instead of the device images, function images (eg, icon images, etc.) related to functions implemented by software may be displayed on the UI unit 46 of the terminal device 16, and a plurality of function images among the function images may be designated by the user to It is made possible to determine (identify) a cooperative function using a plurality of functions related to the plurality of function images. For example, the collaboration function can be determined by specifying a function image related to a function displayed on a home screen of a smartphone or a desktop of a PC. Of course, if a device image related to a hardware device and a function image related to a function implemented by software are displayed on the UI unit 46 of the terminal device 16 and if the user specifies the device image and the function image, it can be recognized that the use related to the device image The device and the collaboration function of the function related to the function image.

In the above example, marker-based AR technology or markerless AR technology is used, but location information AR technology may be used. For example, the terminal device 16 has a GPS function, obtains terminal position information indicating the position of the terminal device 16 , and transmits the terminal position information to the server 14 . The controller 36 of the server 14 refers to the position correspondence information indicating the correspondence between the device position information (the position indicating the device) and the device identification information, and determines a device located within a predetermined range with respect to the position of the terminal device 16 as a candidate cooperation device . For example, it is assumed that the positions of the MFP, PC, printer, and scanner relative to the terminal device 16 are within a predetermined range. In this case, MFPs, PCs, printers, and scanners are determined as candidate cooperation devices. The device identification information of each candidate cooperation device is transmitted from the server 14 to the terminal device 16 and displayed on the UI unit 46 of the terminal device 16 . As the device identification information, an image of a candidate cooperation device may be displayed or a character string such as a device ID may be displayed. The user designates target devices cooperating with each other from among the candidate cooperating devices displayed on the UI unit 46 . The device identification information of the target device designated by the user is transmitted from the terminal device 16 to the server 14 . In the server 14, one or more cooperative functions are determined based on the device identification information of the target device. Information about the one or more cooperative functions is displayed on the UI unit 46 of the terminal device 16 . The process of determining the candidate cooperation apparatus and the process of determining the cooperation function may be performed by the terminal device 16 .

If the photographed device is not recognized even if AR technology or the like is applied, a device image representing the photographed device does not have to be displayed on the device display screen. Thus, the visibility of the identified device can be increased. For example, if there are an identified device and an unidentified device and if both devices are photographed by the camera 42, the device image representing the unidentified device is not displayed. Therefore, the device image representing the recognized device is displayed while being distinguished from the device image representing the unrecognized device, and thus, the visibility of the recognized device can be increased. Alternatively, the device image representing the identified device may be displayed in a highlighted manner. For example, a device image representing an identified device may be displayed in a particular color, may be displayed while the edges of the device image are highlighted, may be displayed while the device image is enlarged, may be displayed three-dimensionally, or may be displayed while the device image is being displayed. The image is displayed while blinking. Thus, the visibility of the identified device can be increased.

Hereinafter, processing for executing the functions of the device will be described. As an example, processing for executing the cooperation function will be described. In this case, a connection request is sent from the terminal apparatus 16 to the target apparatuses cooperating with each other, and the connection between the terminal apparatus 16 and the target apparatus is established. Hereinafter, the connection process will be described with reference to FIG. 13 . FIG. 13 is a sequence diagram showing this process.

First, the user provides an instruction using the terminal device 16 to start an application (program) for executing the function of the device. In response to this instruction, the controller 48 of the terminal device 16 starts the application (S01). The application may be pre-stored in the memory 44 of the terminal device 16, or may be downloaded from the server 14 or the like.

Subsequently, target devices cooperating with each other are identified by applying marker-based AR technology, markerless AR technology, or location information AR technology ( S02 ). Of course, the target device may be identified using technologies other than AR technology. In the case of applying marker-based AR technology or markerless AR technology, the user uses the camera 42 of the terminal device 16 to photograph the target device. For example, in the case of using the image forming apparatus 10 (MFP (B)) and the projector 12 (projector (C)) as target devices, the user photographs the image forming apparatus 10 and the projector 12 with the camera 42 . Accordingly, the device identification information of the image forming apparatus 10 and the projector 12 is obtained, and the image forming apparatus 10 and the projector 12 are identified as target devices. In the case of applying the position information AR technology, the position information of the image forming apparatus 10 and the projector 12 is obtained, the device identification information of the image forming apparatus 10 and the projector 12 is determined based on the position information, and the image forming apparatus 10 and the projector are identified 12.

For example, if the user provides an instruction to display a collaborative function, a collaborative function using the functions of the plurality of identified devices is identified. Information on the identified cooperative function is displayed on the UI unit 46 of the terminal device 16 (S03). The process of identifying the cooperative function may be performed by the server 14 or the terminal device 16 .

Subsequently, after the user specifies the target cooperation function to be executed using the terminal device 16, the terminal device 16 transmits information indicating a connection request to the target apparatuses (eg, the image forming apparatus 10 and the projector 12) executing the cooperation function (S04). For example, if address information representing addresses of target devices cooperating with each other is stored in the server 14 , the terminal device 16 obtains the address information from the server 14 . If the address information is included in the device identification information, the terminal apparatus 16 may obtain the address information from the device identification information of the target device. Alternatively, the address information of the target device may be stored in the terminal device 16 . Of course, the terminal device 16 may obtain the address information of the target device using another method. Using the address information of the target apparatuses (eg, the image forming apparatus 10 and the projector 12 ), the terminal device 16 transmits information indicating a connection request to the target apparatuses (eg, the image forming apparatus 10 and the projector 12 ).

The image forming apparatus 10 and the projector 12 that have received the information representing the connection request allow or disallow connection to the terminal apparatus 16 (S05). For example, if the image forming apparatus 10 and the projector 12 are devices for which connection is not permitted or if the number of devices requesting connection exceeds the upper limit, the connection is not permitted. If the connection from the terminal device 16 is permitted, the operation of changing the setting information specific to the image forming apparatus 10 and the projector 12 may be prohibited so that the setting information is not changed through the terminal device 16 . For example, the change of the color parameters of the image forming apparatus 10 or the set time for transition to the power saving mode may be prohibited. Therefore, the security of target devices cooperating with each other can be increased. Alternatively, in the case of causing the devices to cooperate with each other, the change of the setting information may be restricted compared to the case of using each device alone without cooperating with another device. For example, fewer setting items can be allowed to be changed than in the case of using the device alone. Alternatively, viewing other users' personal information (eg, operation history) may be prohibited. Therefore, the security of the user's personal information can be increased.

Result information indicating whether the connection is permitted or not is transmitted from the image forming apparatus 10 and the projector 12 to the terminal apparatus 16 (S06). If connection to the image forming apparatus 10 and the projector 12 is allowed, communication is established between the terminal apparatus 16 and each of the image forming apparatus 10 and the projector 12 .

Subsequently, the user provides an instruction to execute the cooperation function using the terminal device 16 (S07). In response to this instruction, execution instruction information indicating an instruction to execute the cooperative function is transmitted from the terminal apparatus 16 to the image forming apparatus 10 and the projector 12 (S08). The execution instruction information sent to the image forming apparatus 10 includes information (eg, job information) indicating processing to be executed in the image forming apparatus 10 , and the execution instruction information sent to the projector 12 includes information indicating processing to be executed in the projector 12 information (for example, job information).

In response to the execution instruction information, the image forming apparatus 10 and the projector 12 execute respective functions according to the execution instruction information (S09). For example, if the cooperation function includes processing of sending/receiving data between the image forming apparatus 10 and the projector 12, such as transferring scan data from the image forming apparatus 10 (MFP (B)) to the projector 12 (projector (C)) Communication is established between the image forming apparatus 10 and the projector 12 as in the function of projecting the data by the projector 12 . In this case, for example, the execution instruction information transmitted to the image forming apparatus 10 includes address information of the projector 12 , and the execution instruction information transmitted to the projector 12 includes address information of the image forming apparatus 10 . Communication is established between the image forming apparatus 10 and the projector 12 using these address information.

After the execution of the cooperation function is completed, information indicating the completion of the execution of the cooperation function is transmitted from the image forming apparatus 10 and the projector 12 to the terminal device 16 (S10). Information indicating that the execution of the cooperative function is completed is displayed on the UI unit 46 of the terminal device 16 (S11). If the information indicating the completion of the execution of the cooperation function is not displayed even though a predetermined period of time has elapsed from the point of time when the execution instruction was provided, the controller 48 of the terminal device 16 may cause the UI unit 46 to display the information indicating the error, and may display the information indicating the error again. The image forming apparatus 10 and the projector 12 transmit execution instruction information or information indicating a connection request.

Subsequently, the user determines whether to release the cooperative state of the image forming apparatus 10 and the projector 12 ( S12 ), and executes processing according to the determination result ( S13 ). In the case of releasing the cooperative state, the user provides a release instruction using the terminal device 16 . Therefore, the communication between the terminal apparatus 16 and each of the image forming apparatus 10 and the projector 12 is stopped. Furthermore, the communication between the image forming apparatus 10 and the projector 12 is stopped. Execution instructions can continue to be provided without releasing the cooperative state.

In addition, the number of target devices cooperating with each other can be increased. For example, the device identification information of the third device can be obtained, and the cooperative function performed by the cooperation between the three devices including the image forming apparatus 10 and the projector 12 can be determined. Information indicating that the image forming apparatus 10 and the projector 12 have been identified as target devices cooperating with each other is stored in the server 14 or the terminal apparatus 16 .

Device identification information representing target devices that cooperate with each other and cooperation function information representing the performed cooperation function may be stored in the terminal device 16 or the server 14 . For example, the user account information (user identification information) of the user who uses the terminal device 16 can be obtained, and is expressed between the user account information, the device identification information indicating the target devices that cooperate with each other, and the cooperation function information indicating the executed cooperation function The historical information of the corresponding relationship can be created and stored in the terminal device 16 or the server 14 . The historical information may be created by the terminal device 16 or the server 14 . With reference to the historical information, the cooperative function that has been executed and the device used for the cooperative function are determined.

As history information, target apparatuses (eg, image forming apparatus 10 and projector 12 ) cooperating with each other may store user account information of users who have requested connection and terminal identification information indicating terminal apparatus 16 that has requested connection. With reference to the historical information, a user who has used the device is determined. In the case of determining, for example, a user who used the device when the device was damaged, or in the case of performing a billing process for consumables or the like, the user can be identified using the history information. The historical information may be stored in the server 14 or the terminal device 16, or may be stored in another device.

For example, user account information is pre-stored in the memory 44 of the terminal device 16 . The controller 48 of the terminal device 16 serves as an example of a user identification unit, reads user account information of the user from the memory 44 , and identifies the user using the terminal device 16 . If user account information of a plurality of users is stored in the memory 44 , the user specifies his/her user account information using the terminal device 16 . Therefore, the user account information of the user is read and the user is identified. Alternatively, the controller 48 of the terminal device 16 may identify the user by reading the user account information of the user logged into the terminal device 16 . Alternatively, if only one piece of user account information is stored in the same terminal device 16, the controller 48 of the terminal device 16 may identify the user by reading this one piece of user account information. If the user account is not set and if the user account information is not created, initial setting is performed, thereby creating the user account information.

The usage history of the collaboration function may be managed for each user, and information representing the collaboration function previously used by the user indicated by the read user account information may be displayed on the UI unit 46 of the terminal device 16 . Information representing the usage history may be stored in the terminal device 16 or the server 14 . In addition, information indicating a cooperative function used at a predetermined frequency or more may be displayed. In the case where such a shortcut function is provided, user operations regarding the collaboration function can be simplified.

In the case of executing the single-device function, information indicating an instruction to execute the single-device function is transmitted from the terminal apparatus 16 to the device executing the single-device function. The device performs the single-device function according to the instruction.

In the above example, the cooperative function may be performed by multiple devices. However, the cooperation function is not always executable depending on the combination of devices. Furthermore, depending on the combination of a plurality of functions (eg, a combination of a function implemented by software or a combination of a function implemented by software and a function of a hardware device), the cooperative function is not always executable. This will be described in detail below.

FIG. 14A shows an example of a combination of devices that cannot perform cooperative functions. For example, assume that an MFP (B) and a hair dryer (D) are identified as devices. As shown in FIG. 14A, a device display screen 68 is displayed on the UI unit 46 of the terminal device 16, and device images 70 and 76 related to the recognized devices (MFP (B) and hair dryer (D)) are displayed on the device display on screen 68. If there is no cooperative function executable by the MFP (B) and the blower (D) and if the MFP (B) and the blower (D) are designated as target devices to cooperate with each other, the information on the cooperative function is not displayed, and the message screen 78 is displayed on the UI 46 of the terminal device 16, as shown in FIG. 14B. The message screen 78 displays a message indicating that the cooperative function cannot be performed by the MFP (B) and the blower (D).

The above process will be described in more detail. When the MFP (B) and the blower (D) are identified and designated as target devices to cooperate with each other, the determination unit 38 of the server 14 by referring to the cooperation function management information 34 as described above (for example, the cooperation function management shown in FIG. 7 ) Table) to determine (identify) the cooperative function using the MFP (B) and the blower (D). If the cooperative function using the MFP (B) and the blower (D) is registered in the cooperative function management table, the determination unit 38 determines the cooperative function. On the other hand, if the cooperative function using the MFP (B) and the blower (D) is not registered in the cooperative function management table, the determination unit 38 determines that there is no cooperative function using the MFP (B) and the blower (D). In this case, the controller 36 of the server 14 outputs a message indicating that the combination of the MFP (B) and the hair dryer (D) cannot perform the cooperative function. This message is displayed on the UI unit 46 of the terminal device 16, as shown in Fig. 14B.

Even if there is no available cooperation function as in the above-mentioned case, the cooperation function may become available depending on the operating state of the device, the environment in which the device is installed (surrounding environment), or a change (update) of the function of the device. In the above example, if condensation occurs in the environment where the MFP (B) is installed, the blower (D) can be used to remove or prevent the condensation. In this case, the cooperative function using the MFP (B) and the hair dryer (D) is available, and therefore, information indicating the cooperative function is displayed on the UI unit 46 of the terminal device 16 . For example, the controller 36 of the server 14 monitors the operation state of each device, the environment in which the device is installed (surrounding environment), the update state of the function of each device, etc., and determines the availability of the cooperative function based on the monitoring results. In the case of the combination of the MFP (B) and the blower (D), the controller 36 determines cooperation if the surroundings of the MFP (B) satisfy certain conditions (eg, if condensation occurs in the surroundings of the MFP (B)) The function is available and determines (identifies) the cooperative function using the hair dryer (D). The same applies to the operational states of the devices, ie, if an identified or designated group of devices is in a particular operational state, the controller 36 determines that a cooperative function using this group of devices is available. The same applies to the case where the function of the device is updated and the collaboration function becomes available through the updated function.

Furthermore, there are cases where a cooperative function cannot be executed by a plurality of functions realized by a plurality of software units, and there are cases where a cooperative function cannot be executed by a function realized by a software unit and a hardware device.

Boot processing

In an exemplary embodiment, for example, when specifying an image related to a device, a guide is presented that indicates a hardware device or a function implemented by software capable of performing cooperative functions with the device. The same applies to using the collaborative features of the software. For example, when specifying an image related to a function implemented by software, a guide indicating a hardware device capable of performing a cooperative function together with the function or a function implemented by software is presented. Hereinafter, an example of the boot process according to the exemplary embodiment will be described in detail.

Example 1

Guidance processing according to Example 1 will be described with reference to FIG. 15 . FIG. 15 shows an example of a device display screen according to Example 1. FIG. For example, assume that an MFP (B), a projector (C), and a hair dryer (D) are identified as devices. The device display screen 68 is displayed on the UI unit 46 of the terminal device 16, and the device images 70, 72, and 76 related to the recognized devices (MFP (B), projector (C), and hair dryer (D)) are displayed on the The device display screen 68 is displayed.

In this case, for example, it is assumed that the user selects the MFP (B) and the user designates the device image 70 related to the MFP (B). The MFP(B) corresponds to the first device, and the device image 70 related to the MFP(B) corresponds to the first image related to the first device. In response to the user designating the MFP (B) as the first device, the determination unit 38 of the server 14 determines the second device capable of performing the cooperation function together with the MFP (B) as the first device, by referring to the cooperation function management information 34 (for example, The cooperation function management table shown in FIG. 7). For example, it is assumed that the combination of the MFP (B) and the projector (C) can perform the cooperative function and the combination of the MFP (B) and the blower (D) cannot perform the cooperative function. That is, it is assumed that the cooperation function using the MFP (B) and the projector (C) is registered in the cooperation function management table and the cooperation function using the MFP (B) and the blower (D) is not registered in the cooperation function management table. In this case, the projector (C) is determined as the second device, and the controller 36 of the server 14 performs control to present a guide indicating the projector (C) as the second device. Specifically, the device identification information representing the projector (C) is transmitted from the server 14 to the terminal device 16 under the control of the controller 36 . The controller 48 of the terminal device 16 presents a guide indicating the projector (C) as the second device. For example, as shown in FIG. 15 , the controller 48 of the terminal device 16 causes the arrow 80 indicating the projector (C) as a cooperation partner device to be displayed on the device display screen 68 . The arrow 80 is an image linking the device image 70 (the first image related to the MFP (B) as the first device) and the device image 72 (the second image related to the projector (C) as the second device) to each other . Of course, controller 48 may utilize methods other than arrows to present the guidance indicating projector (C) as the second device. For example, the controller 48 may present a guide indicating the second device by outputting a sound, may cause an indicia superimposed on the second image (eg, the device image 72) related to the second device to be displayed on the device display 68, may A second image related to the second device is caused to be displayed on the device display screen 68 so that the second image is distinguishable from another image, or a character string representing a collaborating partner may be caused to be displayed on the device display screen 68 .

In the above-described case where the user designates the MFP (B) as the first device, the projector (C) as the second device capable of performing the cooperation function together with the MFP (B) is recommended as a candidate cooperation partner device. Therefore, compared to the case where no such candidate is recommended, the user's convenience for specifying the device required for the cooperation function can be increased.

If the user designates a function image (an image corresponding to the first image) related to the first function realized by the software, a guide process similar to the guide process described above is also executed. That is, if the user designates the function image as the first image, a guide indicating that the second function of the cooperative function can be executed together with the first function related to the function image is presented. For example, guidance indicating a second image related to the second function may be presented, or a sound or character string may be used to present guidance indicating the second function. The second function may be a function implemented by software or a function of a hardware device. Of course, if the user specifies a device image related to the first device, then guidance may be presented indicating a second function capable of performing a cooperative function with the first device and implemented by software.

Since the hair dryer (D) is a device that cannot perform a cooperative function together with the MFP (B), guidance indicating the hair dryer (D) as a cooperative partner is not presented.

According to the environment (surrounding environment) in which the MFP (B) as the first device is installed, the operating state of the MFP (B) (for example, the amount of toner, the amount of paper, whether the MFP is in use, or the processing end time), the A change in function (update), a change in function (update) of the blower (D), etc., may present a guide indicating the blower (D) as the second device. For example, if condensation occurs in the surrounding environment of the MFP (B) as the first device, the determination unit 38 of the server 14 determines the blower (D) as the second device required to remove the condensation. In this case, as in the above-mentioned case of the projector (C), a guide indicating the blower (D) as a cooperative partner is presented. For example, displaying an arrow linking the device image 70 (the first image related to the MFP (B) as the first device) and the device image 76 (the second image related to the hair dryer (D) as the second device) to each other, Or use sound to present guidance indicating the blower (D).

The device or object associated with the image displayed on the device display screen 68 is not necessarily identified. For example, there may be a case where the determination unit 38 of the server 14 does not recognize the hair dryer (D). Unidentified devices or objects are excluded from candidate collaboration partners (second devices). Images related to unidentified devices or objects may or may not be displayed on the device display screen 68 .

Example 2

Guidance processing according to Example 2 will be described with reference to FIG. 16 . FIG. 16 shows an example of a device display screen according to Example 2. FIG. In Example 2, the plurality of devices correspond to the second device (cooperating partner device). For example, suppose that MFP (B), projector (C), hair dryer (D), and camera (E) are identified as devices and a foliage plant (F) is identified as a foliage plant. Of course, objects that are not devices are not necessarily recognized.

The device display screen 68 is displayed on the UI unit 46 of the terminal device 16, and the device images 70, 72 related to the identified devices (MFP (B), projector (C), hair dryer (D), and camera (E)) , 76 and 82 and an image 84 related to foliage plants (F) are displayed on the device display screen 68 .

In this case, for example, it is assumed that the user selects the MFP (B) as the first device and the user designates the device image 70 related to the MFP (B) as the first image. Also assume that the projector (C) and the camera (E) are determined as second devices capable of performing cooperative functions together with the MFP (B) as the first device. In this case, a guide indicating the projector (C) and the camera (E) as the second device is presented. For example, guidance is presented simultaneously. In the example shown in FIG. 16 , as in Example 1, an arrow 80 linking the device image 70 related to the MFP (B) and the device image 72 related to the projector (C) to each other is displayed as a guide. In addition, an arrow 86 linking the device image 70 related to the MFP (B) and the device image 82 related to the camera (E) to each other is displayed as a guide.

Priorities can be associated with collaboration functions. The information indicating the priority order in the cooperation function management information 34 (for example, the cooperation function management table shown in FIG. 7 ) is associated with each cooperation function. In the case where guidance indicating a plurality of second devices is presented, information indicating the priority order of each cooperative function is transmitted from the server 14 to the terminal device 16 , and the priority order is displayed on the device display screen 68 . For example, if the priority of the first cooperation function using the MFP (B) and the projector (C) is higher than the second cooperation function using the MFP (B) and the camera (E), the controller 48 of the terminal device 16 causes the instruction Information that the projector (C) used in the first cooperation function has a higher priority than the camera (E) used in the second cooperation function is displayed on the device display screen 68 . The controller 48 may cause the string representing the priority order to be displayed on the device display screen 68, may cause the arrows 80 and 86 to be displayed on the device display screen 68 in different colors, may cause the device images 72 and 82 to be displayed in different display formats On the device display screen 68, alternatively the arrow 80 for the higher priority device image 72 may be caused to be displayed on the device display screen 68 and the arrow 86 for the lower priority device image 82 may not be displayed on the device display on screen 68.

Alternatively, instead of displaying an arrow, the controller 48 of the terminal device 16 may display a character string representing the second device in a specific area of the device display screen 68 . For example, the character string may be displayed in an area where the device image is not displayed. Therefore, it is prevented from occurring that the arrow makes it difficult to see the information displayed on the screen.

In the above-mentioned case where the user designates the MFP (B) as the first device, the projector (C) and the camera (E) as the second devices capable of performing the cooperation function together with the MFP (B) are recommended as candidate cooperation partner devices .

If the candidate collaboration partner device (second device) is not displayed on the device display screen 68 , position information indicating the location where the second device is installed or information indicating guidance indicating the location where the second device is installed may be displayed on the terminal device 16 UI unit 46. For example, the controller 36 of the server 14 obtains the positional information of the second device using the GPS function or the like, and creates a representation of the second device indicating the position relative to the terminal device 16 based on the obtained positional information and the positional information of the terminal device 16 . Information for location guidance. Information representing the guidance may be sent from the server 14 to the terminal device 16 and may be displayed on the UI unit 46 of the terminal device 16 .

As in Example 1, the processing according to Example 2 is applicable to the case of using function images related to functions. For example, if the user designates a function image as the first image, a guide indicating a plurality of second functions capable of executing the cooperative function together with the first function related to the function image may be presented. Of course, if the user designates a device image related to the first device, a guide indicating a plurality of functions capable of performing the cooperative function together with the first device may be presented.

Example 3

The guidance process according to Example 3 will be described with reference to FIGS. 17 to 20 . 17 to 20 each illustrate an example of a device display screen according to Example 3. FIG. In Example 3, the user designates a first device, then a second device, and controls are performed to present guidance indicating a third device capable of performing cooperative functions with the first and second devices. The device recommended as the third device may vary according to the order in which the first and second devices are designated.

For example, suppose that PC(A), MFP(B), projector(C), and camera(E) are recognized as devices, and the recognized devices(PC(A), MFP(B), projector(C), and Device images 70 , 72 , 82 and 88 and image 84 related to the camera (E)) are displayed on the device display screen 68 as shown in FIG. 17 .

In this case, for example, if the user designates the device image 70 related to the MFP(B), the determination unit 38 of the server 14 recognizes the MFP(B) as the first device, and can perform cooperation together with the MFP(B) The functional PC (A), projector (C) and camera (E) are identified as the second device (candidate partner device). As shown in Figure 17, for example, arrows 80, 86 and 90 are shown as guides to indicate the second device. An arrow 90 is an image linking the device image 70 related to the MFP (B) and the device image 88 related to the PC (A) to each other.

Subsequently, it is assumed that the user selects the projector (C) as a cooperation partner device from the set of second devices and the user designates the device image 72 related to the projector (C). In this case, the determination unit 38 of the server 14 determines, by referring to the cooperation function management information 34, a third device capable of executing the cooperation function together with the MFP (B) as the first device and the projector (C) as the second device device. In the cooperation function management table shown in FIG. 7, cooperation functions executable by cooperation between two apparatuses are registered. However, it is of course possible to register a cooperation function that can be performed by cooperation between three or more devices. For example, assume that PC(A) is determined as the third device. In this case, a guide indicating the PC (A) as the third device is presented as shown in FIG. 18 . For example, an arrow 92 linking the device image 72 related to the projector (C) as the second device and the device image 88 related to the PC (A) as the third device to each other is displayed as a guide.

For example, the user may designate the device image 72 related to the projector (C), perform an operation of linking the device image 70 related to the MFP (B) and the device image 72 related to the projector (C) to each other, and link the device image 70 to each other. Overlay on device image 72, or place a pointer on device image 70 and then move the pointer to device image 72 to designate the projector (C) to be used as the second device.

The order in which each device (each device image) is specified corresponds to the order in which the functions of the devices are used or the order in which data is moved between the devices. An operation of specifying a device (eg, an operation of linking an image or superimposing an image on another image) acts as an operation of specifying the order in which functions are used or the order of data movement. In the example shown in FIG. 18 , the MFP (B) as the first device is used first, and the projector (C) as the second device is used second. A guide is presented indicating a third device capable of performing a collaborative function with the first and second devices and used by a third in the collaborative function. That is, a guide indicating the third device used by the third in the cooperation function in which the MFP (B) is used first and the projector (C) is used second is presented. In the example shown in FIG. 18, PC(A) is the third device. In Example 3, cooperation functions executable through cooperation among a plurality of devices are registered in the cooperation function management information 34, and the order of use of the devices is also registered. The determination unit 38 of the server 14 determines the third device by referring to the cooperation function management information 34 .

FIG. 19 shows another example. As in FIG. 17 , device images 70 , 72 , 82 and 88 and image 84 are displayed on the device display screen 68 .

In this case, for example, if the user specifies the device image 72 related to the projector (C), the projector (C) is identified as the first device, and the determination unit 38 of the server 14 will be able to communicate with the projector (C) ), the PC (A), the MFP (B), and the camera (E) that perform the cooperative function together are identified as the second device (candidate collaboration partner device). As shown in Figure 19, for example, arrows 94, 96 and 98 are shown as guides to indicate the second device. Arrow 94 is an image linking the device image 72 related to the projector (C) as the first device and the device image 70 related to the MFP (B) as the second device to each other. An arrow 96 is an image linking the device image 72 and the device image 82 related to the camera (E) as the second device to each other. An arrow 98 is an image linking the device image 72 and the device image 88 related to the PC (A) as the second device to each other.

Subsequently, it is assumed that the user selects the MFP (B) as a cooperation partner device from the set of second devices and the user designates the device image 70 related to the MFP (B). In this case, the determination unit 38 of the server 14 determines, by referring to the cooperation function management information 34, a third device capable of executing the cooperation function together with the projector (C) as the first device and the MFP (B) as the second device device. For example, assume that the camera (E) is determined as the third device. In this case, a guide indicating the camera (E) as the third device is presented as shown in FIG. 20 . For example, an arrow 100 linking the device image 70 related to the MFP (B) as the second device and the device image 82 related to the camera (E) as the third device to each other is displayed as a guide. In this case, the guidance indicating the device (for example, the camera (E)) used by the third in the cooperation function in which the projector (C) is used first and the MFP (B) is used by the second is Presented as a third device.

For example, the user may designate the device image 70 related to the MFP (B), perform an operation of linking the device image 72 related to the projector (C) and the device image 70 related to the MFP (B) to each other, superimpose the device image 72 On the device image 70, or place a pointer on the device image 72 and then move the pointer to the device image 70, to designate the MFP (B) serving as the second device.

In the manner described above, guidance is presented indicating a third device capable of performing a cooperative function with the first and second devices. The device presented (recommended) as the third device varies according to the order in which the first and second devices are designated. The order in which the devices are assigned corresponds to the order in which the functions of the devices are used or the order in which data is moved between the devices. The operation of specifying the device acts as an operation of specifying the order in which functions are used or the order of data movement. The device to be used next in the cooperation function or the device to which the data is destined changes according to the order in which the devices are specified. Therefore, in Example 3, guidance indicating the third device to be used in the collaboration function is presented according to the change.

As in Example 1, the processing according to Example 3 is applicable to the case of using function images related to functions. For example, if the user designates a function image related to the first function, a guide indicating the function image related to the second function capable of executing the cooperative function together with the first function is presented. If the user designates a function image related to the second function, a guide indicating a function image related to a third function capable of performing a cooperative function together with the first and second functions may be presented. In this case, the function presented as the third function changes according to the order in which the first and second functions are specified. Also in Example 3, the cooperative function may be a function using a hardware device and a function implemented by software.

Example 4

The guidance process according to Example 4 will be described with reference to FIGS. 21 and 22 . 21 and 22 each illustrate an example of a device display screen according to Example 4. FIG. In Example 4, if a device that cannot perform the cooperation function together with the first device is specified, guidance indicating a second device (a cooperation partner device) capable of performing the cooperation function together with the first device is presented. For example, assume that an MFP (B), a projector (C), and a hair dryer (D) are identified as devices.

As shown in FIG. 21, a device display screen 68 is displayed on the UI unit 46 of the terminal device 16, and device images 70, 70, 72 and 76 are displayed on the device display screen 68 .

In this case, for example, if the user selects the MFP(B) as the first device and the user designates the device image 70 related to the MFP(B) as the first image, the determination unit 38 of the server 14 recognizes the MFP(B) for the first device. For example, assume that the projector (C) corresponds to a second device capable of performing a cooperative function with the MFP (B) as the first device and the blower (D) corresponds to a device that cannot perform a cooperative function with the MFP (B). In this case, for example, it is assumed that the user performs an operation of linking the device image 70 related to the MFP (B) and the device image 76 related to the hair dryer (D) to each other by specifying the device image 76 related to the hair dryer (D). , superimpose the device image 70 on the device image 76, or place the pointer on the device image 70 and then move the pointer to the device image 76 to designate the hair dryer (D) that cannot perform the cooperation function with the MFP (B) as a cooperation partner device. In the example shown in FIG. 21 , as indicated by arrow 102 , the operation of linking the device image 70 and the device image 76 to each other is performed by the user.

When the user designates the hair dryer (D) that cannot perform the cooperation function together with the MFP (B) as the first device as the cooperation partner device, the controller 36 of the server 14 receives the designation and performs control to present the indicating projector (C) Guidance as a second device capable of performing cooperative functions together with the MFP (B). Therefore, a guide indicating the projector (C) as the second device is presented. For example, as shown in FIG. 22 , the controller 48 of the terminal device 16 causes the arrow 104 indicating the projector (C) as a cooperation partner device to be displayed on the device display screen 68 . For example, the arrow 104 is an image linking the device image 70 related to the MFP (B) as the first device and the device image 72 related to the projector (C) as the second device to each other. Of course, the guidance can be presented by using sounds or displaying strings.

As described above, in Example 4, if a device that cannot perform a cooperative function with the first device is specified, control is performed to present a guide indicating a second device that can perform a cooperative function with the first device. If an arrow or the like serving as a guide is displayed every time the first device is designated, the arrow or the like may clutter the screen. This can be avoided in Example 4.

As in Example 1, the processing according to Example 4 is applicable to the case of using function images related to functions. For example, if a function that cannot perform the cooperative function together with the first function is specified, guidance indicating a second function that can perform the cooperative function together with the first function is presented.

Example 5

Example 5 will be described with reference to FIGS. 23 to 25 . 23 to 25 each show an example of a screen according to Example 5. In Example 5, if the device designated by the user as the collaborative partner device cannot perform the collaborative function with the first device because the device is damaged or in use, then a presentation indicating another device capable of performing the collaborative function with the first device is presented. guide. In this case, guidance indicating the same type of device (eg, a device having the same type of function) as the device designated by the user as the collaboration partner device may be preferentially presented. The controller 36 of the server 14 obtains information representing the operation status of the respective apparatuses (eg, whether the apparatus is performing processing, damaged, or under maintenance) from the respective apparatuses and manages the operation status of the respective apparatuses.

For example, assume that MFP (B) and projectors (C) and (F) are identified as devices. As shown in FIG. 23, a device display screen 68 is displayed on the UI unit 46 of the terminal device 16, and device images 70, 72 related to the recognized devices (MFP (B) and projectors (C) and (F)) and 106 are displayed on the device display screen 68 .

In this case, for example, if the user selects the MFP(B) as the first device and the user designates the device image 70 related to the MFP(B) as the first image, the determination unit 38 of the server 14 recognizes the MFP(B) for the first device. For example, it is assumed that the projectors (C) and (F) correspond to second devices capable of performing a cooperative function together with the MFP (B) as the first device. For example, it is assumed that the user links the device image 106 related to the projector (F) by designating the device image 106 related to the projector (F), performs an operation of linking the device image 70 related to the MFP (B) and the device image 106 related to the projector (F) to each other 70 is superimposed on the device image 106, or the pointer is placed on the device image 70 and then moved to the device image 106 to designate the projector (F) as the partner device. In the example shown in FIG. 23 , as indicated by arrow 108 , the operation of linking the device image 70 and the device image 106 to each other is performed by the user.

When the user designates the projector (F) as the cooperation partner device, the controller 36 of the server 14 receives the designation and checks the operation state of the projector (F). For example, if the projector (F) is damaged or is being used, the controller 36 of the server 14 performs control to present a device other than the indicator projector (F), ie, another device capable of performing a cooperative function with the first device guide. Controller 36 may preferentially present guidance indicating the same type of device as projector (F) (eg, a device having the same type of functionality as projector (F)). For example, if projector (C) is the same type of device as projector (F), the guide indicating projector (C) as the second device is presented preferentially. In this case, as shown in FIG. 24 , for example, the controller 48 of the terminal device 16 causes the arrow 110 indicating the projector (C) as the cooperation partner device to be displayed on the device display screen 68 . For example, the arrow 110 is an image linking the device image 70 related to the MFP (B) as the first device and the device image 72 related to the projector (C) as the second device to each other.

If the user designates a damaged or in-use device as a cooperation partner device, the screen 112 shown in FIG. 25 may be displayed on the UI unit 46 of the terminal device 16 under the control of the controller 36 of the server 14, showing an indication indicating that cooperation cannot be performed the reason for the message.

If the damaged device becomes available after repair or if the device performing the process completes the process and is not performing the process, the controller 36 of the server 14 identifies the device as a device capable of performing cooperative functions with the first device.

According to Example 5, a guide indicating a device that is not damaged or not in use is presented, and thus, user convenience can be increased. In addition, guidance is presented indicating a device of the same type as the device specified by the user, and thus, guidance indicating the device the user intends to use is presented.

Example 6

The boot process according to Example 6 will be described with reference to FIG. 26 . FIG. 26 shows an example of a device selection screen. In Example 6, a candidate list showing information on one or more second devices capable of performing a cooperative function together with the first device is displayed on the UI unit 46 of the terminal device 16 .

For example, assuming that the MFP (B) and the hair dryer (D) are recognized as devices and the device image 70 related to the MFP (B) and the device image 76 related to the hair dryer (D) are displayed on the UI unit 46 of the terminal device 16, As shown in Figure 14A. In this case, for example, if the user selects the MFP(B) as the first device and the user designates the device image 70 related to the MFP(B), the determination unit 38 of the server 14 recognizes the MFP(B) as the first device . For example, assume that the hair dryer (D) is a device that cannot perform a cooperative function together with the MFP (B). In this case, for example, it is assumed that the user performs an operation of linking the device image 70 related to the MFP (B) and the device image 76 related to the hair dryer (D) to each other by specifying the device image 76 related to the hair dryer (D). , superimpose the device image 70 on the device image 76, or place the pointer on the device image 70 and then move the pointer to the device image 76 to designate the hair dryer (D) as a partner device.

When the user designates the hair dryer (D), which cannot perform the cooperation function together with the MFP (B) as the first device, as the cooperation partner device, the controller 36 of the server 14 receives the designation and is able to work with the MFP (B) as a presentation instruction Controlling the directing of the one or more second apparatuses performing the cooperative function, performing control to display a candidate list showing information about the one or more second apparatuses. Therefore, as shown in FIG. 26 , the device selection screen 114 is displayed on the UI unit 46 of the terminal device 16 , and the candidate list is displayed on the device selection screen 114 . The message screen 78 shown in FIG. 14B may be displayed on the UI unit 46 of the terminal device 16 before the device selection screen 114 is displayed.

As shown in FIG. 26 , the candidate list includes the names of respective devices capable of executing the cooperation function together with the MFP (B), device images related to the respective devices, examples of the cooperation function (eg, the name of the cooperation function), and the like. Of course, the candidate list may include at least one of these pieces of information. Each device image may be an image representing the appearance of an actual device (image related to the device in a one-to-one relationship), or may be an image (eg, an icon) that schematically depicts the device. For example, the image representing the appearance of the actual device is an image generated by photographing the appearance of the device, and is an image representing the device itself. An image schematically depicting a device corresponds to an image representing a type of device. As an example of a collaboration function, the candidate list includes the name of one collaboration function or the names of multiple collaboration functions. In the case of displaying the names of a plurality of cooperation functions, the names of the respective cooperation functions may be displayed in a display order corresponding to the order in which the plurality of target devices that cooperate with each other are designated. For example, if the combination of the MFP and the projector is capable of performing multiple cooperative functions (eg, cooperative functions A and B), between when the MFP is designated as the first device and when the projector is designated as the first device, The display order of the plurality of cooperation functions included in the candidate list may be different. For example, if the MFP is designated as the first device, the names of the respective cooperation functions may be displayed in the order of the cooperation function A and the cooperation function B. If the projector is designated as the first device, the names of the respective cooperation functions may be displayed in the order of the cooperation function B and the cooperation function A.

The arrangement order of the second devices in the candidate list may be determined based on, for example, past usage records of the respective second devices. For example, the controller 36 of the server 14 manages the usage records of the respective devices by obtaining information representing the usage records of the devices from the respective devices. For example, the controller 36 displays the second devices in descending order of usage frequency (descending order of usage count) in the candidate list. The past usage record may be a usage record of a user specifying the first image (eg, a user using the terminal device 16 or a user logging in to the server 14 ), or may be a usage record including another user's usage record.

As another example, the controller 36 may display the second devices in the candidate list in descending order of the number of executable cooperative functions. For example, if the number of cooperative functions executable by the projector and the MFP (B) is 3 and if the number of cooperative functions executable by the PC and the MFP (B) is 2, the projector is displayed in the candidate list on the PC.

If the functions of the respective devices are updated and if the cooperative function management information 34 is updated, the controller 36 updates the display of the second device in the candidate list according to the update. For example, if a device that was unable to perform the cooperative function with the MFP before the update becomes capable of performing the cooperative function with the MFP after the update, the controller 36 displays the device in the candidate list as the second device. In addition, the usage records of the respective devices are updated over time, and the controller 36 updates the display order of the second devices in the candidate list based on the updated usage records.

The controller 36 of the server 14 may update the second devices included in the candidate list or may update the display order of the second devices in the candidate list according to the operation states of the first and second devices or the surrounding environment.

Devices that are capable of performing cooperative functions with the MFP but are broken or in use are not necessarily displayed on the candidate list. Also in this case, if the device has been repaired or has become available, the device is displayed on the candidate list.

If the user designates a device name or a device image included in the candidate list, the controller 48 of the terminal device 16 may cause the UI unit 46 of the terminal device 16 under the control of the controller 36 of the server 14 to display a display available through the A list of one or more cooperative functions that the MFP performs with the specified device. For example, if a projector in the candidate list is specified, the controller 48 of the terminal device 16 causes the UI unit 46 to display a list of one or more cooperative functions that can be performed by the MFP and the projector. If the user designates a cooperation function included in the candidate list, control is performed to execute the designated cooperation function.

Each second device displayed in the candidate list may be a device included in a device group pre-registered in the server 14 , a device included in a device group identified using AR technology or the like, a UI unit displayed on the terminal device 16 A device in a device group on 46 , or a device in a device group included in a specific area displayed on the screen of the UI unit 46 . For example, if the user operates a device image related to the device, the device image is displayed in the specific area. The first device may also be a device included in these device groups. The same applies to Examples 1 to 5 above and the following examples.

As in Example 1, the processing according to Example 6 is applicable to the case of using function images related to functions. For example, if a device or function that cannot perform the cooperative function together with the first function is specified, a candidate list showing the device (candidate second device) or function (candidate second device) that can perform the cooperative function together with the first function may be displayed second function). For example, the name of the second function, a function image related to the second function, an example of a cooperation function, and the like are included in the candidate list. Alternatively, if a function that cannot perform the cooperative function together with the first device is specified, a candidate list may be displayed.

According to Example 6, candidate second devices or candidate second functions are displayed as a candidate list, which may be convenient for the user.

Example 7

Example 7 will be described with reference to FIGS. 26 to 32 . In Example 7, as in Example 6, the candidate list is displayed on the UI unit 46 of the terminal device 16 .

As in Example 6, it is assumed that the user designates the MFP(B) as the first device and the user designates a device that cannot perform the cooperative function together with the MFP(B). The controller 36 of the server 14 receives the designation and, as a control to present a guide indicating one or more second devices capable of performing a cooperative function together with the MFP (B), executes control to display a display showing information about the one or more second devices. A candidate list of information for a plurality of second devices. Therefore, the device selection screen 114 is displayed on the UI unit 46 of the terminal device 16 as shown in FIG. 26 .

For example, if the user specifies a device name, the controller 48 of the terminal device 16, under the control of the controller 36 of the server 14, causes the UI unit 46 of the terminal device 16 to display a list of devices of the same type as the specified device (the second device list of). For example, if the user designates a projector, the device selection screen 116 is displayed on the UI unit 46 of the terminal device 16 as shown in FIG. 27 . A list of projectors as the second device is displayed on the device selection screen 116 . For example, if the determination unit 38 of the server 14 determines (identifies) the projectors aaa, bbb, and ccc as projectors corresponding to the second device, a list showing these projectors is displayed. The user selects the projector to be used as the second device from the list.

Also, a message asking the user whether to add a target device for collaboration is displayed on the device selection screen 116 . For example, if the user specifies the second device, and then the user provides an instruction to add a target device for collaboration (for example, if the user selects "Yes" in FIG. 27 ), the device selection screen 118 is displayed on the terminal device 16 as shown in FIG. 28 . On the UI unit 46, and on the device selection screen 118, a candidate list showing information on one or more third devices capable of performing cooperative functions with the first and second devices is displayed. The candidate list has the same configuration as the candidate list showing information on one or more second devices. The display order of the third device may be different from the display order in the candidate list of the second device. If the user designates the third device, as in the device selection screen 116 (see FIG. 27 ) displayed in response to the designation of the second device, a list of the third device is displayed. The same applies to the addition of a fourth device, a fifth device, and so on.

If an image (eg, an icon) schematically depicting a device is included in the candidate list as a device image and if the device image is designated by the user, a list of second devices (eg, a projector) related to the device image is displayed, As shown in Figure 27. That is, the device image represents this type of device, and generally represents the device. Therefore, if the device image is specified, a list of second devices related to the device image is displayed. On the other hand, an image representing an actual device (an image related to the device in a one-to-one relationship) is an image representing the device itself. Therefore, if the device image is included in the candidate list and designated by the user, the user designates the device itself. In this case, the list of the second devices shown in FIG. 27 is not displayed, and only a message asking the user whether to add a target device for cooperation may be displayed.

If the target device for cooperation is not added (for example, if the user designates “No” in FIG. 27 ), the controller 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display the diagram under the control of the controller 36 of the server 14 The function selection screen 120 shown in 29 is displayed. A list of cooperative functions executable by a plurality of devices designated by the user is displayed on the function selection screen 120 . For example, if the MFP (B) is designated as the first device and if the projector aaa is designated as the second device, a list of cooperative functions that can be performed by the MFP (B) and the projector aaa is displayed. If the user specifies the collaboration function from the list and provides an instruction to execute the collaboration function, the collaboration function is executed through the MFP(B) and the projector aaa.

On the other hand, in the case of executing a cooperative function not included in the list of cooperative functions, the user requests execution of the cooperative function. For example, as shown in FIG. 30 , a screen 122 for making a request is displayed on the UI unit 46 of the terminal device 16 , and the user inputs the name of the cooperation function to be executed on the screen 122 . Information representing the request is sent from the terminal device 16 to the server 14 . In response to receipt of the request, the controller 36 of the server 14 determines whether the collaborative function associated with the request is executable by the designated devices (eg, the first and second devices). If the cooperative function related to the request is not executable, the controller 48 of the terminal device 16, under the control of the controller 36 of the server 14, causes the UI unit 46 of the terminal device 16 to display a message indicating that the cooperative function related to the request is not executable, such as shown in Figure 31. If the collaborative function related to the request is executable, the controller 48 of the terminal device 16, under the control of the controller 36 of the server 14, causes the UI unit 46 of the terminal device 16 to display a message indicating that the collaborative function related to the request is to be executed, such as shown in Figure 32. If the user provides an execution instruction, the cooperative function is executed. In addition, a collaboration function requested by the user can be registered. For example, as shown in FIG. 32, the controller 48 of the terminal device 16, under the control of the controller 36 of the server 14, causes the UI unit 46 of the terminal device 16 to display a display asking the user whether to register the requested cooperation function as a candidate for cooperation from now on function message. If the user selects "register", information on the requested cooperative function is registered in the cooperative function management information 34 from now on and displayed while being included in the list of cooperative functions. If the user selects "do not register", the information about the requested collaboration function is not registered and not included in the list of collaboration functions. For example, if the requested cooperation function corresponds to exceptional processing, if other cooperation functions are used more frequently, or if the user wants to prevent a situation in which the number of cooperation functions included in the list increases and the list becomes complicated, the The requested collaboration function was not registered.

As in Example 1, the processing according to Example 7 is applicable to the case of using function images related to functions.

According to Example 7, candidate second devices or candidate second functions are displayed as a candidate list, which may be convenient for the user. In addition, the target devices for cooperation can be easily added using the candidate list.

Example 8

Example 8 will be described with reference to FIGS. 33 to 35 . In Example 8, as in Examples 6 and 7, the candidate list is displayed on the UI unit 46 of the terminal device 16 .

As in Examples 6 and 7, it is assumed that the user designates the MFP(B) as the first device and the user designates a device that cannot perform the cooperative function together with the MFP(B). In this case, as shown in FIG. 26 , the device selection screen 114 is displayed on the UI unit 46 of the terminal device 16 .

33 The device selection screen 124 is shown. On the device selection screen 124, a list (a second device list) of devices (ie, devices capable of executing the cooperation function) necessary to execute the cooperation function designated by the user (eg, "print display screen") is displayed. If the user designates a device (second device) in the list and if the user provides an instruction to execute the cooperation function, the user-designated cooperation function is executed through the first and second devices designated by the user.

If the user provides an instruction to select a device not included in the device list displayed on the device selection screen 124 (if "Yes" is selected in response to the query "Do you want to select another device?"), the controller of the terminal device 16 48 causes the UI unit 46 of the terminal device 16 to display the device selection screen 126 shown in FIG. 34 under the control of the controller 36 of the server 14. On the device selection screen 126, a candidate list of another device capable of executing the collaboration function designated by the user is displayed. If the user designates a device (second device) on the device selection screen 126 and if the user provides an instruction to execute the cooperation function, the cooperation function designated by the user is executed by the first and second devices designated by the user.

If the user provides an instruction to select a device not included in the device list displayed on the device selection screen 126 (if "Yes" is selected in response to the query "Do you want to select another device?"), the controller of the terminal device 16 48 causes the UI unit 46 of the terminal device 16 to display a screen 128 shown in FIG. 35 for inputting information on a target apparatus for cooperation under the control of the controller 36 of the server 14. The user enters information (eg, the name or type of device) about the target device of the collaboration on screen 128 . The information about the target device input by the user is sent from the terminal device 16 to the server 14 . In response to receiving the information, the controller 36 of the server 14 determines whether the collaborative function specified by the user is executable by the first device and the target device specified by the user. If the cooperation function is not executable, the controller 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display a screen for setting another device as a target device under the control of the controller 36 of the server 14 . If the cooperation function is executable and if an instruction to execute the cooperation function is provided, the user-designated cooperation function is executed through the first and second devices (target devices that cooperate with each other) designated by the user.

As in Example 1, the processing according to Example 8 is applicable to the case of using function images related to functions.

According to Example 8, when a cooperative function is specified in the candidate list, a device necessary to execute the cooperative function is displayed, and thus, the user's convenience for selecting a device can be increased.

Exemplary embodiments are applicable to environments where multiple users use multiple devices. For example, the terminal device 16 functions as a user interface even if a user interface such as a touch screen is removed from the apparatus. In another case, for example, if the user temporarily uses the device while traveling, the terminal device 16 implements a user interface suitable for the user, ie, displaying one or more functions of the device designated by the user and using the device A user interface for one or more collaboration functions.

Hereinafter, processing related to Examples 1 to 8 will be described.

Processing of switching the display of information about the collaboration function

In an exemplary embodiment, the display of the information on the cooperation function may be switched according to the order in which the device images related to the device are linked to each other. In this case, if the device designated as the cooperation partner device is incapable of performing the cooperation function together with the first device, as in Examples 1 to 8 above, control is performed to present the indication that the cooperation function can be performed together with the first device the guide of the second device. On the other hand, if the user designates a second device capable of performing the cooperation function together with the first device, the display of information on the cooperation function is switched according to the order in which the device images are linked to each other. Hereinafter, this process will be described in detail with reference to FIGS. 36 to 38B .

FIG. 36 shows a cooperation function management table as another example of the cooperation function management information 34. In this cooperation function management table, for example, information representing a combination of device IDs, information representing names (types) of target devices that cooperate with each other, information representing one or more cooperation functions (cooperation function information), information representing links The order information and the information indicating the priority order are associated with each other. The linking order corresponds to the order in which the device images related to the devices are linked to each other. The priority order is the priority order in which the information on the cooperation function is displayed. For example, a device with device ID "A" is a PC, and a device with device ID "B" is an MFP. The cooperation between the PC (A) and the MFP (B) realizes, for example, a scan and transfer function and a print function as a cooperation function. The scan and transfer function is a function of transferring image data generated by scanning with the MFP (B) to the PC (A). The print function is a function of sending data (eg, image data or document data) stored in the PC (A) to the MFP (B) and printing the data by the MFP (B). For example, if linking from MFP(B) to PC(A), that is, if linking from a device image related to MFP(B) to a device image related to PC(A), the priority order of scan and transfer function is "1", and the priority order of the print function is "2". In this case, the information on the scan and transfer function is displayed in preference to the information on the print function. On the other hand, if linking from PC(A) to MFP(B), that is, if linking from a device image related to PC(A) to a device image related to MFP(B), the priority order of printing functions is "1", and the priority order of the scan and transmit function is "2". In this case, the information on the print function is displayed in preference to the information on the scan and transfer function.

37A to 38B each show an example of a screen displayed on the UI unit 46 of the terminal device 16 . For example, assume that MFP (B) and PC (A) are identified. As shown in FIG. 37A, the device display screen 68 is displayed on the UI unit 46 of the terminal device 16, and the device image 70 related to the MFP (B) and the device image 88 related to the PC (A) are displayed on the device display screen 68 on. In this state, the user links the device images representing the target device to each other using a pointer (eg, the user's finger, pen, or stylus). The controller 48 of the terminal device 16 detects the touch of the pointer on the device display screen 68 and detects the movement of the pointer on the device display screen 68 . For example, as indicated by arrow 130, the user touches the device image 70 on the device display screen 68 with the operator and moves the operator to the device image 88 on the device display screen 68, thereby linking the device image 70 to the device image 88. Therefore, the MFP (B) related to the device image 70 and the PC (A) related to the device image 88 are designated as target devices cooperating with each other and the link order is designated. The order in which the device images are linked corresponds to the linking order. The MFP (B) corresponds to the first device, and the PC (A) corresponds to the second device. In the example shown in FIG. 37A, the linking is made from the device image 70 to the device image 88, that is, from the MFP (B) to the PC (A). Information indicating the link order of the devices is transmitted from the terminal device 16 to the server 14 . The controller 48 of the terminal device 16 may cause an image representing the movement trajectory performed by the user to be displayed on the device display screen 68 . After the devices are linked to each other, the controller 48 of the terminal device 16 can replace the trajectory with a predetermined straight line or the like, and can cause the straight line to be displayed on the device display screen 68 .

When the target devices (for example, MFP(B) and PC(A)) cooperating with each other are specified in the above-described manner, the determination unit 38 of the server 14 determines in the cooperation function management table shown in FIG. 36 the cooperation with the PC(A) and the MFP (B) Combination of associated collaborative functions. Therefore, the cooperative function performed by the cooperation between the PC (A) and the MFP (B) is determined. When the user designates the link order of the devices, the determination unit 38 determines the priority order associated with the link order in the cooperation function management table. Specifically, referring to FIG. 36 , since the PC (A) and the MFP (B) are designated as target devices that cooperate with each other, the cooperation functions performed by these devices are the scan and transfer function and the print function. In addition, since the link is made from the MFP (B) to the PC (A) (B→A), the priority order of the scan and transfer function is "1", and the priority order of the print function is "2".

Information about the determined cooperation function and information about the determined priority order are sent from the server 14 to the terminal device 16 . The controller 48 of the terminal device 16 causes the UI unit 46 to display the information on the cooperative function as the information on the candidate cooperative function according to the priority order.

For example, as shown in FIG. 37B , the controller 48 of the terminal device 16 causes the UI unit 46 to display the cooperation function display screen 132 and display information on the cooperation function candidates on the cooperation function display screen 132 . Since the priority order of the scan and transfer function is "1" and the priority order of the print function is "2", the information on the scan and transfer function is displayed in priority (eg, above) the information on the print function. For example, as the information on the scan and transfer function, a description of the scan and transfer function "transfer data scanned by the MFP (B) to the PC (A)" is displayed. Further, as the information on the print function, a description of the print function "Print data in PC (A)" is displayed.

If the user specifies a cooperative function and provides an execution instruction, the specified cooperative function is executed. For example, if the user presses the "Yes" button, the collaborative function associated with the "Yes" button is performed. In addition, a “back” button is displayed on the cooperation function display screen 132 . If the user presses the "back" button, the process of connecting the device stops.

The process of determining the cooperation function and the process of determining the priority order may be performed by the terminal device 16 .

Instead of moving the operator between device images, by drawing circles around the device images, target devices that cooperate with each other can be specified and their link order can be specified. For example, the order of drawing operations corresponds to the linking order. Alternatively, target devices cooperating with each other and their link order may be specified according to a voice instruction provided by the user.

38A and 38B illustrate an example of another operation. For example, as shown in FIG. 38A, the user touches the device image 88 on the device display screen 68 with the operator and moves the operator to the device image 70 in the direction indicated by the arrow 134, thereby linking the device image 88 to the device image 70 . Therefore, the PC (A) related to the device image 88 and the MFP (B) related to the device image 70 are designated as target devices cooperating with each other, and the link order is also designated. In this example, the linking is made from device image 88 to device image 70, ie, from PC (A) to MFP (B). Referring to the cooperation function management table shown in FIG. 36, the priority order of the print function is "1", and the priority order of the scan and transfer function is "2". In this case, as shown in FIG. 38B , the information on the print function is displayed in priority over (eg, above) the information on the scan and transfer function on the cooperation function display screen 136 .

As described above, the device images related to the device are linked to each other, thereby determining a cooperative function using the function of the device. According to the order in which the images are linked to each other, that is, the order in which the devices are linked to each other, the display order of the information on the cooperative function changes. The linking order of the devices is also regarded as the order in which functions are used in the respective devices or the order in which data is moved between the devices cooperating with each other. The operation of linking devices (operation of linking images) is also regarded as an operation of specifying the order of function use or the order of data movement. Therefore, as a result of changing the display order of the information about the collaboration function according to the link order, the information about the collaboration function that the user expects to use is preferentially displayed. In other words, information on collaboration functions that the user is more likely to use is preferentially displayed. For example, if linking from MFP(B) to PC(A), the user is expected to use the collaboration function of "first use the function of MFP(B), then transfer data from MFP(B) to PC(A)". On the other hand, if linking from PC(A) to MFP(B), users are expected to use the collaboration of "first use the functions of PC(A), then transfer data from PC(A) to MFP(B)" Function. Therefore, as a result of changing the display order of the information about the collaboration function according to the link order of the images, the information about the collaboration function that the user is more likely to use is preferentially displayed. In addition, the order of function use or the order of data movement is specified without a special operation other than the operation of linking the device images, and information on the cooperative function expected to be used by the user is displayed.

The above-described display switching processing can be applied to the case of using function images related to functions. For example, according to the order in which the function image related to the first function and the function image related to the second function are designated, the display of the information related to the cooperation function is switched.

The above-described display switching process can be applied to the information on the cooperation function displayed in the candidate list (for example, see FIG. 26 ) according to Example 6 and the like. That is, if the MFP (B) is designated as the first device, the PC (A) is displayed in the candidate list as a candidate second device (candidate cooperation partner), and the information on the plurality of cooperation functions is The information on the related cooperative functions (information on the cooperative functions executable by the MFP (B) and the PC (A)) is displayed in the order shown in FIG. 37B . On the other hand, if the PC (A) is designated as the first device, the MFP (B) is displayed in the candidate list as a candidate second device (candidate cooperation partner), and the information on the plurality of cooperation functions is displayed as a B) The information about the cooperation function is displayed in the order shown in FIG. 38B.

Collaborative processing using partial images

The function of the device assigned to the collaborative function may vary according to the location in the device image relative to the device. When the user designates a specific position in the device image, information on the cooperation function using the function corresponding to the specific position is preferentially displayed. Hereinafter, this process will be described in detail.

FIG. 39 shows an example of the device function management table. The data of the device function management table is stored in the server 14 as the device function management information 32 . In the device function management table, for example, the device ID, the information indicating the name (eg, type) of the device, the information indicating the position in the device image, the information (function information) indicating the function corresponding to the position, and the image ID are each other association. The position in the device image is a specific position (specific part) in the device image related to the device, eg, a specific position in the device image schematically representing the device or a specific position in the device image captured by a camera. Different functions are associated with each specific location in the device image.

40A and 40B each show an example of a screen displayed on the UI unit 46 of the terminal device 16 . For example, assume that MFP (B) and PC (A) are identified. As shown in FIG. 40A , a device display screen 68 is displayed on the UI unit 46 of the terminal device 16 , and device images 70 and 88 are displayed on the device display screen 68 . For example, in the device image 70, a specific position (partial image 70a) corresponding to the body portion of the MFP (B) is assigned a print function. In the device image 70, specific positions (partial image 70b) corresponding to the document cover, document glass, and automatic document feeder of the MFP (B) are assigned a scan function. In the device image 70, a specific position (partial image 70c) corresponding to the post-processing device is assigned a bookbinding function. The binding function is a function of binding output sheets. In the device image 88, a specific position (partial image 88a) corresponding to the main body of the PC (A) is assigned a data storage function. In the device image 88, a screen display function is assigned to a specific position (partial image 88b) corresponding to the display portion of the PC (A). The data storage function is a function of storing data received from another device in the PC (A). The screen display function is a function to display data received from another device on the PC (A).

The controller 48 of the terminal device 16 may cause the name of the function (eg, print, scan, etc.) assigned to a particular location in the device image to be displayed on the device display screen 68 . Therefore, the user is provided with information clearly representing the correspondence between the specific position and the function. Of course, the name of the function is not necessarily displayed.

When the user designates a position in the device image to which a function is assigned, the functions assigned to the designated position are designated as target functions that cooperate with each other. The user links a specific position (partial image) to which a function is assigned in the device image representing the target device cooperating with each other using the pointer. For example, as indicated by arrow 138, the user touches the partial image 70b on the device display screen 68 with the operator and moves the operator to the partial image 88b, thereby linking the partial image 70b to the partial image 88b. Therefore, the MFP (B) related to the device image 70 including the partial image 70b and the PC (A) related to the device image 88 including the partial image 88b are designated as target devices cooperating with each other, and assigned to the scanning of the partial image 70b The function and the screen display function assigned to the partial image 88b are designated. In addition, the chaining order can be specified through the chaining operation. In this case, the order in which the partial images are linked corresponds to the linking order. In the example shown in FIG. 40A, the linking is performed from the partial image 70b to the partial image 88b, that is, from the MFP (B) to the PC (A). The scan function and the screen display function are designated as functions for the cooperation function. Information representing the link order of the devices and information representing the specific position designated by the user in the device image are transmitted from the terminal device 16 to the server 14 .

When the target devices (for example, PC(A) and MFP(B)) cooperating with each other are identified, the determination unit 38 of the server 14 determines in the cooperation function management table shown in FIG. ) is a collaboration function implemented by collaboration between . Further, the determination unit 38 refers to the device function management table shown in FIG. 39 to determine the function assigned to the specific position designated by the user in the device image. Furthermore, among the cooperative functions realized by the cooperation between the PC (A) and the MFP (B), the determination unit 38 assigns higher priority to the cooperative functions that use the functions assigned to the position designated by the user, and to those that do not use all the functions A lower priority is assigned to cooperating functions of the aforementioned functions.

The information on the cooperative function determined in the above-described manner and the information indicating the priority order are transmitted from the server 14 to the terminal device 16 . The controller 48 of the terminal device 16 causes the UI unit 46 to display the information on the cooperative function according to the priority order as the information on the candidate cooperative function.

For example, as shown in FIG. 40B , the controller 48 of the terminal device 16 causes the display of the UI unit 46 to display the cooperation function display screen 140 and displays information on the candidate cooperation function on the cooperation function display screen 140 . Since the user specifies the scan function and the screen display function in this order, with regard to the cooperative function performed by the cooperation between the scan function and the screen display function, the information on the scan, transfer and display function takes precedence over the information on other cooperative functions (for example, above it). For example, information on the scan, transfer and display function takes precedence over information display on the scan, transfer and store function performed through cooperation between the scan function and the data storage function. The scan, transfer, and display function is a function of transferring data generated by scanning with the MFP (B) to the PC (A) and displaying the data on the screen of the PC (A). The scan, transfer, and store function is a function of transferring data generated by scanning by the MFP (B) to the PC (A) and storing the data in the PC (A). In the example shown in FIG. 40B , descriptions of the respective cooperation functions are displayed as information on the respective cooperation functions.

According to the cooperation process using the partial image, in the case where each target device cooperating with each other has a plurality of functions, the functions are individually designated, and information on the cooperation functions using the designated functions is preferentially displayed. Therefore, priority is given to displaying collaboration functions that are intended for users to use.

The cooperative function may be a function using a combination of parts of the device, a function using a combination of the entire device and parts of the device, or a function using a combination of the entire device.

Collaborative processing using partial images is applicable to the case of using functional images related to functions. For example, different functions are assigned to positions in the function image, and a cooperative function using the function assigned to the user-specified position is determined.

The above-described Examples 1 to 8 are also applicable to collaborative processing using partial images. For example, if the user designates a partial image included in the first image related to the first device, control may be performed to present a guide indicating a first step that can perform the collaborative function together with the function assigned to the part related to the partial image The whole or part of two devices. As another example, if the user specifies the entire first image related to the first device, control may be performed to present guidance indicating the portion of the second device capable of performing cooperative functions with the first device. To present the guidance, the whole or part of the second device may be displayed while being included in the candidate list described in Example 6 above. If the user specifies the whole or part of the first device, then guidance may be presented indicating the whole or part of the second device capable of performing cooperative functions with the whole or part of the first device. If the whole or part of the second device that cannot perform the cooperative function with the whole or part of the first device specified by the user is specified, then guidance may be presented indicating the first device that is capable of performing the cooperative function with the whole or part of the first device. The whole or part of two devices. Hereinafter, these processes will be described in detail.

For example, when specifying the first image (the entire image) related to the first device, the controller 48 of the terminal device 16, under the control of the controller 36 of the server 14, presents guidance indicating one or more functions of the second device, The function is capable of performing a cooperative function with the first device. More specifically, the controller 48 of the terminal device 16 presents a guide indicating one or more partial images related to one or more parts included in the second device (the one or more part images in the second image related to the second device). one or more partial images), the one or more partial images related to the one or more parts are assigned a function capable of performing a cooperative function with the first device. For example, as in Examples 1 to 8, the controller 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display an image depicting an arrow serving as a guide, present the guide with sound, or cause the UI unit 46 to display a character serving as a guide string. Referring to FIG. 40A , if the user designates the device image 70 related to the MFP (B) as the first image, a guide is presented, which indicates the connection with the computer assigned the function of the PC (A) capable of executing the cooperative function together with the MFP (B). Part of the relevant part of the image. For example, if the screen display function of the PC (A) is a function capable of executing a cooperative function together with the MFP (B), a guide indicating a partial image 88b related to the screen display function is presented. For example, an arrow linking the device image 70 and the partial image 88b related to the MFP(B) to each other is displayed, a guide indicating the screen display function is presented with sound, a character string indicating the screen display function is displayed, or the partial image 88b is displayed so that The partial image 88b is distinguishable from another partial image. Guidance may be presented when the user designates the device image 70 as the first image or when the user designates a part of the PC (A) to which a function that cannot perform the cooperative function together with the MFP (B) is assigned.

As another example, if a portion of the image included in the first image related to the first device is specified, the controller 48 of the terminal device 16 may, under the control of the controller 36 of the server 14, present guidance that can be correlated with the instructions assigned to the The second device (the whole device) that performs the cooperative function together with the function of the part (the function of the first device) related to the part of the image. Referring to FIG. 40A , for example, if the user designates a partial image 70a related to the MFP (B), a guide indicating a second device capable of performing a cooperation function together with a print function assigned to the part related to the partial image 70a is presented Related second image. For example, if the PC (A) is a device capable of performing a cooperative function together with the MFP (B), a guide indicating a device image 88 related to the PC (A) is presented. For example, an arrow linking the partial image 70a and the device image 88 to each other is displayed, a guide indicating PC(A) is presented with a sound, a character string indicating PC(A) is displayed, or the device image 88 is displayed so that the device image 88 can be combined with Another device image is differentiated. Guidance may be presented when the user designates a partial image included in the device image 70 or when the user designates a device image related to a device that cannot perform a cooperative function together with a function related to the partial image.

As another example, if a partial image (first partial image) included in the first image related to the first device is specified, the controller 48 of the terminal device 16 presents a guide under the control of the controller 36 of the server 14, which guide indicates One or more functions of the second device capable of performing the cooperative function together with the function assigned to the part related to the first part image (the function of the first device). More specifically, the controller 48 of the terminal device 16 presents a guide indicating one or more partial images related to one or more parts included in the second device (all of which are in the second image related to the second device). including one or more partial images, referred to as second partial images), i.e., one or more second partial images related to the one or more parts that are assigned a One or more functions of the collaborative functions are performed together with the functions of the parts related to the first part of the image. Referring to FIG. 40A, for example, if the user designates a partial image 70a related to the MFP (B), a guide is presented that indicates a second partial image related to such a part, which is assigned with the ability to be assigned to the partial image The printing function of the part concerned with 70a performs the function of the PC (A) of the cooperative function together. For example, if the screen display function of the PC (A) is a function capable of executing a cooperative function together with the print function, a guide indicating a partial image 88b related to the screen display function is presented. For example, an arrow linking the partial image 70a and the partial image 88b to each other is displayed, a guide indicating the screen display function is presented using sound, a character string indicating the screen display function is displayed, or the partial image 88b is displayed so that the partial image 88b can be linked to another part images are differentiated. Guidance may be presented when the user designates a partial image included in the device image 70 or when the user designates a partial image related to a function that cannot perform a cooperative function together with a function related to the partial image.

Three or more partial images can be designated, and thus, the user can designate three or more functions. For example, if the user specifies two partial images (two functions: a first function and a second function), a guide may be presented indicating that a function (third function) capable of performing a cooperative function together with these two functions is assigned part. In this case, the determination unit 38 of the server 14 may change the third function to be presented and guided according to the order in which the partial images related to the first function and the partial images related to the second function are designated.

Multiple functions of the same device can be designated as target functions that cooperate with each other. For example, the user may designate the screen display function and the data storage function of the PC (A) as target functions that cooperate with each other, or the user may designate the scan function of the MFP (B) and the screen display function and the data storage function of the PC (A) as each other Collaborative target function. Also in this case, a guide is presented that indicates that a portion of a function (eg, a second function) capable of performing the cooperative function together with the first-specified function (eg, the first function) is assigned. The determination unit 38 of the server 14 may determine the cooperative function using the respective functions according to the designated order and may assign a higher priority to the cooperative function.

Another example of collaborative processing using partial images

Hereinafter, another example of cooperative processing using partial images will be described with reference to FIGS. 41 and 42 .

FIG. 41 shows an example of the device function management table. The data of the device function management table is stored in the server 14 as the device function management information 32 . In the device function management table, for example, a device ID, information representing the name of the device (eg, the type of the device), information representing the name of the part of the device (eg, the type of the part), as part identification for identifying the part The part ID of the information, the information representing the function assigned to the part (the function of the part), and the part image ID for identifying the part image related to the part are associated with each other. The partial image is an image representing the appearance of the part of the device obtained by being photographed by the camera. Of course, a partial image schematically representing a part of the device may be associated with that part. For example, different functions are assigned to various parts of the device.

Specifically, the screen display function is assigned to the display section of the PC (A), and the information indicating the screen display function is associated with the partial image ID of the partial image related to the display section. The screen display function is a function to display information on the PC (A). The data storage function is assigned to the main body of the PC (A), and the information indicating the data storage function is associated with the partial image ID of the partial image related to the main body. The data storage function is a function to store data in the PC (A).

The printing function is assigned to the main body of the MFP (B), and the information indicating the printing function is associated with the partial image ID of the partial image related to the main body. The scanning function is assigned to the reading section of the MFP (B) (for example, the parts corresponding to the document cover, document glass, and automatic document feeder of the MFP (B)), and the information indicating the scanning function is related to the reading section. Part image ID association for part images. The bookbinding function is assigned to the post-processing apparatus of the MFP (B), and the information indicating the bookbinding function is associated with the partial image ID of the partial image related to the post-processing apparatus. The binding function is a function of binding output sheets.

The functions assigned to the parts of the device are determined (identified) using, for example, markerless AR technology. For example, if a part of the apparatus is photographed by a camera (eg, the camera 42 of the terminal device 16 ), appearance image data representing the part is sent from the terminal device 16 to the server 14 . The determination unit 38 of the server 14 determines (identifies) the function associated with the appearance image data in the device function management table. Therefore, the function assigned to the photographed portion is determined (identified). For example, if the main body of the MFP (B) is photographed by the camera 42 , image data representing the appearance of the main body of the MFP (B) is transmitted from the terminal device 16 to the server 14 . The determination unit 38 of the server 14 determines the print function associated with the appearance image data in the device function management table. Therefore, it is determined that the function assigned to the body portion of the MFP (B) is the printing function.

Of course, the function assigned to the part of the device can be determined (identified) using marker-based AR technology. For example, various parts of the device are provided with indicia such as a two-dimensional barcode obtained by encoding part identification information (eg, part ID) for identifying the part. If a marker on a part is photographed by a camera and the marker-based AR technology is applied thereto, part identification information (eg, part ID) of the part is obtained. The application of the marker-based AR technology may be performed by the terminal device 16 or the server 14 . After thus obtaining the partial identification information, the determination unit 38 of the server 14 determines (identifies) the function associated with the partial identification information (eg, the partial ID) in the device function management table.

FIG. 42 shows an example of the cooperation function management table. The data of the cooperative function management table is stored in the server 14 as the cooperative function management information 34 . The cooperative function management table is information indicating cooperative functions each of which uses the functions of a plurality of parts. In the cooperation function management table, for example, information indicating a combination of parts of a device, information indicating a combination of part IDs, and information indicating a cooperation function using functions of a plurality of parts included in the combination are associated with each other. Of course, in the cooperation function management table, information representing a combination of a part of the device and the entire device and information representing a cooperation function using the function of the part of the device and the function of the entire device may be associated with each other.

Specifically, the print function as the cooperation function is assigned to the combination of the display portion of the PC (A) and the main body portion of the MFP (B), and the information representing the print function as the cooperation function is combined with the information representing the display portion of the PC (A). The information associated with the combination of the part ID and the part ID of the main body of the MFP (B). For example, the print function as a cooperation function is a function of sending data stored in the PC (A) to the MFP (B) and printing the data by the MFP (B).

The printing function as the cooperative function is assigned to the combination of the main body of the MFP (B) and the main body of the projector (C), and the information indicating the printing function as the cooperative function and the part ID indicating the main body of the MFP (B) It is associated with the information of the combination of the part ID of the main body of the projector (C). For example, the printing function as the cooperation function is a function of transmitting data projected by the projector (C) to the MFP (B) and printing the data by the MFP (B).

The scan-and-project function as the cooperative function is assigned to the combination of the reading section of the MFP (B) and the main body section of the projector (C), and the information indicating the scan-and-project function as the cooperative function is combined with the information indicating the MFP (B) The information of the combination of the part ID of the reading part and the part ID of the main body part of the projector (C) is associated. For example, the scan-and-project function as a cooperation function is a function of transmitting data generated by scanning by the MFP (B) to the projector (C) and projecting the data by the projector (C).

The cooperative function may be a function using functions of a plurality of sections included in the same apparatus, or may be a function using functions of sections included in a plurality of different apparatuses. A collaborative function may be a function that uses the functions of three or more parts.

For example, after determining (identifying) parts of the device (eg parts of multiple different devices or parts of the same device) using marker-based AR techniques or markerless AR techniques, the determination unit 38 of the server 14 The cooperation function associated with the combination of the identified plurality of parts is determined (identified) in the cooperation function management table. Therefore, a cooperative function using the functions of a plurality of identified (eg, photographed) parts is determined (identified). For example, if the body part of the MFP (B) and the body part of the projector (C) are photographed by the camera 42 of the terminal device 16 and if the body part of the MFP (B) and the body part of the projector (C) are recognized, the server The determination unit 38 of 14 determines, in the cooperation function management table, the printing function or the like as the cooperation function associated with the combination of the main body of the MFP (B) and the main body of the projector (C).

The above-described Examples 1 to 8 are also applicable to such cooperative processing. For example, if the first part of the apparatus is determined (identified), the controller 48 of the terminal device 16, under the control of the controller 36 of the server 14, presents a guide indicating the apparatus capable of performing the cooperative function together with the function assigned to the first part the second part.

Specify target devices that cooperate with each other by overlaying device images

Target devices that cooperate with each other can be specified by superimposing a plurality of device images on each other. Hereinafter, this process will be described with reference to FIGS. 43A , 43B, 43C, 44A, and 44B. 43A , 43B, 43C, 44A, and 44B each show an example of a screen displayed on the UI unit 46 of the terminal device 16 .

For example, assume that MFP (B) and PC (A) are identified. As shown in FIG. 43A , a device display screen 68 is displayed on the UI unit 46 of the terminal device 16 , and device images 70 and 88 related to the recognized device are displayed on the device display screen 68 . In this state, the user superimposes the device image related to the first device on the device image related to the partner device (second device) using a pointer (eg, the user's finger, pen or stylus). For example, as shown in FIG. 43B , the user designates the device image 70 with the operator and superimposes the device image 70 on the device image 88 , as indicated by arrow 142 . For example, the user superimposes the device images on each other by performing a drag and drop operation. Specifically, the user drags the device image 70 and releases it at the location where the device image 70 is superimposed on the device image 88 . This drag-and-drop operation is a technique according to the prior art. Alternatively, the device images to be superimposed on each other may be designated according to a voice instruction provided by the user. For example, the device images 70 and 88 may be designated as target device images and may be superimposed on each other according to voice instructions provided by the user.

As a result of superimposing the device images 70 and 88 on each other, the MFP (B) related to the device image 70 and the PC (A) related to the device image 88 are designated as target devices that cooperate with each other. For example, the first designated MFP (B) corresponds to the first device. If the second designated PC (A) is a device that cannot perform the cooperative function together with the MFP (B), as in Examples 1 to 8 above, a presentation indicating a device capable of performing the cooperative function together with the MFP (B) is presented. guide.

The controller 48 of the terminal device 16 can cause the device image being dragged to be displayed on the UI unit 46 in a recognizable manner. For example, the device image being dragged may be displayed translucently or in a specific color.

If the device image 70 is superimposed on the device image 88 and if the PC (A) can execute the cooperative function together with the MFP (B), the confirmation screen 144 is displayed on the UI unit 46 of the terminal device 16 as shown in FIG. 43C . The confirmation screen 144 is a screen for confirming whether or not the designated devices are caused to cooperate with each other. If the user provides a collaboration instruction on the confirmation screen 144 (eg, if the user presses the "Yes" button), information about the collaboration function is displayed on the UI unit 46 of the terminal device 16 .

For example, as shown in FIG. 44A , the controller 48 of the terminal device 16 causes the UI unit 46 to display the cooperation function display screen 146 and display information on the cooperation function candidates on the cooperation function display screen 146 . By causing the PC (A) and the MFP (B) to cooperate with each other, for example, a scan and transfer function and a print function are realized. Therefore, information on the scan and transfer function and information on the print function are displayed on the cooperation function display screen 146 .

If the user specifies the cooperation function and the user provides an execution instruction, a connection request is sent from the terminal device 16 to the target apparatuses that cooperate with each other. As shown in FIG. 44B , while the connection request is being processed, a waiting screen 148 is displayed on the UI unit 46 of the terminal device 16 . When the connection between the terminal device 16 and the target device is successfully established, the designated cooperation function is executed.

As described above, device images related to the devices are superimposed on each other, thereby determining a cooperative function using the functions of the device. Therefore, functions can be made to cooperate with each other without special operations other than image operations, and cooperation between functions can be performed with a simple operation. Also in this case, guidance indicating the second device capable of performing the cooperative function together with the first device is presented, and therefore, user convenience in using the cooperative function can be increased compared to the case where the guidance is not presented.

The collaboration function may be determined by superimposing the partial image on the device image or the partial image. This process will be described with reference to FIGS. 45A and 45B . 45A and 45B each show an example of a screen displayed on the UI unit 46 of the terminal device 16 .

As in the above-described collaborative processing using partial images, the functions of the device vary depending on the position in the device image related to the device. By superimposing the partial image included in the device image on the partial image included in the same or different device images, a cooperative function using functions related to the two partial images is determined. Hereinafter, this processing will be described in detail.

For example, assume that MFP (B) and PC (A) are identified. As shown in FIG. 45A , a device display screen 68 is displayed on the UI unit 46 of the terminal device 16 , and device images 70 and 88 are displayed on the device display screen 68 . For example, each of the partial images 70a, 70b, 70c, 88a, and 88b is displayed as an image that can be moved separately from the other partial image.

If the user designates a partial image and if the partial image is superimposed on another partial image, a cooperation function using the functions related to the two partial images is determined, and information on the cooperation function is displayed on the UI unit 46 of the terminal device 16 superior. This determination process may be performed by the determination unit 38 of the server 14 or the terminal device 16 .

For example, as indicated by the arrow 150 in FIG. 45B, if the user drags the partial image 70b with the operator and drops it on the partial image 88b, the MFP (B) related to the device image 70 including the partial image 70b and the The PC (A) related to the device image 88 including the partial image 88b is designated as a target device to cooperate with each other, and the scan function assigned to the partial image 70b and the screen display function assigned to the partial image 88b are also designated as targets to cooperate with each other Function.

In the server 14, the functions assigned to the respective partial images are managed. For example, identification information for identifying a partial image, function information representing a function associated with the partial image, and cooperation function information representing a cooperation function performed through cooperation between functions are stored in the server 14 in association with each other. If a partial image is selected on the device display screen 68 and superimposed on another partial image, identification information representing the partial images superimposed on each other is transmitted from the terminal device 16 to the server 14 . In the example shown in FIG. 45B , the identification information representing the partial images 70b and 88b is transmitted from the terminal device 16 to the server 14. The determination unit 38 of the server 14 determines the functions assigned to the partial images 70b and 88b based on the identification information, and determines a cooperative function using the functions. Information about the cooperative function is transmitted from the server 14 to the terminal device 16 and displayed on the terminal device 16 .

As described above, in the case where each target device cooperating with each other has a plurality of functions, a function is selected in each target device, and information on the cooperative function using the designated function is preferentially displayed. Therefore, priority is given to displaying collaboration functions that are intended for users to use.

The priority order of the collaboration function display can be changed according to the order in which the partial images are superimposed on each other. In this case, the information on the cooperation function using the function related to the superimposed partial image is preferentially displayed.

In the case where the partial images are superimposed on each other, the first designated partial image corresponds to the first image, and the function related to the partial image corresponds to the first function. The second specified partial image (the partial image on which the first image is superimposed) corresponds to the second image, and the function related to the partial image corresponds to the second function. If the second specified function cannot perform the cooperative function with the first specified function, then as in Examples 1 to 8 above, a guide is presented indicating the function capable of performing the cooperative function with the first specified function . In the above example, if the screen display function related to the second specified partial image 88b cannot perform the cooperative function together with the scan function related to the first specified partial image 70b, a guide is presented which indicates that the A function that performs a collaborative function together with a specified scan function. In this case, a guide may be presented indicating the entire device (eg, the device image itself) capable of performing the cooperative function with the first specified function (eg, the scan function), or may be presented indicating that there is a Guidance of a portion (eg, a portion of an image) of a device that together perform the functions of the cooperating function with the specified functions.

Processing for switching the display of the single device function and the cooperation function

In an exemplary embodiment, control of switching between display of a function using a single device alone (hereinafter referred to as a "single device function") and display of a cooperative function can be performed.

For example, if only one device is recognized within a predetermined recognition period, the controller 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display one or more functions regarding this one device (eg, the image forming apparatus 10 ) information as single device function information. The length of the identification period can be changed by the user. The device may be identified by applying AR technology or other technology. The process of identifying the device may be performed by the server 14 or the terminal device 16 . For example, the starting point of the identification period may be the point in time when this one device is identified or the point in time specified by the user (eg, the point in time when the identification process starts).

For example, if another device is not recognized within the recognition period from the point in time when this one device is recognized, the controller 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display one or more information about this one device Information of a plurality of functions is regarded as single-device function information. In this case, this one device is treated as the device identified within the identification period. The information on the device may be information transmitted from the server 14 to the terminal device 16 or information stored in the terminal device 16 in advance.

As another example, if only one device is identified within the identification period starting from the point in time specified by the user, the controller 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display one or more functions regarding this one device information as single device function information.

As another example, if an instruction to display a single device function is provided after identifying a device, the controller 48 of the terminal device 16 may cause the UI unit 46 of the terminal device 16 to display information about one or more functions of the one device as a single device. Device function information. The controller 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to always display or display a button image for providing an instruction to display a single-device function in the case of photographing a device (or in the case of recognizing a device). If the user presses the button image, the controller 48 causes the UI unit 46 to display information about one or more functions of this one device.

When the identification period elapses, the controller 48 of the terminal device 16 may cause the UI unit 46 of the terminal device 16 to display a confirmation screen. For example, the confirmation screen is a screen used by the user to provide an instruction to extend the identification period. If the user provides an instruction to extend the identification period through the confirmation screen and if another device is not photographed within the extended period, the controller 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display one or more of the identified devices function information.

The display control of the single-device function information will be further described. For example, suppose the device is identified using marker-based AR technology or marker-free AR technology. For example, if only one device is photographed within a predetermined photographing period, the controller 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display information on one or more functions of this one device as single-device function information. The starting point of the photographing period may be the time point when this one device is photographed or the time point specified by the user (eg, the photographing starting point). The length of the shooting period can be changed by the user. After the one device is photographed, the one device is identified using marker-based AR technology or marker-free AR technology. The identification process may be performed by the server 14 or the terminal device 16 .

For example, if another device is not photographed within the photographing period from the time point when this one device is photographed, the controller 48 of the terminal device 16 causes the UI unit 46 of the terminal device 16 to display one or more information about this one device The information of each function is regarded as single device function information. In this case, this one device is handled as a device photographed during the photographing period.

As another example, if only one apparatus is photographed within the photographing period from the user-specified point in time, the controller 48 of the terminal device 16 causes the UI unit 46 of the terminal apparatus 16 to display information about one or more functions of this one apparatus. information as single device function information.

As another example, if an instruction to display a single-device function is provided after photographing a device, the controller 48 of the terminal device 16 may cause the UI unit 46 of the terminal device 16 to display information about one or more functions of the one device as a single device. Device function information.

For example, if only one device is photographed during the photographing period (eg, if the second device is not photographed within the photographing period from the point in time when the first device was photographed or if within the photographing period from the starting point specified by the user) photographing only one device), the controller 48 of the terminal device 16 transmits the image data generated by photographing to the server 14 . The shooting period may be measured by the controller 48 or a timer. The determination unit 38 of the server 14 determines (identifies) the device based on the image data and determines one or more functions of the device. Information about the one or more functions is sent from the server 14 to the terminal device 16 and displayed on the UI unit 46 of the terminal device 16 . Of course, the server 14 may manage the time in place of the terminal device 16 and may transmit information about one or more functions of the identified device to the terminal device 16 .

When the photographing period elapses, the controller 48 of the terminal device 16 may cause the UI unit 46 of the terminal device 16 to display a confirmation screen. For example, the confirmation screen is a screen used by the user to provide an instruction to extend the shooting period. If the user provides an instruction to extend the photographing period through the confirmation screen and if another device is not photographed within the extended period, the controller 48 of the terminal device 16 transmits the image data obtained by the photographing to the server 14 and causes the UI unit 46 displays one or more functions of this one device. The length of the extended period can be changed by the user.

As another example, if an instruction to display a single-device function is provided after photographing one device, the controller 48 of the terminal device 16 may transmit image data generated by photographing to the server 14, and thus, may receive information about the photographed information from the server 14. Information about one or more functions of the device.

As another example, each time image data is generated by capturing a device, the controller 48 of the terminal device 16 may send the image data to the server 14 and, thus, may receive from the server 14 one or more functions regarding the captured device Information. In this case, if only one device is photographed during the photographing period, the controller 48 of the terminal device 16 causes the UI unit 46 of the terminal 16 to display information on this one device as single-device function information.

On the other hand, if multiple devices are identified within the identification period, the controller 36 of the server 14 executes the cooperative function mode. In the cooperative function mode, as in Examples 1 to 8 above, guidance indicating a second device capable of performing a cooperative function together with the first device is presented.

For example, if the second device is identified within the identification period from the point in time when the first device is identified, the cooperative function mode is executed. In this case, the first device is also treated as the device identified within the identification period. Furthermore, if the second device is identified within the identification period from the time the first device is identified, the controller 36 of the server 14 may set a new identification period from the time the second device is identified. The same applies for subsequent periods, ie if the third device is identified within the new identification period, another new identification period is set.

As another example, if a plurality of devices are photographed within an identification period from a time point designated by the user, the cooperative function mode may be executed.

As another example, if an instruction to display a collaborative function is provided after identifying a plurality of devices, the collaborative function mode may be executed. The controller 48 of the terminal device 16 causes the UI unit 46 to display at all times or, in the case of photographing a plurality of devices (or in the case of recognizing a plurality of devices), a button image for providing an instruction to display one or more cooperative functions . If the user presses the button image, the cooperative function mode is executed.

As another example, if the second device is identified during a period in which no instructions to perform the functions of the first device are provided after the first device is identified, the cooperative function mode may be performed.

The execution of the collaborative functional mode will be further described. For example, suppose multiple devices are identified using marker-based AR technology or marker-free AR technology. For example, if a plurality of devices are photographed within a predetermined photographing period, the cooperative function mode is executed. For example, if the second device is photographed within a photographing period from the point in time when the first device is photographed, the cooperative function mode is executed. In this case, the first device is also treated as a device that was photographed during the photographing period. Further, if the second device is photographed within the photographing period from the time point when the first device is photographed, the controller 36 of the server 14 may set a new photographing period from the time point when the second device is photographed. The same applies to subsequent periods, ie, if the third device is photographed within a new photographing period, another new photographing period is set.

As another example, if a plurality of devices are photographed within a photographing period from a time point designated by the user, the cooperative function mode may be executed.

For another example, if an instruction to display a cooperative function is provided after photographing a plurality of devices, the cooperative function mode can be executed.

As another example, if the second device is photographed during a period in which no instruction to perform the function of the first device is provided after photographing the first device, the cooperative function mode may be performed.

As described above, in the process of switching the display of the single-device function and the cooperative function, if one device is identified (eg, photographed), information on one or more functions of this one device is displayed, and if it is identified (eg, photographed) shooting) multiple devices, then execute the cooperative function mode. Therefore, information on functions that can be performed with the identified (eg, photographing) device is provided to the user, which may be convenient for the user.

Since the single-device function or the cooperative function becomes available only by recognizing the device by applying the AR technology, each function is available through a simple operation compared to the case where the user manually sets the function for executing the function, and the user's time and effort can be reduced.

In an exemplary embodiment, the device image related to the identified device and the device image superimposed on each other may be three-dimensionally displayed to be distinguished from the background image. That is, these images can be displayed as three-dimensional images. For example, the background image is displayed two-dimensionally, and the device image is displayed three-dimensionally. Therefore, the visibility of the device image can be increased. In addition, the color of the device image designated by the user may be changed, or the designated device image may blink, so that the designated device image can be distinguished from other device images.

According to an exemplary embodiment, a collaboration function using the functions of target devices that cooperate with each other is determined by applying AR technology, and information on the collaboration function is displayed. Therefore, even if the user cannot know from the appearance of the device which cooperation function can be performed by the target devices cooperating with each other, the information on the cooperation function is provided to the user. Furthermore, by having multiple devices cooperate with each other, functions that cannot be performed by a single device become available, which can be convenient. In addition, only by applying AR technology to identify target devices that cooperate with each other, the cooperation function becomes available. Therefore, compared with the case where the user manually sets for executing the cooperation function, the cooperation function becomes available through a simple operation, and the user's time and effort can be reduced.

For example, each of the image forming apparatus 10 , the server 14 and the terminal apparatus 16 is realized by cooperation between hardware and software resources. Specifically, each of the image forming apparatus 10 , the server 14 and the terminal apparatus 16 includes one or more processors (not shown) such as a central processing unit (CPU). The one or more processors read and execute programs stored in a storage device (not shown), thereby realizing the functions of the respective units of the image forming apparatus 10 , the server 14 , and the terminal apparatus 16 . The program is stored in the storage device through a recording medium such as a compact disc (CD) or a digital versatile disc (DVD) or through a communication path such as a network. Alternatively, the respective units of the image forming apparatus 10, the server 14, and the terminal apparatus 16 may be implemented by hardware resources such as a processor, an electronic circuit, or an application specific integrated circuit (ASIC). Devices such as memory can be used for implementation. Alternatively, the respective units of the image forming apparatus 10, the server 14, and the terminal apparatus 16 may be implemented by a digital signal processor (DSP) or a field programmable gate array (FPGA).

The foregoing description of the exemplary embodiments of the present invention has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Obviously, many modifications and variations will be apparent to those skilled in the art. The embodiment was chosen and described in order to best explain the principles of the invention and its practical application, to thereby enable others skilled in the art to understand the invention for various embodiments and with various modifications as are suited to the particular use contemplated. It is intended that the scope of the present invention be defined by the appended claims and their equivalents.

Claims (22)

1. An information processing apparatus, comprising:

a controller that performs control to present guidance indicating a second device capable of performing the cooperative function together with the first device if a first image related to the first device required to perform the cooperative function is specified,

wherein the controller monitors at least one of an operation state of the first device, an environment in which the first device is installed, and an update state of the first device, and determines the availability of the cooperation function based on the monitoring result.

2. The information processing apparatus according to claim 1, wherein the controller performs control to present the guidance if an image related to a device incapable of performing the cooperative function with the first device is further specified.

3. The information processing apparatus according to claim 2, wherein the controller performs control to present the guidance if an operation of linking the first image and an image related to a device that cannot perform the cooperative function with the first device to each other is performed.

4. The information processing apparatus according to claim 2, wherein the controller performs control to present the guidance if the first image and an image related to a device that cannot perform the cooperative function with the first device are superimposed on each other.

5. The apparatus according to any one of claims 1 to 4, wherein if a partial image included in the first image is specified, the controller performs control to present the guidance indicating the second device that can perform the cooperative function together with a function corresponding to the partial image.

6. The apparatus according to claim 1, wherein as the control to present the guidance, the controller performs control to display a candidate list showing information on one or more second devices capable of performing the cooperative function.

7. The apparatus according to claim 6, wherein if a second device is specified from among the one or more second devices on the candidate list, the controller performs control to display information on the cooperative function using the specified second device.

8. The information processing apparatus according to claim 1, wherein the controller performs control to display the cooperative function while changing the cooperative function according to an order in which the first device and the second device are specified.

9. The information processing apparatus according to claim 1, wherein if the first device and the second device are specified, the controller further performs control to present guidance indicating a third device capable of performing a cooperative function together with the first device and the second device.

10. The information processing apparatus according to claim 9, wherein the controller performs control to present the guidance while changing the third device according to an order in which the first device and the second device are specified.

11. An information processing apparatus, comprising:

a controller that performs control to present guidance indicating a second function capable of performing the cooperative function together with the first function if a first image related to the first function required to perform the cooperative function is specified,

wherein the controller monitors at least one of an operation state of a device corresponding to the first and second functions, an environment in which the device corresponding to the first and second functions is installed, and an updated state of the device corresponding to the first and second functions, and determines the availability of the cooperative function based on the monitoring result.

12. The information processing apparatus according to claim 11, wherein the controller performs control to present the guidance if an image related to a function that cannot execute the cooperative function together with the first function is further specified.

13. The information processing apparatus according to claim 12, wherein the controller performs control to present the guidance if an operation of linking the first image and an image related to a function that cannot perform the cooperative function together with the first function to each other is performed.

14. The information processing apparatus according to claim 12, wherein the controller performs control to present the guidance if the first image and an image related to a function that cannot perform the cooperative function together with the first function are superimposed on each other.

15. The information processing apparatus according to any one of claims 11 to 14, wherein as the control to present the guidance, the controller performs control to display a candidate list showing information on one or more second functions that can execute the cooperative function.

16. The information processing apparatus according to claim 15, wherein an order in which the one or more second functions are arranged in the candidate list is determined based on past usage records of the one or more second functions.

17. The apparatus according to claim 11, wherein the controller performs control to display the cooperative function while changing the cooperative function in accordance with an order in which the first function and the second function are specified.

18. The information processing apparatus according to claim 11, wherein if the first function and the second function are specified, the controller further performs control to present guidance indicating a third function that can perform a cooperative function together with the first function and the second function.

19. The information processing apparatus according to claim 18, wherein the controller performs control to present the guidance while changing the third function in accordance with an order in which the first function and the second function are specified.

20. The apparatus according to claim 11, wherein the first function and the second function are included in a group of functions registered in advance, a group of functions of one or more identified devices, a group of functions displayed on a display, or a group of functions displayed in a specific area of a screen of the display.

21. An information processing method, comprising the steps of:

if a first image related to a first device required to perform a cooperative function is specified, performing control to present guidance indicating a second device capable of performing the cooperative function together with the first device,

wherein at least one of an operation state of the first device, an environment in which the first device is installed, and an update state of the first device is monitored, and availability of the cooperation function is determined based on the monitoring result.

22. An information processing method, comprising the steps of:

if a first image related to a first function required to execute a cooperative function is specified, performing control to present guidance indicating a second function capable of executing the cooperative function together with the first function,

wherein at least one of an operation state of a device to which the first function and the second function correspond, an environment in which the device to which the first function and the second function correspond is installed, and an updated state of the device to which the first function and the second function correspond is monitored, and the availability of the cooperation function is determined based on a monitoring result.

Images