JP2014191600A - Order support device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an order support device for electronically receiving an order for purchasing a product listed in a catalog by using a general paper medium catalog familiar to middle-aged and elderly users as it is.
When a user presents and browses a catalog of a paper medium in front of the camera of the order support apparatus, the order support apparatus takes a picture of the paper of the catalog, recognizes the image, and opens the catalog type. Recognize the page and display the page on the monitor. Here, when the user designates a product in the catalog by voice, the order support apparatus recognizes the voice and selects the product in the catalog sheet to be a purchase target. The user orders the selected item to be purchased using the order support device.
[Selection] Figure 1

Description

  The present invention relates to an order support apparatus that electronically accepts a purchase order for a product from a user using a product catalog of paper media.

  Conventionally, there is a purchasing method using a so-called catalog viewer in which an electronic catalog having a paper surface equivalent to that of a paper medium catalog is displayed on a display of a user device, and an order for product purchase is received electronically. However, this requires complicated operation methods for the device, such as launching user devices and viewers, and specifying catalogs. Compared with conventional methods using paper catalogs, order sheets, and telephones, digital systems such as those for middle-aged and elderly people are required. There is a problem that the threshold is high for a user unfamiliar with the operation of the device.

  An electronic commerce method for placing an order by using a digital catalog to display information on a product that the user is interested in on the display of the user device is already widely used. As a method of using a conventional paper medium catalog, for example, in Patent Document 1, an article of an information magazine including an article number, such as a general paper medium catalog or advertisement, is photographed with a camera of a user device. Thus, there is shown a method in which when photographed image data is transmitted to an information server device, the information server device recognizes an article number and displays article data related information based on the article number on a portable terminal device.

  Here, the process of recognizing the article number from the captured image data is described in detail. First, several rectangular area images are cut out from the captured image data, their attributes are analyzed, and a certain rectangular area image is analyzed. It is determined as a number image. Further, a technique is disclosed in which the extracted article number image is decomposed for each character to recognize the article number.

  As described above, a so-called image recognition technique for extracting specific information from captured image data is already known. For example, an image obtained by photographing a three-dimensional object is a so-called fluoroscopic image. The closer the image is, the larger the image is, and the smaller the image is, the smaller the image is. ) Are presented (details will be described later). Japanese Patent Application Laid-Open No. H10-228561 presents a technique for correcting a distorted circular image taken with a fisheye lens into a planar regular image. Non-patent document 1 describes an algorithm for calculating a three-dimensional perspective transformation matrix that performs coordinate transformation between coordinate information of an object in a captured image and coordinate information in real space. As will be described later, the algorithms of these existing techniques are also used in the present invention.

  Further, in relation to the image recognition technology for captured images, Non-Patent Document 2 describes a SURF (SpeededUpRobustFeatures) algorithm that extracts feature points of an image at high speed. Also in the present invention, these algorithms can be used, and a program in which this algorithm operates can be used in OpenCV (Non-patent Document 3).

  Furthermore, a voice recognition technique will be described. Speech recognition is a mechanism that replaces "voice", which is a human voice, with "characters", but it is a language that has knowledge of how to arrange acoustic data and words arranged in a single syllable of waveform data of human voice. A method of holding a model, dividing acquired waveform data of human speech into single syllables using a language model, and matching with the stored acoustic data is mainly used. However, with this method, for example, a natural language recognition requires a large processing load and requires a dedicated speech recognition engine.

  On the other hand, by limiting the number of syllables that should be recognized or reducing the number of vocabularies and reducing the number of comparisons with the database, the accuracy and speed of speech recognition can be improved even with a simple processing device. Techniques to improve are presented. Patent Document 3 discloses a technique for reducing the number of vocabularies to be collated and compared by narrowing down the word to be recognized next when a certain word is voice-recognized by hierarchizing the objects to be speech-recognized. ing.

JP 2010-231431 A JP 2009-176273 A JP-A-8-202386

OpenCV, sampling, interpolation, geometric transformation (perspective projection transformation of images, perspective projection transformation of omnidirectional images), [ONLINE] [retrieval on February 22, 2013], Internet <URL: http: // opencv. jp / sample / sampling_and_geometricaltransforms. html # omni_perspective> Herbert Bay, T .; Tuyterrars, and L. VanGool, "SURF: Speeded Up Robust. Features", InECCV, pp. 404-417, 2006. OpenCV, SIFT / SURF, [ONLINE] [searched on February 22, 2013], Internet <URL: http: // opencv. jp / opencv-2svn / cpp / features2d_feature_detection_and_description. html>

  However, in the method disclosed in Patent Document 1, image recognition of an article number image is performed on an article of an information magazine including an article number, and a predetermined feature such as an article number image is included in image data. Therefore, it cannot be said that a general paper medium catalog or advertisement page is recognized.

  The present invention provides an order support apparatus for electronically receiving an order for purchasing a product listed in a catalog by using a general paper medium catalog familiar to a user unfamiliar with the operation of a digital device. This is a proposal.

  In order to solve the above problems, the first invention of the present application is presented with one or more image data on a paper surface, a database for storing a product name and a product name reading of each product on each paper surface, and presented. A page recognizing unit for photographing a paper surface with a camera, identifying the paper image by recognizing the captured image data using the image data of the database, and displaying information relating to the identified paper surface; For the waveform data, the product listed on the specified paper is selected using the database, and the speech waveform data obtained by reading these product names is preferentially collated to recognize the speech. A voice recognition unit for identifying a product by means of, an order receiving unit for receiving an order for purchase of the identified product, a transmission / reception unit for transmitting / receiving information to / from a predetermined external device, And control means for controlling the stage, is an order support apparatus, characterized in that it comprises a. As a result, the user can place an order for product purchase electronically with a simple voice operation using the conventional product catalog.

  A second invention is the order support apparatus according to the first invention, wherein the voice recognition unit identifies a product by voice recognition of a user's voice using an external library. As a result, it is possible to place an order in the accuracy of voice recognition and natural language, and the user can place an order for product purchase with a simpler operation.

  A third invention is the order support apparatus according to the first or second invention, wherein the page recognition means photographs the presented paper surface with a camera provided with a fisheye lens. As a result, the restriction on the position where the page of the catalog or the like is presented is relaxed.

  According to a fourth aspect of the invention, the page recognition means detects the page turning by comparing image feature amount information between the image data obtained by photographing the currently presented paper surface and the image data photographed immediately before, and uses the specified paper surface. The first to third aspects of the invention are characterized in that the paper surface is specified again by preferentially comparing the image data that has been photographed and the image data of the newly specified paper surface of the database with priority. The order support apparatus in any of the above. Accordingly, it is possible to predict in advance that the currently presented paper surface is a newly specified paper surface, and it is possible to quickly re-specify the paper surface by image processing.

According to the present invention, the order support device captures the page of the catalog being browsed and recognizes the image data of the captured image data, thereby recognizing the number and page of the catalog and the product, and electronically purchasing the product. This is to support the provision of services such as orders, and can be used with less burden on the user without troublesome operations on the order support apparatus.

It is a figure which shows the whole structure of this embodiment. It is a functional block diagram explaining in more detail about the order support apparatus 100 of this embodiment. It is a figure which shows the example of a record structure of the catalog goods database. 2 is a diagram illustrating a hardware configuration of an order support apparatus 100. FIG. It is a figure which shows the example of the flow of a user's order using this apparatus. It is an example of the screen displayed while using this apparatus. It is a figure which shows the typical usage example of this apparatus. It is a figure which shows another example of use of this apparatus. It is a figure which shows the example which guides | lays out to put a catalog in the optimal position. It is a block diagram of the process of image recognition. It is a figure explaining the cutout range of an image. It is a figure which shows the coordinate of the four corners of the figure before conversion of perspective correction, and the coordinate of the four corners of the figure after conversion. It is a flowchart which shows the process which concerns on the order assistance using this apparatus.

  Hereinafter, the configuration of a system according to an embodiment of the present invention will be described in more detail with reference to the drawings.

  FIG. 1 is a diagram showing an overall configuration of the present embodiment. Reference numeral 100 denotes an order support apparatus that transmits / receives information to / from a retailer's order receiving apparatus 300 and an external library 400 via a network 500. The apparatus includes a computer program (detailed description will be described later) and several databases (detailed description will be described later).

  The order support apparatus 100 includes a display unit 105 such as a display and an input unit 104 such as a keyboard and a mouse, and includes a camera 111 and a microphone 112 at the top of the display, and a speaker 113.

  Although the network 500 is shown in a simplified manner, it is a combination of a plurality of networks via a gateway such as the Internet, and is composed of a LAN (Local Area Network), a telephone line, a mobile communication network, etc. The access point is configured to enable transmission / reception of information between the order support apparatus 100 and the order receiving apparatus 300 of the retailer or the external library 400 using wired or wireless communication.

  FIG. 2 is a functional block diagram for explaining the order support apparatus 100 of this embodiment in more detail. The order support apparatus 100 includes a catalog product database 170 that stores image data, product names, and product name readings of all pages of a predetermined catalog in units of pages, and a voice guidance 180 that stores recorded voices for voice operation guidance. With a database.

  Further, a page recognition means 150 for recognizing which page of the catalog currently presented by matching the image data captured from the camera 111 and the image data stored in the catalog product database 170, and a user Order receiving means 160 for accepting orders for product purchases from, an image / sound processing unit 190 for performing analysis processing on images and sound used by these means, and control means 130 for controlling these means in association with the database. Each program of the transmission / reception means 120 for transmitting / receiving data to / from the retailer's order receiving apparatus 300 and the external library 400 through the network 500 is provided. The image / sound processing unit 190 includes a perspective correction unit (191) and a fish-eye correction unit (192) that perform image correction, an image recognition unit (193) that performs image recognition, and a voice recognition unit (194) that performs speech recognition. Each functional module is provided.

  FIG. 3 is a diagram showing an example of the record configuration of the catalog product database 170. As shown in FIG. The catalog product database 170 includes an identifier for identifying the catalog number and an identifier representing the page in the booklet, and holds image data of a page for image recognition and collation for each page. Furthermore, it has the reading of the brand name and brand name of all the goods published in the page.

  FIG. 4 is a diagram illustrating a hardware configuration of the order support apparatus 100. The order support apparatus 100 is a computer system as hardware. The order support apparatus 100 includes a control unit 101, a storage unit 102, a peripheral device I / F unit 103, an input unit 104, a display unit 105, and a communication unit 106, which are connected via a bus 109. Note that the hardware configuration shown in FIG. 4 is merely an example, and various configurations can be adopted depending on purposes.

  The control unit 101 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The CPU calls a program stored in the storage unit 102, ROM, recording medium or the like to a work memory area on the RAM, executes it, drives and controls each device connected via the bus 109, and performs processing performed by the computer. Realize. The ROM is a non-volatile memory, and permanently stores a boot program of the order support apparatus 100, a program such as BIOS, data, and the like. The RAM is a volatile memory, and temporarily stores a program, data, and the like loaded from the storage unit 102, ROM, recording medium, and the like, and includes a work area used by the control unit 101 for performing various processes.

  The storage unit 102 is an HDD (hard disk drive), and stores a program executed by the control unit 101, data necessary for program execution, an OS (operating system), and the like. As for the program, a control program corresponding to an OS (operating system), a program code that functions as the page recognition unit 150, a program code that functions as the order receiving unit 160, and a program code that functions as the image / sound processing unit 190 are stored. Has been. These program codes are read by the control unit 101 as necessary, transferred to the RAM, and read and executed by the CPU to function as various means.

  The peripheral device I / F unit 103 is a port for connecting a peripheral device to the order support apparatus 100, and the computer via the peripheral device I / F unit 103 allows peripheral devices such as a camera 111, a microphone 112, and a speaker 113. Send / receive data to / from.

  The input unit 104 inputs data. For example, the input unit 104 includes an input device such as a keyboard, a mouse, and a touch panel. An operation instruction, an operation instruction, data input, and the like are input to the order support apparatus 100 via the input unit 104. It can be performed. The display unit 105 is a display device such as a liquid crystal panel.

  The communication unit 106 includes a communication control device, a communication port, and the like, and is a wired or wireless communication interface that mediates communication between the order support apparatus 100 and the network 500. Communication means for performing communication control with the order receiving apparatus 300. The bus 109 is a path that mediates transmission / reception of control signals, data signals, and the like between the devices.

  FIG. 5 is a diagram showing an example of an order flow of a user who uses this apparatus. First, when the user starts up the apparatus, the front cover of the catalog on which the product to be ordered is posted is presented in front of the camera 111 of the order support apparatus 100. The control means 130 of the order support apparatus 100 activates the page recognition means 150 and takes a picture of the catalog cover (S01). The page recognition means 150 identifies the number of the catalog from the image data on the cover of the catalog (details will be described later), and the identified number is voiced together with the operation start guidance and output from the speaker 113 (S02).

  Subsequently, the user turns the catalog page in order to search for a product to be ordered, and browses a page on which the product to be ordered is posted (S03). The page recognition unit 150 of the order support apparatus 100 captures the currently opened page with the camera 111, identifies the number of the page from the catalog image data (details will be described later), and the content related to the page. (For example, an image of the page or an advertisement of the posted product) and a microphone icon are displayed on the screen (S04, see FIG. 6A).

  When the product to be ordered is determined, the user clicks the microphone icon on the screen with the mouse or touches it on the touch panel. Here, the control means 130 of the order support apparatus 100 activates the order receiving means 160, enters the voice input mode, displays a screen (see FIG. 6B) that prompts the user to read the product name, and plays guidance. The user reads out the name of the product he wants to order according to this (S05). Thereafter, when a silent time of about 1 second is detected, the voice input mode is canceled. During this period, the order support apparatus 100 records the voice with the microphone 112, performs voice recognition processing (details will be described later), and displays the recognition result on the screen (see S06, FIG. 6C).

  The user decides whether to select this product (select the circle icon) or not select (select the x icon) for this recognition result. When the user selects this product, a shopping cart screen is displayed, indicating that the product has been selected for purchase. Further, by performing voice recognition processing with high accuracy, a product may be selected without confirming the recognition result by the user (see FIG. 6D). Here, when there is at least one item selected for purchase, an order icon is displayed, so the user can select this icon at any time to order the item in the shopping cart. (S07). Information on the purchased product is transmitted to the terminal 300 of the store through the network 500 via the transmission / reception means 120.

  FIG. 7 is a diagram illustrating a typical usage example of the present apparatus. As described with reference to FIG. 1, the camera 111 and the microphone 112 are installed and fixed on the liquid crystal display monitor that is the display unit 105 of the order support apparatus 100. When using this apparatus, the user presents the page of the catalog between the user and the liquid crystal display monitor as seen from the user (see FIG. 7A). FIG.7 (b) is the schematic diagram which looked at this from the side. As shown in this figure, the fixed camera 111 is preferably tilted forward so that the entire surface of the catalog can be projected well.

  FIG. 8 is a diagram showing another example of use of the present apparatus. The order support apparatus 100 may be a tablet computer, for example, while having the configuration described in FIG. In this case, the input unit 104 may be incorporated in the display unit 105 as a touch panel, and the camera 111, the microphone 112, and the speaker 113 may be those incorporated in the computer.

  When a tablet-type computer is used as the order support apparatus 100, the orientation of the camera 111 may be fixed in one forward direction as shown in FIG. 8A and cannot be tilted forward. There is a risk that the camera may not be within the shooting range. In such a case, as shown in FIG. 8B, a fisheye lens that can be retrofitted can be used to widen the angle of view (theoretically 180 degrees) so that the entire sheet of paper is within the shooting range of the camera.

  Regardless of whether the order support apparatus 100 is a tablet type or a display monitor is independent, as described above, the user presents the front of the camera 111 so that the page of the catalog faces the user as viewed from the user. At this time, the order support apparatus 100 projects the image currently captured by the camera 111 on the display, and guides the position of the catalog to be presented to an optimum position for recognition to fix the presentation position. Desirable (see FIG. 9).

  FIG. 10 is a block diagram of the image recognition process of the page recognition unit 150 for identifying the catalog number and page in the order support apparatus 100. First, the page recognition unit 150 performs binarization processing on the captured image data captured by the camera 111 (S11). Predetermined parameters are used for binarization parameters (color specification (color coordinate) system and slice level). This is for ensuring consistency with the sample image (image data of the catalog product database) to be finally compared.

  Further, image clipping processing is performed on the binarized captured image data (S12). As described above, as long as the position of the catalog to be photographed is placed at the optimum position for recognition, the cutout position of the image may be determined without going through a procedure such as contour extraction. FIG. 11 is a diagram for explaining an image cut-out position. FIG. 11A shows a cutout range (a range surrounded by a dotted line in the figure) when the image is taken with the normal camera 111. FIG. 11B shows a cut-out range (a range surrounded by a dotted line in the figure) when the image is taken by the camera 111 having a fisheye lens.

Next, image correction processing is performed on the captured image data that has been cut out (S13 in FIG. 10). Many methods for perspective correction have been proposed. Here, as an example, a method called projective transformation is introduced. As shown in FIG. 12, the coordinates of the four corners of the graphic before conversion are (X1, Y1), (X2, Y2), (X3, Y3), (X4, Y4), and the coordinates of the four corners of the graphic after conversion are ( Assuming x1, y1), (x2, y2), (x3, y3), (x4, y4), arbitrary coordinates (X, Y) before conversion are converted to (x, y) by the following equation: .
(X, y) = ((AX + BY + C) / (GX + HY + 1), (DX + EY + F) / (GX + HY + 1))
Here, the values of A, B, C, D, E, F, G, and H are obtained by solving the following eight simultaneous equations.
X1 * A + Y1 * B + C-x1 * X1 * G-x1 * Y1 * H = x1
X1 * D + Y1 * E + F-y1 * X1 * G-y1 * Y1 * H = y1
X2 * A + Y2 * B + C-x2 * X2 * G-x2 * Y2 * H = x2
X2 * D + Y2 * E + F-y2 * X2 * G-y2 * Y2 * H = y2
X3 * A + Y3 * B + C-x3 * X3 * G-x3 * Y3 * H = x3
X3 * D + Y3 * E + F-y3 * X3 * G-y3 * Y3 * H = y3
X4 * A + Y4 * B + C-x4 * X4 * G-x4 * Y4 * H = x4
X4 * D + Y4 * E + F-y4 * X4 * G-y4 * Y4 * H = y4

  In addition, fish-eye correction (192) is performed on the captured image data captured by the camera 111 having a fish-eye lens. As described above, Patent Document 2 presents a technique for correcting a distorted circular image as taken with a fisheye lens into a planar regular image. The method of Patent Literature 2 converts an image in the cutout area E centered on the cutout center point P (x0, y0) on the distorted circular image into a planar regular image. Here, the cut-out center point P (x0, y0) and the cut-out area E are fixed so as to use the dotted line range and the center point shown in FIG.

  By doing so, the intersection G (x0, y0, z0) of the phantom spherical surface H and the straight line passing through the cut-out center point P (x0, y0) and parallel to the Z-axis described in Patent Document 2, A tangent plane that touches the virtual spherical surface H at a point is uniquely determined. Further, a UV coordinate system is defined on this tangent plane so that G (x0, y0, z0) is the origin (this tangent plane is referred to as a UV plane). Here, the point G (x0, y0, z0) can also be expressed by an azimuth angle and a zenith angle, and can also be expressed as an azimuth angle of 0 degrees and a zenith angle (90 ° −θ) degrees (see FIG. 8B). In the present application, the value of the coordinates of G (x0, y0, z0) is uniquely determined.

  Further, an angle φ formed by the U axis and the X ′ axis is defined. Here, the X ′ axis is an axis that passes through the intersection point G (x0, y0, z0) and is parallel to the X axis. Since this is a parameter indicating the rotation factor of the UV coordinate system, it is referred to as a “planar inclination angle”, but in the present application, the value of this angle φ is also set to one. Further, the radius R in the following expression is a numerical value unique to the fisheye lens used for photographing, and thus this value is also set to one.

Here, the pixel value of the pixel on the planar regular image arranged at the position indicated by the coordinates (u, v) by associating the coordinates (u, v) with the coordinates (x, y) using the following expression: Is determined based on the pixel value of the reference pixel on the distorted circular image arranged at the position indicated by the corresponding coordinates (x, y).
x = R [(u−x0) A + (v−y0) B + (w−z0) E] / √ ((u−x0) ** 2+ (v−y0) **** + (w−z0) ** 2 )
y = R [(u−x0) C + (v−y0) D + (w−z0) F] / √ ((u−x0) ** 2+ (v−y0) ** 2+ (w−z0) ** 2 )
here,
A = 1- (1-cosφ) (y02 + z02)
B = −z0sinφ + x0y0 (1-cosφ)
C = z0sinφ + x0y0 (1-cosφ)
D = 1- (1-cosφ) (z02 + x02)
E = y0sinφ + z0x0 (1-cosφ)
F = −x0sinφ + y0x0 (1-cosφ)
w = mR (where m is a predetermined conversion magnification)
In the above formula, “** 2” after √ (root sign) represents the square).

  Subsequently, processing by the image recognition unit (193) is performed. A feature extraction process is performed on the captured image data subjected to the image correction process (S14). In the feature extraction processing, feature amount information extracted from the captured image data subjected to the image correction processing (for example, feature points are extracted by the SURF algorithm described in Non-Patent Document 2 described above, (Feature).

  Finally, by performing a round-robin matching process between the feature amount information extracted from the captured image data and the feature amount information of the image data in the catalog product database 170, the closest image is selected, and the page number ( If it is a cover page, the catalog number) is determined as the expected current page (the page that is expected to be currently viewed), and the content related to the page (for example, an image of the page or an advertisement of the posted product) is displayed on the screen. Display and complete image identification.

  The voice recognition unit (194) is a mechanism for replacing “voice”, which is a human voice, with “character”. Various methods have been proposed for the speech recognition technology. However, as I mentioned earlier, basically, we have a language model with acoustic data collected for each single syllable, and knowledge of how to arrange words, and acquired from the outside. Since speech waveform data is mainly decomposed into single syllables using a language model and replaced with characters by matching with the stored acoustic data, processing is particularly necessary for spoken language speech recognition in natural language. The load is high and a dedicated speech recognition engine is required.

  On the other hand, for example, as used in toys and lighting equipment in recent years, restricting syllables by limiting words to speech recognition targets, reducing the number of vocabularies to be compared, and reducing the number of comparison processes with a database. There is a contrivance to improve the accuracy and speed of speech recognition even if it is not a dedicated processing device.

  The method for restricting the word (vocabulary) to be recognized differs depending on the object to be recognized. For example, Patent Document 3 discloses, as voice recognition by a navigation device, place names in order to reduce the number of place names to be recognized. A method is shown in which the number of place names to be recognized is limited by hierarchizing and dividing into hierarchies such as prefectures and municipalities.

  In the present application, the object to be voice-recognized is the product name of the product purchased by the user, but since the page currently being viewed is known by the image recognition of the catalog page described above, the catalog product database 170, it is possible to know what product name is listed on the page, and it is possible to preferentially set the product name that is subject to speech recognition or to limit the number of target products. it can. Thereby, it is possible to improve the accuracy and speed of speech recognition even using the order support apparatus 100 of the present application.

  Specifically, the control unit 130 of the order support apparatus 100 activates the order receiving unit 160 to enter the voice input mode, and when the voice from the user is recorded with the microphone 112 while in the voice input mode, the input is performed. The converted voice signal is converted into waveform data to obtain user's voice waveform data. With reference to the catalog product database 170, the product names of all products listed on the catalog page currently being viewed are changed. By preferentially matching speech waveform data created by concatenating single syllable acoustic data using reading, speech recognition is performed at high speed without errors, and the recognition results are displayed on the screen (Fig. 6 (c)).

  In addition, the external library 400 may be used for the processing of the voice recognition unit (194). When the order support apparatus 100 records the voice with the microphone 112 while in the voice input mode, the order support apparatus 100 transmits a list of readings of the input voice signals and the product names that are candidates for voice recognition via the transmission / reception means 120 to the network 500. To the external library 400. The external library 400 performs voice recognition and returns the recognition result to the order support apparatus 100 via the network 500. As an external library for performing such voice recognition, Google's Google speech search, Apple's SIRI (registered trademark), and the like are known.

  FIG. 13 is a flowchart showing processing related to page recognition of this apparatus. As described above, when identification of captured captured image data is completed and the page number of the expected current page is determined, the paper is photographed by the camera 111, and there is a large change by comparing with the image captured immediately before. And the presence or absence of a page turning operation is detected (S21).

  Specifically, when the feature information on the left side as viewed from the camera (on the right side as viewed from the user) is greatly changed by comparing the feature amount information with the image taken immediately before, for example, the page number is increased in the page direction. If it is judged that the page has been turned and the feature information on the right side (left side as viewed from the user) changes significantly, it is judged that the page has been turned in the direction of decreasing the page number, and the expected current page is increased or decreased. (S22), and a paper surface check described later is performed. Note that with regard to this page turning, it is expected that a plurality of transitional images will be taken continuously, so care should be taken to increase or decrease the page count only once during a series of page turning.

  If there is no page turning, it is considered that a product is being selected. Therefore, the speaker 113 may be used for waiting for the product to be ordered or by voice guidance.

  When there is an order by voice, the control means 130 of the order support apparatus 100 first checks whether there is an ordered product among the products on the expected current page (S23). For this, the catalog product database 170 is referred to, the product names of all products listed on the current expected page are acquired, and the speech recognition target is narrowed down to these products in the speech recognition by the order receiving means 160. The accuracy and speed of recognition can be improved. Here, when there is no ordered product on the predicted current page, the page on which the ordered product is posted with reference to the catalog product database 170 is set as the predicted current page, and the displayed content is changed accordingly (S24). ), And confirms the page as described later.

  When a predetermined time (for example, 10 seconds) has passed since the previous page recognition time, the currently opened page is photographed using the camera 111, and processing by the image recognition unit (193) is performed (S25). At this time, it is assumed that the expected current page is currently opened (S26), and a paper space check described later is performed.

  In each of the operations described above, when a paper space check is to be performed, first, a paper surface check is performed on the predicted current page by matching processing with the image data of the page of the catalog product database 170 (S27). In this way, the recognition speed can be improved by limiting the image recognition target. Here, if a match is obtained, the predicted current page is left as it is. If the recognition results are different, a round-robin matching process is performed, and the matched page is set as a new predicted current page (S28). If the predetermined time has not elapsed, the process ends, and the process is repeated from the confirmation of the page turning operation again.

  According to the present embodiment, when the user browses the catalog in front of the order support apparatus 100, the camera 111 recognizes the pages being browsed sequentially and displays the pages of the catalog and related information on the monitor display. An order support device is provided that allows the user to select a product and place an order for product purchase, which is advantageous for both the user and the store.

  The present invention is not limited to the above-described embodiments, and various modifications and applications are possible within the scope of the claims. For example, the order support apparatus 100 is depicted as being connected to the network 500 by wire, but may be connected wirelessly. Moreover, although tablet PC is illustrated in description, even if it is a smart phone, it is possible to implement this invention similarly.

DESCRIPTION OF SYMBOLS 100 Order support apparatus 120 Transmission / reception means 130 Control means 150 Page recognition means 160 Order reception means 170 Catalog goods database 180 Voice guidance 190 Image / sound processing part 300 Retailer's order reception apparatus 400 External library 500 Network

Claims (4)

A database that stores image data of one or more pages, the names of products listed on each page, and readings of product names;
Page recognition means for photographing the presented paper surface with a camera, identifying the photographed image data using the image data of the database to identify the paper surface, and displaying information on the identified paper surface;
With respect to the waveform data of the user's voice, the products listed on the specified paper are selected using the database, and the waveform data of the voice obtained from reading these product names is preferentially verified. A voice recognition unit that identifies the product by voice recognition,
Order accepting means for accepting an order for purchasing the specified product;
Transmitting / receiving means for transmitting / receiving information to / from a predetermined external device;
An order support apparatus comprising: control means for controlling each means. The voice recognition unit
The order support apparatus according to claim 1, wherein a product is specified by recognizing a user's voice using an external library. The page recognition means
The order support apparatus according to claim 1, wherein the presented paper surface is photographed by a camera provided with a fisheye lens. The page recognition means
The page data is detected by comparing the image feature amount information between the image data obtained by photographing the currently presented paper surface and the image data obtained immediately before, and the specified paper surface is changed. The order support apparatus according to any one of claims 1 to 3, wherein the paper surface is specified again by preferentially comparing the image data of the newly specified paper surface.

Images