JP2017219942A - Contact detection device, projector device, electronic blackboard device, digital signage device, projector system, contact detection method, program, and storage medium. - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a contact detection device that can deal with a multi-tap.SOLUTION: A contact detection device comprises: a range-finding unit 13 that acquires three-dimensional information on a hand and screen; setting means 50 that sets a contact object surface on the basis of the three-dimensional information on the screen; transformation means 52 that transforms the three-dimensional information on the hand to two-dimensional information by a projective transformation; end part candidate detection means 53 that detects a plurality of end candidates of each of a plurality of fingers on the basis of the two-dimensional information; ranking means 54 that implements comprehensive ranking of a proximity level to the contact object surface with respect to the plurality of end part candidates of each of the plurality of fingers; and determination means 55 that determines whether a portion highest in the ranking of the proximity level to each of the two-dimensional information on the plurality of fingers contacts with the contact object surface.SELECTED DRAWING: Figure 22

Description

  The present invention relates to a contact detection device, a projector device, an electronic blackboard device, a digital signage device, a projector system, a contact detection method, a program, and a storage medium.

  In recent years, technology for performing contact determination of an object with respect to an object has been actively developed (see, for example, Patent Documents 1 to 5).

  However, the techniques disclosed in Patent Documents 1 to 5 have room for improvement with regard to handling multi-tap.

  The present invention relates to a contact detection device for detecting contact between an object having a plurality of branch portions and an object, a detection system for detecting a plurality of end candidates for each of the plurality of branch portions, and the plurality of branch portions. The closest edge candidate acquisition system for obtaining the closest edge candidate that is the edge candidate closest to the object from each of the plurality of edge candidates, and the closest edge of each of the plurality of branch portions Determination means for determining whether or not a candidate for a part is in contact with the object.

  According to the present invention, it is possible to cope with multi-tap.

It is a figure for demonstrating schematic structure of the projector system which concerns on one Embodiment of this invention. It is a figure for demonstrating a projector apparatus. It is a figure for demonstrating a ranging part. It is a figure for demonstrating the external appearance of a ranging part. It is a figure for demonstrating an imaging part. It is a flowchart for demonstrating the pre-processing performed by a process part. It is a flowchart for demonstrating the input operation information acquisition process performed by a process part. It is FIG. (1) for demonstrating input operation information acquisition processing. It is a figure for demonstrating an example of the hand area | region by which the projective transformation was carried out. It is a figure for demonstrating the case where FIG. 9 is downsampled. It is a figure for demonstrating the result of the convex hull process with respect to FIG. It is a figure for demonstrating the result of the convex hull process with respect to FIG. It is FIG. (2) for demonstrating input operation information acquisition processing. FIG. 10 is a third diagram illustrating the input operation information acquisition process. It is FIG. (1) for demonstrating a multi tap. It is a figure for demonstrating the problem of the multi tap of a prior art. It is FIG. (1) for demonstrating the contact detection method of this embodiment. It is FIG. (2) for demonstrating the contact detection method of this embodiment. It is FIG. (3) for demonstrating the contact detection method of this embodiment. It is FIG. (4) for demonstrating the contact detection method of this embodiment. It is FIG. (5) for demonstrating the contact detection method of this embodiment. It is a block diagram which shows the structure of a process part. It is a figure for demonstrating the modification 1 of a projector apparatus. It is a figure for demonstrating the modification 1 of a ranging part. It is a figure for demonstrating the modification 2 of a ranging part. It is FIG. (1) for demonstrating the modification 3 of a ranging part. It is FIG. (2) for demonstrating the modification 3 of a ranging part. It is a figure for demonstrating the modification 2 of a projector apparatus. It is a figure for demonstrating the case where the surface of a target object contains a level | step difference. It is a figure for demonstrating the case where the surface of a target object contains a curved surface. It is a figure for demonstrating an example of an electronic blackboard apparatus. It is a figure for demonstrating an example of a digital signage apparatus.

Embodiment 1
Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 shows a schematic configuration of a projector system 100 according to an embodiment.

  The projector system 100 includes a projector device 10 and an image management device 30. An operator (user) touches the vicinity of the projection surface of the screen 300 or an image projected on the projection surface (hereinafter also referred to as a “projection image”) by input operation means such as a finger, a pen, and a pointing stick. ). The projected image may be either a still image or a moving image.

  The projector device 10 and the image management device 30 are mounted on a desk, a table, a dedicated pedestal or the like (hereinafter referred to as “mounting table 400”). Here, a three-dimensional orthogonal coordinate system is used, and a direction orthogonal to the mounting surface of the mounting table 400 is defined as a Z-axis direction. Further, it is assumed that the screen 300 is installed on the + Y side of the projector device 10. A surface on the −Y side of the screen 300 is a projection surface. In addition, various things, such as a board surface of a white board and a wall surface, can be utilized as a projection surface.

  The image management device 30 holds a plurality of image data, and sends projection target image information (hereinafter also referred to as “projection image information”) to the projector device 10 based on an instruction from the operator. The communication between the image management device 30 and the projector device 10 may be wired communication via a cable such as a USB (Universal Serial Bus) cable, or may be wireless communication. As the image management apparatus 30, a personal computer (personal computer) in which a predetermined program is installed can be used.

  Further, when the image management apparatus 30 has an interface of a detachable recording medium such as a USB memory or an SD card, an image stored in the recording medium can be used as a projected image.

  The projector device 10 is a so-called interactive projector device, and includes a projection unit 11, a distance measuring unit 13, a processing unit 15, and the like as shown in FIG. 2 as an example. These are housed in a housing (not shown).

  In the projector device 10 according to the present embodiment, the distance measuring unit 13 and the processing unit 15 constitute a contact detection device of the present invention. The screen 300 is an object.

  The projection unit 11 includes a light source, a color filter, various optical elements, and the like, similarly to the conventional projector device, and is controlled by the processing unit 15.

  When the processing unit 15 performs bidirectional communication with the image management apparatus 30 and receives projection image information, the processing unit 15 performs predetermined image processing and projects the image onto the screen 300 via the projection unit 11.

  As shown in FIG. 3 as an example, the distance measuring unit 13 includes a light emitting unit 131, an imaging unit 132, a calculation unit 133, and the like. The appearance of the distance measuring unit 13 is shown in FIG. 4 as an example. Here, the light emission part 131, the imaging part 132, and the calculating part 133 are accommodated in the housing.

  The light emitting unit 131 has a light source that emits near-infrared light, and emits light (detection light) toward the projection image. The light source is turned on and off by the processing unit 15. As this light source, an LED, a semiconductor laser (LD), or the like can be used. Further, an optical element or a filter for adjusting light emitted from the light source may be provided. In this case, for example, the emission direction (angle) of the detection light is adjusted, the detection light is structured light, intensity-modulated light, light that gives a texture to the imaging target, or the like. be able to.

  As shown in FIG. 5 as an example, the imaging unit 132 includes an imaging element 132a and an imaging optical system 132b. The image sensor 132a is an area-type image sensor. The shape of the image sensor 132a is a rectangular shape. The imaging optical system 132b guides the light emitted from the light emitting unit 131 and reflected by the imaging object to the imaging element 132a. Here, since the imaging element 132a is an area type, two-dimensional information can be acquired in a lump without using light deflecting means such as a polygon mirror.

  The imaging target of the imaging unit 132 may be a projection plane on which no projection image is projected, a projection image projected on the projection plane, or an input operation unit and a projection image.

  The imaging optical system 132b is a so-called coaxial optical system, and an optical axis is defined. Hereinafter, the optical axis of the imaging optical system 132b is also referred to as “the optical axis of the distance measuring unit 13” for convenience. Here, a direction parallel to the optical axis of the distance measuring unit 13 is an a-axis direction, and a direction orthogonal to both the a-axis direction and the X-axis direction is a b-axis direction. In addition, the angle of view of the imaging optical system 132b is set so that the entire area of the projected image can be captured.

  Returning to FIG. 2, the distance measurement unit 13 has a projection image in which the a-axis direction is a direction inclined counterclockwise with respect to the Y-axis direction, and the position at which the optical axis of the distance measurement unit 13 intersects the projection plane. It is arrange | positioned so that it may become -Z side rather than the center. That is, with respect to the Z-axis direction, the arrangement position of the distance measurement unit 13 and the position where the optical axis of the distance measurement unit 13 intersects the projection image are on the same side with respect to the center of the projection image.

  The computing unit 133 calculates distance information to the imaging target based on the light emission timing at the light emitting unit 131 and the imaging timing of the reflected light at the imaging element 132a. That is, here, a TOF (Time Of Flight) method is used as a calculation method of distance information. Then, three-dimensional information of the captured image, that is, a depth map is acquired. The center of the acquired depth map is on the optical axis of the distance measuring unit 13.

  The calculation unit 133 acquires a depth map of the imaging target at a predetermined time interval (frame rate) and notifies the processing unit 15 of the acquired depth map.

  The processing unit 15 detects contact between the input operation unit and the projection plane based on the depth map obtained by the calculation unit 133, obtains the position and movement of the input operation unit, and obtains input operation information corresponding thereto. Ask. Further, the processing unit 15 notifies the image management apparatus 30 of the input operation information.

  When receiving the input operation information from the processing unit 15, the image management device 30 performs image control according to the input operation information. Thereby, the input operation information is reflected in the projection image.

  Next, the preprocessing performed by the processing unit 15 will be described with reference to the flowchart of FIG. This preprocessing is performed in a state where there is no input operation means in the imaging area of the imaging unit 132, such as when the power is turned on or before the input operation is started.

  In the first step S <b> 201, a depth map in a state where there is no input operation means, that is, three-dimensional information of the projection plane is acquired from the calculation unit 133.

  In the next step S203, a contact target surface is set based on the acquired depth map. Here, with respect to the three-dimensional information of the projection plane, a position 3 mm away from the −a direction is defined as the contact target plane.

  By the way, the depth map from the calculation unit 133 includes a measurement error in the distance measurement unit 13. Therefore, the measurement value of the depth map may enter the inside of the screen 300 (+ Y side of the projection plane). Therefore, the measurement error in the distance measuring unit 13 is added to the three-dimensional information on the projection plane as an offset.

  Note that the value of 3 mm used here is an example, and is preferably set to about a measurement error (for example, standard deviation σ) in the distance measuring unit 13.

  Further, when the measurement error in the distance measuring unit 13 is small or when no offset is required, the three-dimensional information itself of the projection plane can be used as the contact target plane.

  Further, the contact target surface is not expressed by an approximate expression assuming that the entire surface is a single plane, but has data for each pixel. In addition, when the contact target surface includes a curved surface or a step, the contact target surface is divided into minute planes, and an abnormal value is removed for each pixel by performing median processing or averaging processing for each minute plane. Have data.

  In the next step S205, the set three-dimensional data of the contact target surface is stored as data for each pixel. Hereinafter, the set three-dimensional data of the contact target surface is also referred to as “contact target surface data”.

  By the way, the implementation of the preprocessing is not limited to when the power is turned on or before the input operation is started. For example, when the projection surface is deformed over time, it may be appropriately implemented without the input operation means.

  Next, input operation information acquisition processing performed by the processing unit 15 during interactive operation will be described with reference to the flowchart of FIG. Here, the input operation means is assumed to be an operator's finger, but is not limited to this, and may be a pen or a pointing stick, for example.

  In the first step S102, it is determined whether or not a new depth map has been sent from the computing unit 133. If a new depth map has not been sent from the computing unit 133, the determination here is denied, and it waits for a new depth map to be sent from the computing unit 133. On the other hand, if a new depth map has been sent from the calculation unit 133, the determination here is affirmed, and the process proceeds to step S103.

  Here, the depth map is a depth map in a state where the input operation means is in the imaging area of the imaging unit 132.

  In this step S103, based on the depth map from the calculation unit 133 and the stored contact target surface data, the input operation means is within a predetermined distance L1 (see FIG. 8) from the contact target surface with respect to the -a direction. Judge whether there is. If there is an input operation means within a predetermined distance L1 from the contact target surface, the determination here is affirmed and the process proceeds to step S104. On the other hand, if there is no object within the predetermined distance L1 from the contact target surface with respect to the -a direction, the determination here is denied and the process returns to step S102. Here, L1 = 100 mm. Hereinafter, for convenience, the area of the input operation means in the depth map is also referred to as a “hand area”.

  That is, the input operation means located at a position exceeding the distance L1 from the contact target surface with respect to the -a direction is regarded as irrelevant to the contact, and the subsequent processing is not executed. Thereby, excessive processing is removed, and the calculation load can be reduced.

  In step S104, the hand region is extracted from the depth map.

  By the way, in this embodiment, it corresponds also to several input operation means. For example, if there are two input operation means, at least two hand regions are extracted. At least two means that there is something that is erroneously extracted as a hand region. For example, when an arm, an elbow, a part of clothes, or the like falls within a predetermined distance L1 from the contact target surface in the −a direction, they are erroneously extracted as a hand region. Note that it is preferable from the viewpoint of calculation load that the number of erroneous extractions at this stage is small, but there is no inconvenience even if there is an erroneous extraction.

  In addition, although L1 = 100 mm here, this value is an example. However, if the value of L1 is too small, the hand area to be extracted becomes small, and subsequent image processing tends to be difficult. If the value of L1 is too large, the number of erroneous extractions increases. According to experiments by the inventors, the value of L1 is preferably 100 mm to 300 mm.

  In the next step S105, the extracted hand region is projectively transformed. Thereby, three-dimensional information is converted into two-dimensional information. Here, a pinhole camera model is applied to the distance measuring unit 13, and projective transformation is performed on a plane orthogonal to the optical axis of the distance measuring unit 13. By making the three-dimensional information into two-dimensional information, the subsequent image processing is facilitated, and the calculation load is reduced.

  FIG. 9 shows an example of a hand region that has undergone projective transformation. The white part is the hand area. In this example, half of an adult's hand and a finger silhouette appear as input operation means. FIG. 10 shows a case where the depth map is down-sampled to 1/10 in the example shown in FIG.

  For the converted two-dimensional information, the area of the image was calculated, and those other than those within a predetermined range were excluded as non-hand regions. This is because a clearly small area is considered to be noise, and a clearly large area is not considered to be a part including a fingertip such as a body or clothes. This processing reduces the subsequent calculation load.

  In the next step S106, convex hull processing is performed on the two-dimensional image of the hand region. Here, the convex hull process is a process for obtaining a minimum convex polygon including all points of the hand region which is a white portion. When there are a plurality of hand regions, a convex hull process is performed on each hand region. FIG. 11 shows the result of the convex hull processing for FIG. FIG. 12 shows the result of the convex hull processing for FIG.

  In the next step S107, a fingertip candidate is detected for each hand region. Here, a plurality of vertices obtained by the convex hull process are set as fingertip candidates in the hand region.

  The processing here is performed for a plurality of finger regions in the hand region (see FIG. 17). In the i finger areas (i is 2 to 5), the vertex closest to the screen 300 (that is, the fingertip candidate) among all the vertices by the convex hull process is denoted as Kj. 17 to 21 indicate each detected vertex Kj, and the number in the circle indicates the number j of each vertex Kj.

  By the way, a pattern matching method using a template is conceivable as a method for extracting the tip portion. However, in this method, when the two-dimensional information is different from the template, the detection rate is greatly reduced. In order to perform pattern matching, an image having a corresponding resolution (number of pixels) is required as a template. On the other hand, in the case of convex hull processing, ultimately, if there is one pixel at the tip, it can be detected as a vertex (fingertip candidate).

  For example, looking at the silhouette in FIG. 10, it can be seen that the actual tip portion is only one pixel, but as shown in FIG. 11, it is properly detected as a vertex (that is, a fingertip candidate).

  In the next step S108, the stored contact target surface data is referred to, and for all fingertip candidates in the hand region, a predetermined distance L2 from the contact target surface with respect to the -a direction (see FIGS. 13 and 14). Search for fingertip candidates within. Here, as an example, L2 = 30 mm.

  In the next step S109, it is determined whether or not there is a corresponding fingertip candidate by referring to the search result. If there is a corresponding fingertip candidate, the determination here is affirmed and the process proceeds to step S110. On the other hand, if there is no corresponding fingertip candidate, the determination here is denied and the process returns to step S102.

  In the next step S110, all the corresponding fingertip candidates are ranked comprehensively with respect to the contact target surface.

  FIG. 17 shows an example in which five vertices are detected in each of four finger regions (here, the index finger region, the middle finger region, the ring finger region, and the little finger region) of the hand region. Here, the overall ranking of the proximity to the contact target surface is performed on five vertices (total of 20 vertices) of the four finger regions, and the numbers 1 to 20 indicating the ranking results are assigned to these 20 It is attached to each vertex. That is, these 20 vertices are closer to the contact target surface as the number j is smaller.

  In the next step S111, fingertip candidates other than the fingertip candidate closest to the contact target surface (also referred to as “nearest fingertip candidate”) are excluded for each finger area. That is, the fingertip candidates other than the fingertip candidate having the highest degree of proximity to the contact target surface (for example, the degree of the short distance in the a direction to the contact target surface and the degree of the perpendicular to the contact target surface) are excluded. Perform exclusion processing.

  In the next step S112, it is determined whether or not the exclusion process has been completed for all finger regions. If the exclusion process has not ended, the process returns to step S111, and if it has ended, the process proceeds to step S113.

  In step S113, the remaining fingertip candidates are ranked in the closest degree to the contact target surface. That is, the closest degree of the closest fingertip candidates of a plurality of (for example, four) finger regions is ranked according to the closest degree to the contact target surface. When step S113 is executed, the process proceeds to step S114.

  In step S114, contact / non-contact between the closest fingertip candidate and the screen 300 for each finger area is determined. Details will be described later. When step S114 is executed, the process proceeds to step S115.

  In step S115, it is determined whether there is a closest fingertip candidate determined as “contact” in step S114. If the determination here is affirmed, the process proceeds to step S116, and if the determination is negative, the process returns to step S102.

  In step S116, input operation information is obtained based on the contact status and contact position of the closest fingertip candidate. For example, if the contact is a short time of about one frame or several frames, it is determined that the input operation is a click. If the contact continues and the position moves between frames, it is determined that the input operation is to write a character or a line. In such a determination, if the closest fingertip candidate determined as “contact” in step S114 is single, it is determined as a single-tap input operation, and if it is determined as multiple, it is determined as a multi-tap input operation. To do. Specific examples of the multi-tap input operation include a screen enlargement operation and a reduction operation.

  In the next step S117, the obtained input operation information is notified to the image management apparatus 30. Thereby, the image management apparatus 30 performs image control according to the input operation information. That is, the input operation information is reflected in the projected image. When step S117 is executed, the process returns to step S102.

  Next, steps S111 to S113 will be described in more detail. Here, there are a total of 20 vertices in the four finger areas of the hand area as shown in FIG. That is, by performing the convex hull processing on each finger region as described above, twenty vertices (K1, K2,..., K19, K20) are generated as a whole in the four finger regions.

  Here, in each finger region, a range that is centered on a vertex (closest vertex) closest to the contact target surface among a plurality of (here, 5) vertices and that excludes vertices other than the closest vertex It is defined as an exclusion range (see FIGS. 18 to 21).

  In FIG. 18, the exclusion range is a circle centered on the vertex K1 closest to the contact target surface among the vertices K1 to K20, and the state before excluding other vertices K2, K3, K4, and K10 in the circle It is shown.

  In FIG. 19, the exclusion range is a circle centered on the vertex K1 closest to the contact target surface among the vertices K1 to K20, and shows a state after excluding other vertices K2, K3, K4, and K10. .

  In FIG. 20, the exclusion range is a circle centered on the vertex K5 closest to the contact target surface among the vertices K5 to K9 and K11 to K20, and excludes other vertices K7, K8, K16, and K19 in the circle. The state before doing is shown.

  In FIG. 21, the four exclusion ranges are circles centered on the vertices K11, K1, K5, and K6 closest to the contact target surface in each of the four finger regions, and the state after excluding the other vertices is shown. ing.

  First, in FIG. 17, the inside of a broken circle centering on the vertex K1 closest to the contact target surface among the 20 vertices is set as an exclusion range (see FIG. 18). Four vertices K2, K3, K4, and K10 included in this exclusion range are excluded (see FIG. 19). When these four vertices K2, K3, K4, and K10 are excluded, the vertex closest to the contact target surface after K1 is K5. Therefore, as the next step, the inside of a broken circle centered on K5 is set as an exclusion range (see FIG. 20), and the vertices K7, K8, K16, and K19 included in this exclusion range are excluded. By repeating such steps, the four vertices K1, K5, K6, and K11 shown in FIG. 21 are finally obtained. These four vertices K1, K5, K6, and K11 are ranked in the closest degree to the contact target surface, and the ranking result is output.

  Then, based on the ranking result, it can be determined whether or not each of the four finger regions shown in FIG. 21 is in contact with the contact target surface. A specific determination method will be described later. Here, the case where a plurality of apexes as fingertip candidates are detected in each of the index finger region, the middle finger region, the ring finger region, and the little finger region has been described as an example, but in short, five finger regions (thumb region, index finger region) The middle finger region, the ring finger region, the little finger region), and a plurality of fingertip candidates may be detected in each of the plurality of finger regions.

  In the above description, the case where multi-tap input is performed with a plurality of fingers of one hand has been described as an example, but the same argument holds when multi-tap input is performed with both hands or a plurality of hands.

  As is clear from the above description, the processing unit 15 has a function of setting a contact target surface, a function of extracting a hand region, a function of detecting a fingertip candidate (end candidate), and a function of determining contact. Have. These functions may be realized by processing according to the program by the CPU, may be realized by hardware, or may be realized by processing according to the program by the CPU and hardware.

  In the present embodiment, the processing for acquiring the depth map in the calculation unit 133, the processing for setting the contact target surface in the processing unit 15, and the processing for extracting the hand region are all three-dimensional processing. And the process which sets the contact target surface in the process part 15, and the process which extracts a hand area are the processes using only a depth map. Moreover, the process which detects the edge part candidate in the process part 15 is a two-dimensional process. And the process which performs the contact determination in the process part 15 is a three-dimensional process.

  That is, in the present embodiment, using only a depth map, a combination of three-dimensional processing and two-dimensional processing is used to detect edge candidates of a plurality of branch portions (for example, fingers) of the input operation means (for example, hands), and further branch The edge candidates are narrowed down for each portion, and the contact determination of the edge candidates (nearest fingertip candidates) narrowed down for each branch portion is simultaneously performed.

  Here, 30 mm is used as the value of L2, but this value is an example. In the strict sense, L2 = 0 mm in the contact between the contact target surface and the tip of the input operation means. However, in practice, there is also a measurement error in the calculation unit 133, and even if it is not in contact, it may be more convenient to consider it as contact if it is close, and the value of L2 is several mm to It is preferable to set a value of several tens of mm.

  As is clear from the above description, according to the present embodiment, the distance measurement unit 13 constitutes the three-dimensional information acquisition unit of the present invention, and the processing unit 15 sets the setting unit, the conversion unit, the convex hull processing unit, and the end candidate. A detection means, an exclusion means, a ranking means, and a determination means are configured. In other words, the distance measuring unit 13 and the processing unit 15 constitute a contact detection device capable of simultaneously detecting a plurality of points of contact of the present invention.

  In the process performed by the processing unit 15, the contact detection method of the present invention is implemented.

  As described above, the projector device 10 according to the present embodiment includes the projection unit 11, the distance measurement unit 13, the processing unit 15, and the like.

  The projection unit 11 projects an image on the screen 300 based on an instruction from the processing unit 15. The distance measuring unit 13 includes a light emitting unit 131 that emits detection light (detection light) toward the projection image, an imaging optical system 132b, and an imaging element 132a, and includes at least a projection image and an input operation unit. The imaging unit 132 that captures one image and the calculation unit 133 that acquires a depth map from the imaging result of the imaging unit 132 are included.

  The processing unit 15 has a function of setting a contact target surface, a function of extracting a hand region, a function of detecting edge candidates, a function of determining contact, and the like. Based on the depth map from the distance measuring unit 13 The contact between the input operation means and the screen 300 is detected, and the input operation information indicated by the input operation means is obtained.

  When detecting the contact between the input operation means and the screen 300, the processing unit 15 extracts a hand region based on a depth map that is three-dimensional information, performs projective transformation on the hand region to two-dimensional information, A fingertip candidate is detected for each finger area. Then, the processing unit 15 narrows down the fingertip candidates again based on the three-dimensional information, and simultaneously performs contact determination of the narrowed fingertip candidates (nearest fingertip candidates). Here, the fingertip candidate is on the depth map, and contact determination is performed on the fingertip candidate. Therefore, the fingertip position and the contact determination position are the same. Moreover, since contact determination is performed with respect to each fingertip candidate, it is determined that it is a fingertip at the same time that it is determined as contact.

  In this case, it is possible to accurately detect the contact between the input operation unit and the screen 300 and the position of the input operation unit at that time, and therefore it is possible to acquire the input operation information with high accuracy.

  In the function of setting the contact target surface in the processing unit 15, the contact target surface is set at a position away from the screen 300 by a predetermined distance. In this case, it is possible to avoid the input operation means from entering the inside of the screen 300 (+ Y side of the projection plane) by calculating the measurement error in the distance measuring unit 13.

  With the function of extracting the hand region in the processing unit 15, when the input operation unit is within a predetermined distance L1 from the contact target surface with respect to the -a direction, the region including the input operation unit is extracted. In this case, as a step before detecting the edge candidate, it is considered that a portion whose distance from the contact target surface exceeds L1 is irrelevant to the contact, and it is possible to delete unnecessary information and reduce processing. .

  In the function of detecting the end candidate in the processing unit 15, the three-dimensional imaging information is converted into two-dimensional information by projective transformation, the convex hull processing is performed on the two-dimensional information, and the end candidate is detected. In this case, even a low-resolution image can be detected as an edge candidate if there is even one pixel as the tip.

  In the function of determining contact in the processing unit 15, when the closest fingertip candidates (closest edge candidate) of a plurality of finger areas are less than or equal to a predetermined value L2 with respect to the −a direction, The contact / non-contact of the nearest fingertip candidate of the finger region and the screen 300 is simultaneously determined. In this case, even if the tip of the input operation means is in a non-contact state with respect to the screen 300, it can be determined that the input operation means is in contact if it is close to the contact target surface. Therefore, even when an unspecified number of people use it or the input operation means is dirty, it can be determined that the user has touched the screen 300 even if he / she does not want to touch it directly.

  The light emitting unit 131 of the distance measuring unit 13 emits near infrared light. In this case, the arithmetic unit 133 can acquire a depth map with high accuracy even in an environment where there is much visible light. In addition, it is possible to suppress problems such as that the light emitted from the light emitting unit 131 interferes with the image (visible light) projected from the projector device 10 and the image becomes difficult to see.

  Further, the image sensor 132a of the distance measuring unit 13 has a two-dimensional image sensor. In this case, the depth map can be acquired in one shot.

  Further, since the projector system 100 according to the present embodiment includes the projector device 10, as a result, a desired image display operation can be correctly performed.

  In the above embodiment, the projector device 10 and the image management device 30 may be integrated.

  By the way, there is an OS (operating system) that supports multi-tap, such as Windows 10 / iOS / Android.

  In order to cope with such multi-tap, as shown in FIG. 15, it is required to perform contact determination for each of a plurality of (for example, four) fingers. That is, it is required to simultaneously determine whether or not each of a plurality of fingers is in contact. 15 to 21, for the sake of convenience, information imitating an actual hand is shown as the two-dimensional information of the hand region.

  However, for example, if four points close to the screen are extracted from a plurality of candidate points, contact determination may be performed by concentrating on one finger as shown in FIG.

  Therefore, the inventors have developed the contact detection device of the present embodiment to cope with multi-tap.

  The contact detection device according to the present embodiment is a contact detection device that detects contact between an object (here, a hand) having a plurality of branch portions (here, a plurality of fingers) and the screen 300, and is a three-dimensional view of the hand and the screen 300. A distance measuring unit 13 (three-dimensional information acquisition unit) that acquires information, a setting unit 50 that sets a contact target surface based on the three-dimensional information on the screen 300, and two-dimensional information by projective transformation of the three-dimensional information of the hand. Conversion means 52 for converting to a plurality of fingers, edge candidate detection means 53 for detecting a plurality of edge candidates (fingertip candidates) for each of a plurality of fingers, and a plurality of edge candidates for each of the plurality of fingers. And a ranking means 54 for comprehensively ranking the degree of proximity to the contact target surface, and whether or not the end candidate having the highest proximity ranking of each of the two-dimensional information of the plurality of fingers is in contact with the contact target surface. Or A determination means 55 for, and a. Here, the setting unit 50, the conversion unit 52, the end candidate detection unit 53, the ranking unit 54, and the determination unit 55 are components of the processing unit 15 (see FIG. 22).

  In this case, the candidate points for contact determination can be distributed to a plurality of fingers by detecting a plurality of edge candidates for each of the plurality of fingers. That is, it is possible to suppress the contact determination candidate points from being concentrated on a specific finger. Then, by performing contact determination between the edge candidate having the highest degree of proximity to the contact target surface (hereinafter also referred to as “nearest edge candidate”) among the plurality of edge candidates of each of the plurality of fingers, The presence / absence of contact between each of the plurality of fingers and the screen 300 can be simultaneously determined with high accuracy.

  As a result, it is possible to support multi-tap.

  In the processing unit 15, the ranking means 54 is not essential. In other words, the closest edge candidate of each finger region can be obtained by comparing the proximity of the plurality of edge candidates to the contact target surface without giving a comprehensive ranking to the plurality of edge candidates. .

  In the present embodiment, the “detection system” of the present invention is configured to include the distance measuring unit 13, the setting unit 50, the conversion unit 52, and the end candidate detection unit 53. However, the present invention is not limited to this. However, it can be changed as appropriate.

  For example, the “detection system” of the present invention may detect a plurality of end candidates for each of a plurality of branch portions by a pattern matching method using a template instead of the convex hull processing.

  Also, the ranking means 54 ranks the closest candidate to the edge candidate having the highest degree of proximity to the contact target surface of the plurality of fingers, and the determination means 55 determines the ranking result by the ranking means 54. It is preferable to make a determination based on the above.

  At this time, for example, among the closest end portion candidates of a plurality of fingers, those in which the degree of proximity is in the upper half are determined as “contacting”, and those in the lower half are determined as “not in contact” You may do it. Further, for example, a location having the highest degree of proximity among the closest locations of a plurality of fingers may be determined to be “in contact”, and other locations may be determined to be “not in contact”. Further, for example, it is possible to determine that the position with the lowest degree of proximity among the closest points of a plurality of fingers is determined as “not in contact”, and determine other points as “in contact”.

  Note that the ranking of “nearest proximity” may not be performed. That is, step S113 in FIG. 7 may be omitted. In this case, as a method for determining contact, when the distance between the closest edge candidate and the contact target surface is less than a reference value (for example, several mm to 30 mm), it is determined that the contact is made, and when the distance is equal to or greater than the reference value It may be determined that “not in contact”. Also, the distance from the closest edge candidate of the plurality of fingers to the contact target surface (for example, the distance in the a direction between the closest edge candidate and the contact target surface, or the length of the perpendicular from the closest edge candidate to the contact target surface) Is determined to be “in contact” with a finger whose distance from the closest point to the contact target surface is less than the average value. You may determine with "not contacting".

  Further, when there is a fingertip candidate within a predetermined distance L2 from the contact target surface (YES in step S109 in FIG. 7), the fingertip candidate is considered to be in contact with the contact target surface (determined). ), Steps S113 to S115 in FIG. 7 may be omitted. In this case, the determination of the contact / non-contact between the closest fingertip candidate of each of the plurality of finger regions and the contact target surface by the determination unit 55 is substantially made in step S109.

  Moreover, it is preferable that the edge part candidate detection means 53 detects a some edge part candidate within the range according to the distance from the ranging part 13 to a contact target surface.

  In this case, the exclusion range can be expanded in accordance with the distance from the distance measuring unit 13 to the contact target surface, which is a noise countermeasure for the distance measuring unit 13.

  By the way, it is preferable to set the exclusion range (inside the broken circle) shown in FIG. 18 or the like according to the distance from the depth sensor (three-dimensional information acquisition unit) to the object, the distance in the a direction in FIG. Depth sensor (three-dimensional information acquisition part) by expanding the exclusion range that expands according to the above distance, assuming that the thickness of the finger is different depending on the adult and child, physique difference, gender, etc., and the solid angle in the a direction is the same ) Noise countermeasures.

  Further, the distance measuring unit 13 as the three-dimensional information acquisition unit uses the TOF method.

  However, when the three-dimensional information acquisition means uses the TOF method, a slight change is required. For example, the depth value can be obtained in detail if it is a depth sensor of TOF type such as Microsoft's Kinect V2 or a type of depth sensor that measures the distance to the target object of light emitted by itself, It is known that the influence of noise components caused by subtle atmospheric fluctuations and thermal noise of sensors increases.

  Therefore, when the distance from the distance measuring unit 13 to the contact target surface is L, and the average of the thicknesses of the plurality of fingers is X, the plurality of end portion candidates are within a circle whose diameter is larger than X / L. By being present (setting the inside of the circle as an exclusion range), the influence of noise components can be reduced. Note that the value of X can be obtained from the hand region (two-dimensional information).

  Further, a pattern projection method or a stereo camera method may be used as the three-dimensional information acquisition unit. An example of the pattern projection method is Microsoft's Kinect V1.

  At this time, when the distance from the three-dimensional information acquisition unit to the contact target surface is L, the average thickness of the plurality of fingers is X, and the focal length of the three-dimensional information acquisition unit is f, the plurality of edge candidate is It is preferable to be in a circle whose diameter is a value larger than Xf / L. In this case, the influence of noise components can be reduced. Note that the value of X can be obtained from the hand region (two-dimensional information).

  In addition, it is preferable that a plurality of end portion candidates be detected as the distance from the three-dimensional information acquisition unit to the contact target surface decreases.

  Moreover, it is preferable that the detection system further includes a convex hull processing means 57 that performs a convex hull processing on the two-dimensional information, and the plurality of end candidates are obtained by the convex hull processing. The convex hull processing means 57 can be a component of the processing unit 15 (see FIG. 22).

  In the above embodiment, the hand area is obtained by calculating the vertex of the two-dimensional information from the point cloud data obtained from the depth sensor by the convex hull, which is very simple. Because it is simple, the depth sensor can be established immediately as a stereo camera method, pattern projection method, or TOF method. However, convex hull processing may be performed after preparing data that makes it easy to obtain a human silhouette by machine learning or the like. . In the case of Kinect V1, a development kit is released by Microsoft, and a skeleton image of the human body can be obtained by the function Nui Skeleton Get Next Frame. Many libraries are open to the public regarding machine learning. If a result obtained by preparing an image including a plurality of hands and performing machine learning in advance is used, a system similar to a Microsoft-provided library can be constructed.

  That is, the detection system may further include a machine learning result applying unit that applies the machine learning result to the two-dimensional information obtained by projectively transforming the three-dimensional information of the hand before the convex hull processing is performed. The machine learning result applying unit can be a component of the processing unit.

  In particular, in the case of a stereo camera, an RGB camera is always prepared. By using either the left or right RGB camera image and the machine learning result, it is possible to suppress fluctuations in the convex hull processing result due to the inferior depth sensor accuracy.

  The contact detection apparatus further includes an exclusion processing unit 58 (exclusion unit) that excludes end candidates other than the end candidate having the highest degree of proximity to the contact target surface from the plurality of end candidates of each of the plurality of fingers. It is preferable to provide. The exclusion processing unit 58 can be a component of the processing unit 15 (see FIG. 22). By including at least one of the ranking means 54 and the exclusion processing means 58, the “nearest edge candidate acquisition system” of the present invention can be configured.

  The projector device 10 includes a projection unit 11 that projects an image on a projection surface, and a contact detection device according to the present embodiment for detecting contact between the projection surface and a hand (object).

  In this case, a projector apparatus that can use the multi-tap function can be realized.

  In addition, according to the projector system including the projector device 10 and the image management device 30 (control device) that performs image control based on the input operation obtained by the projector device 10, the interactive system that can use the multi-tap function. A simple system.

  The contact detection method according to the present embodiment is a contact detection method for detecting contact between a hand (object) having a plurality of fingers (branch portions) and the screen 300 (target object). A step of acquiring information, a step of setting a contact target surface based on the three-dimensional information of the screen 300, a step of converting the three-dimensional information of the hand into two-dimensional information by projective transformation, and the two-dimensional information Detecting a plurality of edge candidates for each of the plurality of fingers, performing a comprehensive ranking of the proximity to the contact target surface on the plurality of edge candidates for each of the plurality of fingers, and a plurality of branch portions. Determining whether or not the end candidate having the highest degree of proximity among the plurality of end candidates is in contact with the contact target surface.

  In this case, the candidate points for contact determination can be distributed to a plurality of fingers by detecting a plurality of edge candidates for each of the plurality of fingers. That is, it is possible to suppress the contact determination candidate points from being concentrated on a specific finger. Then, by performing contact determination between the edge candidate having the highest degree of proximity to the contact target surface (hereinafter also referred to as “nearest edge candidate”) among the plurality of edge candidates of each of the plurality of fingers, The presence / absence of contact between each of the plurality of fingers and the screen 300 can be simultaneously determined with high accuracy.

  As a result, it is possible to support multi-tap.

  In the contact detection method of the present embodiment, the step of ranking is not essential. In other words, the closest edge candidate of each finger region can be obtained by comparing the proximity of the plurality of edge candidates to the contact target surface without giving a comprehensive ranking to the plurality of edge candidates. .

  In the present embodiment, the present invention includes a step of acquiring three-dimensional information, a step of setting a contact target surface, a step of converting into two-dimensional information, and a step of detecting a plurality of edge candidates. The “detecting step” is configured, but is not limited to this, and can be changed as appropriate.

  For example, in the “detecting step” of the present invention, instead of the convex hull processing, a plurality of edge candidates for each of a plurality of branch portions may be detected by a pattern matching method using a template.

  In addition, as a program for causing the image management apparatus 30 (computer) to execute the contact detection method of the present embodiment, a contact target surface is obtained based on a procedure for acquiring three-dimensional information of the hand and the screen 300 and the three-dimensional information of the screen 300. A procedure for converting the three-dimensional information of the hand into two-dimensional information by projective transformation, a procedure for detecting a plurality of edge candidates for each of a plurality of fingers based on the two-dimensional information, A procedure for comprehensively ranking the proximity to the contact target surface for the plurality of edge candidates for each finger, and the edge candidate having the highest proximity ranking among the plurality of edge candidates for each of the plurality of branch portions And a procedure for determining whether or not the object is in contact with the contact target surface.

  In this program, the procedure for ranking is not essential. In other words, the closest edge candidate of each finger region can be obtained by comparing the proximity of the plurality of edge candidates to the contact target surface without giving a comprehensive ranking to the plurality of edge candidates. .

  Furthermore, a storage medium (for example, a memory or a hard disk) in which the program is stored can be provided.

  In the above-described embodiment, the case of direct multi-touch by hand has been described as an example, but the same argument holds for the case of multi-touch while wearing a glove, for example.

  Moreover, in the said embodiment, the ranging part 13 may be externally attached in the state which can be removed with respect to a housing | casing via the attachment member not shown (refer FIG. 23). In this case, the depth map acquired by the distance measuring unit 13 is notified to the processing unit 15 in the housing via a cable or the like. In this case, the distance measuring unit 13 can be arranged at a position away from the housing.

  In the above embodiment, at least a part of the processing in the processing unit 15 may be performed by the image management apparatus 30. For example, when the input operation information acquisition process is performed by the image management device 30, the depth map acquired by the distance measuring unit 13 is notified to the image management device 30 via a cable or by wireless communication.

  In the above-described embodiment, at least a part of the processing in the processing unit 15 may be performed by the calculation unit 133. For example, processing (steps S103 to S114) for detecting contact in the input operation information acquisition processing may be performed by the calculation unit 133.

  In the above embodiment, the projector device 10 may include a plurality of distance measuring units 13. For example, when the angle of view in the X-axis direction is very large, a plurality of imaging optical systems having a reduced angle of view can be used rather than covering the angle of view with a single ranging unit 13 having an ultra-wide-angle imaging optical system. It may be cheaper to arrange the distance measuring units 13 along the X-axis direction. That is, it is possible to realize a projector device having an ultra-wide angle in the X-axis direction at a low cost.

  As an example, FIG. 24 shows a configuration example of the distance measuring unit 13 (three-dimensional information acquisition unit) using the pattern projection method. Here, the light emitting unit 131 of the distance measuring unit 13 emits a certain structured light. The structured light is light suitable for a method known as the structured light method, and includes, for example, striped light and matrix light. The irradiation area is naturally wider than the projected image. Since the emitted light is near-infrared light, there is no inconvenience such as difficulty in seeing the projected image. At this time, the imaging unit 132 images the structured light that is reflected and deformed by the imaging object. The computing unit 133 compares the light emitted from the light emitting unit 131 with the light imaged by the imaging unit 132, and obtains a depth map based on the triangulation method. Here, the focal length of the imaging unit 132 is referred to as the focal length of the distance measuring unit 13 for convenience.

  Further, as an example, as in the modification of the distance measuring unit 13 using the TOF (Time-Of-Flight) method shown in FIG. 25, the light emitting unit 131 of the distance measuring unit 13 is intensity-modulated at a predetermined frequency. Light may be emitted. The irradiation area is naturally wider than the projected image. Since the emitted light is near-infrared light, there is no problem such that the projected image is difficult to see. The imaging unit 132 includes a single two-dimensional imaging element capable of measuring a phase difference and an imaging optical system. At this time, the imaging unit 132 images light that is reflected by the imaging object and has a phase shift. The computing unit 133 compares the light emitted from the light emitting unit 131 with the light imaged by the imaging unit 132, and obtains a depth map based on the time difference or the phase difference.

  As an example, FIG. 26 shows a configuration example of the distance measuring unit 13 using the stereo camera method. Here, the light emitting unit 131 of the distance measuring unit 13 emits light for applying a texture to the object. The irradiation area is naturally wider than the projected image. Since the emitted light is near-infrared light, there is no problem such that the projected image is difficult to see. In this case, it has the two imaging parts 132 which image the texture pattern projected on the imaging target object. Therefore, there are two optical axes corresponding to each imaging unit. Then, the calculation unit 133 calculates a depth map based on the parallax between images captured by the two imaging units 132. That is, the arithmetic unit 133 performs a process called stereo parallelization on each image, and converts an image when it is assumed that the two optical axes are parallel. Therefore, the two optical axes do not have to be parallel. Note that the optical axes after the stereo collimation process overlap when viewed from the X-axis direction (see FIG. 27), and correspond to the optical axis of the distance measuring unit 13 in the above embodiment. Here, the focal length of the imaging unit 132 is referred to as the focal length of the distance measuring unit 13 for convenience.

  Moreover, although the said embodiment demonstrated the case where the projector apparatus 10 was mounted and used for the mounting base 400, it is not limited to this. For example, as shown in FIG. 28, the projector device 10 may be used by being suspended from a ceiling. Here, the projector device 10 is fixed to the ceiling with a suspension member.

  Moreover, in the said embodiment, a projection surface is not limited to a plane. In the method of determining contact with the projection surface and the closest point to the screen in the finger area instead of the fingertip, there is a case where the projection surface is erroneously detected if it is not a flat surface.

  As shown in FIG. 29 as an example, the distance measuring unit 13 and the processing unit 15 can be used even when there is a step on the surface of the object. In this case, even if the input operation means other than the tip part touches the stepped portion, it is possible to determine the fingertip and determine the contact without erroneous detection.

  For example, if the back of the hand touches the corner of the step, it is determined that the contact is made at that position in the conventional method. However, in the present invention, the back of the hand is not a fingertip candidate, so it is not determined that the contact is made. Since the fingertip is determined based on the distance between the fingertip candidate and the contact target surface, and the contact is determined, no erroneous detection occurs.

  Even in this case, the three-dimensional information of all the end candidate points Kj is within a certain distance (30 mm in this case) in the -a direction from the contact target surface, and is the most touch target surface for each finger region. Search for an edge candidate point close to. If there is a corresponding end candidate, the end candidate is the tip of the input operation means, and the tip is in contact with the contact target surface.

  That is, in the above embodiment, even if there is a step on the contact target surface, the contact detection can be performed with high accuracy because it is based on the three-dimensional information of the contact target surface.

  The distance measuring unit 13 and the processing unit 15 can also be used when the surface of the object includes a curved surface as shown in FIG. 30 as an example. For example, the target object may be a blackboard used in an elementary school or a junior high school. Even in this case, contact detection with high accuracy is possible because it is based on the three-dimensional information of the contact target surface.

  The distance measuring unit 13 and the processing unit 15 can also be used for an electronic blackboard device or a digital signage device.

  FIG. 31 shows an example of an electronic blackboard device. The electronic blackboard device 500 includes a projection panel (object) on which various menus and command execution results are displayed, a panel unit 501 that stores a coordinate input unit, a storage unit that stores a controller and a projector unit, a panel unit 501, It comprises a stand that supports the storage unit at a predetermined height, and a device storage unit 502 that stores a computer, a scanner, a printer, a video player, and the like (see JP 2002-278700 A). The contact detection device including the distance measurement unit 13 and the processing unit 15 is stored in the device storage unit 502, and the contact detection device appears when the device storage unit 502 is pulled out. The contact detection device detects contact between the input operation means and the projection panel. Communication between the controller and the processing unit 15 may be wired communication via a cable such as a USB cable, or may be wireless communication.

  FIG. 32 shows an example of a digital signage apparatus. In the digital signage apparatus 600, glass is an object, and the surface of the glass is a projection plane. The image is rear projected from the rear of the glass by the projector body. The contact detection device including the distance measuring unit 13 and the processing unit 15 is installed on a handrail. Communication between the projector main body and the processing unit 15 is wired communication via a USB cable. Thereby, an interactive function can be given to the digital signage device.

  Thus, the distance measuring unit 13 and the processing unit 15 are suitable for a device having an interactive function or a device to which an interactive function is to be added.

  Below, the thought process that the inventors have come up with the above embodiment will be described.

  2. Description of the Related Art In recent years, so-called interactive projector apparatuses having a function of writing characters and drawings on a projected image projected on a screen and a function of executing operations such as enlargement, reduction, and page feed of the projected image are commercially available. These functions use an operator's finger touching the screen, a pen and a pointing rod held by the operator as input operation means, and the position where the tip of the input operation means contacts the screen (object) and This is realized by detecting movement and sending the detection result to a computer or the like.

  For example, in Patent Document 1, an acquisition unit that acquires images captured by a plurality of cameras in time series, a calculation unit that calculates a distance from the camera to the object based on the images, and the object is a predetermined XY. A correction unit that corrects the calculated distance to the distance from the camera to the XY plane when the difference in area of the object between the plurality of images acquired in time series is equal to or less than a predetermined threshold when the plane is reached There is disclosed a position calculation system including:

  Further, Patent Document 2 discloses a detection unit that detects a fingertip within a predetermined range from a display screen from an image captured by an imaging unit, and a three-dimensional coordinate calculation unit that calculates a three-dimensional coordinate of the detected fingertip. , According to the coordinate processing means for associating the calculated three-dimensional coordinates of the fingertip and the two-dimensional coordinates on the display screen, the two-dimensional coordinates on the associated display screen, and the distance between the fingertip and the display screen Thus, there is disclosed an image display device comprising image display means for displaying an image in a lower hierarchy of the currently displayed image on a display screen.

  In Patent Document 3, a distance image sensor is used to detect a position of an object existing on a predetermined surface at a point of interest, and a detected position and a periphery of the position at the point of interest. Identifying means for identifying the edge of the object from the color image, estimation means for estimating the position of the identified edge based on the position of the object, and determination for determining contact with the surface according to the position of the edge An information processing apparatus having a means is disclosed.

  Further, in Patent Document 4 and Patent Document 5, a projector that projects an image on a projection surface, a depth camera that continuously acquires an image of an environment on the projection surface, and depth information obtained in an initial state from the depth camera, An initial depth map is constructed, and the depth map processing device that determines the position of the touch motion area based on the initial depth map and the images obtained continuously from the depth camera from the depth camera before the determined touch motion area. Corresponding to each blob based on the relationship in time and space between the object detection device that detects at least one candidate candidate blob located within a predetermined interval and the center of gravity of the blob obtained from adjacent images An automatic switching system in a bi-directional mode in a virtual touch screen system is disclosed having a tracking device in a point array.

  However, Patent Documents 1 to 5 have room for improvement with regard to handling multi-tap.

  Thus, the inventors have come up with the above embodiment to cope with multi-tap.

  DESCRIPTION OF SYMBOLS 10 ... Projector apparatus, 11 ... Projection part (part of projector apparatus), 13 ... Distance measurement part (three-dimensional information acquisition means, a part of detection system), 30 ... Image management apparatus (control apparatus), 50 ... Setting means (Part of the detection system), 52 ... conversion means (part of the detection system), 53 ... end candidate detection means (part of the detection system), 54 ... ranking means (one of the closest end part candidate acquisition systems) Part), 55 ... determination means, 57 ... convex hull processing means (part of the detection system), 58 ... exclusion processing means (exclusion means, part of the nearest edge candidate acquisition system), 100 ... projector system, 131 ... Light emitting unit (part of three-dimensional information acquisition unit), 132 ... imaging unit (part of three-dimensional information acquisition unit), 300 ... screen (object), 500 ... electronic blackboard device, 600 ... digital signage device.

JP 2014-202540 A JP 2008-210348 A JP 2012-48393 A JP 2013-8368 A Japanese Patent Laying-Open No. 2015-212927

Claims (20)

A contact detection device for detecting contact between an object having a plurality of branch portions and an object,
A detection system for detecting a plurality of edge candidates for each of the plurality of branch portions;
A closest edge candidate acquisition system for obtaining a closest edge candidate that is an edge candidate closest to the object from the plurality of edge candidates of each of the plurality of branch portions;
A contact detection device comprising: a determination unit that determines whether or not the closest end portion candidate of each of the plurality of branch portions is in contact with the object. The detection system is
3D information acquisition means for acquiring 3D information of the object and the object;
Setting means for setting a contact target surface based on the three-dimensional information of the object;
Conversion means for converting the three-dimensional information of the object into two-dimensional information by projective transformation;
The contact detection device according to claim 1, further comprising: an end candidate detection unit that detects a plurality of end candidates of each of the plurality of branch portions based on the two-dimensional information.   The contact detection according to claim 2, wherein the end candidate detection unit detects the plurality of end candidates within a range corresponding to a distance from the three-dimensional information acquisition unit to the contact target surface. apparatus.   The contact detection apparatus according to claim 2, wherein the three-dimensional information acquisition unit uses a TOF method.   When the distance from the three-dimensional information acquisition means to the contact target surface is L, and the average thickness of the plurality of branch portions is X, the plurality of edge candidates have a value larger than X / L as a diameter. The contact detection device according to claim 4, wherein the contact detection device is in a circle.   The contact detection apparatus according to claim 2, wherein the three-dimensional information acquisition unit uses a pattern projection method or a stereo camera method.   When the distance from the three-dimensional information acquisition unit to the contact target surface is L, the average thickness of the plurality of branch portions is X, and the focal length of the three-dimensional information acquisition unit is f, the plurality of ends The contact detection device according to claim 6, wherein the part candidate is in a circle whose diameter is a value larger than Xf / L.   The contact detection apparatus according to any one of claims 2 to 7, wherein the plurality of edge candidates are detected as the distance from the three-dimensional information acquisition unit to the contact target surface is shorter. . The detection system further includes a convex hull processing means for performing a convex hull processing on the two-dimensional information,
The contact detection device according to claim 2, wherein the plurality of end portion candidates are obtained by the convex hull process.   The contact detection apparatus according to claim 9, wherein the detection system further includes a machine learning result application unit that applies a machine learning result to the two-dimensional information before the convex hull processing is performed. The closest edge candidate acquisition system includes ranking means for performing a comprehensive ranking of the proximity to the object to the plurality of edge candidates of each of the plurality of branch portions,
The contact detection apparatus according to claim 1, wherein the determination unit performs the determination based on a ranking result obtained by the ranking unit.   The nearest edge candidate acquisition system includes an exclusion unit that excludes edge candidates other than the edge candidate closest to the object from the plurality of edge candidates of each of the plurality of branch portions. The contact detection device according to any one of claims 1 to 11, wherein the contact detection device is characterized in that A projection unit that projects an image onto a projection surface;
A contact detection device according to any one of claims 1 to 12, which detects contact between the projection surface and the object.   An electronic blackboard device comprising the contact detection device according to any one of claims 1 to 12.   A digital signage apparatus provided with the contact detection apparatus as described in any one of Claims 1-12. A projector device according to claim 13;
A projector system comprising: a control device that performs image control based on an input operation obtained by the projector device. A contact detection method for detecting contact between an object having a plurality of branch portions and an object,
Detecting a plurality of end candidates for each of the plurality of branch portions;
Obtaining a closest edge candidate that is an edge candidate closest to the object from the plurality of edge candidates of each of the plurality of branch portions;
Determining whether or not the nearest edge candidate of each of the plurality of branch portions is in contact with the object. The detecting step includes
A sub-step of obtaining three-dimensional information of the object and the object;
A sub-step of setting a contact target surface based on the three-dimensional information of the object;
A sub-step of converting the three-dimensional information of the object into two-dimensional information by projective transformation;
The contact detection method according to claim 17, further comprising: a sub-step of detecting a plurality of end portion candidates for each of the plurality of branch portions based on the two-dimensional information. A program for causing a computer to execute the contact detection method according to claim 18,
Obtaining three-dimensional information of the object and the object;
A procedure for setting a contact target surface based on the three-dimensional information of the object;
A procedure for converting the three-dimensional information of the object into two-dimensional information by projective transformation;
A procedure for detecting a plurality of end candidates for each of the plurality of branch portions based on the two-dimensional information;
Obtaining a closest edge candidate that is an edge candidate closest to the object from the plurality of edge candidates of each of the plurality of branch portions;
A step of determining whether or not an end candidate having the highest degree of proximity among the closest end candidate of each of the plurality of branch portions is in contact with the contact target surface.   A storage medium storing the program according to claim 19.