CN114179082B - Image force touch detection device and reproduction method based on contact force information - Google Patents
Image force touch detection device and reproduction method based on contact force information Download PDFInfo
- Publication number
- CN114179082B CN114179082B CN202111484857.1A CN202111484857A CN114179082B CN 114179082 B CN114179082 B CN 114179082B CN 202111484857 A CN202111484857 A CN 202111484857A CN 114179082 B CN114179082 B CN 114179082B
- Authority
- CN
- China
- Prior art keywords
- image
- information
- force sensor
- base
- eva foam
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 23
- 238000000034 method Methods 0.000 title claims abstract description 14
- 239000006260 foam Substances 0.000 claims abstract description 22
- VYPSYNLAJGMNEJ-UHFFFAOYSA-N Silicium dioxide Chemical compound O=[Si]=O VYPSYNLAJGMNEJ-UHFFFAOYSA-N 0.000 claims abstract description 16
- 239000000741 silica gel Substances 0.000 claims abstract description 16
- 229910002027 silica gel Inorganic materials 0.000 claims abstract description 16
- 239000000758 substrate Substances 0.000 claims abstract description 10
- 239000011159 matrix material Substances 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 230000004927 fusion Effects 0.000 claims description 3
- 238000000605 extraction Methods 0.000 abstract description 4
- 239000010408 film Substances 0.000 description 20
- 238000010586 diagram Methods 0.000 description 6
- 229920001296 polysiloxane Polymers 0.000 description 4
- 230000003993 interaction Effects 0.000 description 3
- 239000010409 thin film Substances 0.000 description 2
- 241000282414 Homo sapiens Species 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1628—Programme controls characterised by the control loop
- B25J9/1633—Programme controls characterised by the control loop compliant, force, torque control, e.g. combined with position control
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01L—MEASURING FORCE, STRESS, TORQUE, WORK, MECHANICAL POWER, MECHANICAL EFFICIENCY, OR FLUID PRESSURE
- G01L1/00—Measuring force or stress, in general
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01L—MEASURING FORCE, STRESS, TORQUE, WORK, MECHANICAL POWER, MECHANICAL EFFICIENCY, OR FLUID PRESSURE
- G01L1/00—Measuring force or stress, in general
- G01L1/04—Measuring force or stress, in general by measuring elastic deformation of gauges, e.g. of springs
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Force Measurement Appropriate To Specific Purposes (AREA)
- Manipulator (AREA)
Abstract
The invention relates to an image force touch detection device and a reproduction method based on contact force information, wherein the detection device comprises a base, a substrate silica gel pad, a film force sensor, a surface silica gel pad and an EVA foam pad; the base is fixedly arranged with the tail end flange plate of the robot, and a plurality of image sensors are arranged on the side edge of the base and used for collecting surface image information of an object to be measured; the substrate silica gel pad, the film force sensor, the surface silica gel pad and the EVA foam cushion are sequentially laminated and fixed on the lower surface of the base, the EVA foam cushion senses the surface shape of a measured object through contact deformation, deformation pressure is conducted onto the film force sensor, and the film force sensor is used for detecting deformation pressure values of different parts. The detection device provided by the invention has a compact structure, the shape information of the surface of the detected object can be accurately expressed through the contact force information, and the image force touch reproduction method guided by the information effectively improves the extraction quality of the image features.
Description
Technical Field
The invention belongs to the technical field of man-machine interaction, and particularly relates to an image force touch detection device and a reproduction method based on contact force information.
Background
The force touch man-machine interaction technology is an important component of man-machine interaction in virtual reality, and by simulating the force sense feeling process of human beings on real objects, the force touch information in the virtual environment is truly fed back to people, so that the virtual interactivity and the degree of presence are greatly improved. While graphics as a projection of a three-dimensional scene onto a two-dimensional camera imaging plane retains rich information of the object surface, image-based haptic rendering technology has attracted widespread attention of researchers.
The two-dimensional image includes three-dimensional geometric shape information, two-dimensional pattern information, detail texture information, and the like. The various information carried by the two-dimensional image is subjected to force touch reproduction, and then fed back to an operator through a force touch reproduction device. The extraction of image features is a first step in the haptic reproduction of images and is also a key step in determining the haptic model and haptic reproduction effects. However, the present method simply relies on the color information of the image to perform the haptic modeling, and the method is easy to calculate, but is easily affected by various parameters of the image itself. The surface shape information of the object can be detected based on the force sensor matrix, but the resolution of the force sensor matrix is limited by the size of the force sensor, so that the surface texture information of the object cannot be detected. In order to solve the above problems, it is needed to provide a device and a method for improving the image feature extraction quality in the image force touch reproduction.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides an image force touch detection device and a reproduction method based on contact force information, which adopt the following technical scheme:
an image force touch detection device based on contact force information comprises a base, a substrate silica gel pad, a film force sensor, a surface silica gel pad and an EVA sponge pad;
the base is fixedly arranged with the tail end flange plate of the robot, and a plurality of image sensors are arranged on the side edge of the base and used for collecting surface image information of an object to be measured;
The substrate silica gel pad, the film force sensor, the surface silica gel pad and the EVA foam cushion are sequentially laminated and fixed on the lower surface of the base, the EVA foam cushion senses the surface shape of a measured object through contact deformation, deformation pressure is conducted onto the film force sensor, and the film force sensor is used for detecting deformation pressure values of different parts.
Further, the base is of a square structure, a convex groove structure is arranged at the central position of each side of the base, and the image sensor A, the image sensor B, the image sensor C and the image sensor D are respectively arranged in the groove structures of each side.
Further, the film force sensor is in a matrix form, and the detection range of the film force sensor comprises the deformation sensing range of the EVA foam cushion.
An image force touch reproduction method based on the detection device comprises the following steps:
Step 1: controlling the robot to move to the upper side of the measured object, collecting surface image information of the measured object from different angles through a plurality of image sensors, and splicing the collected surface image information into a complete image capable of reflecting the surface information of the measured object through an image fusion algorithm;
step 2: controlling the robot to move towards the detected object, enabling the EVA foam cushion of the detection device to contact with the detected object and deform, and obtaining deformation pressure values of different parts detected by the film force sensor;
step 3: acquiring the conversion relation between the coordinate system of the complete image and the coordinate system of the film force sensor in the step 1, and corresponding deformation pressure values generated by different parts of the measured object on the EVA foam cushion to the surface parts of the measured object in the complete image;
Step 4: obtaining image depth information from the complete image obtained in the step 1 through an image processing algorithm, and obtaining three-dimensional shape information of the surface of the object to be measured by combining deformation pressure values of different parts detected by the film force sensor;
Step 5: and using the three-dimensional shape information of the surface of the measured object, and enabling a person to sense the shape of the surface of the measured object through the force feedback equipment.
The beneficial effects of the invention are as follows: (1) The image force touch detection device provided by the invention has a compact structure, and can accurately express the shape information of the surface of the object to be detected through the contact force information; (2) The image force touch reproduction is guided through the contact force information, so that the extraction quality of the image features is effectively improved.
Drawings
FIG. 1 is a diagram of an application system of a detection device of the present invention;
FIG. 2 is a schematic diagram of the structure of the detecting device of the present invention;
FIG. 3 is a schematic diagram of a detecting device for detecting an object to be detected according to the present invention;
FIG. 4 is a schematic diagram of a thin film force sensor in the detection device of the present invention;
FIG. 5 is a schematic diagram showing deformation of EVA foam mat when the object to be measured is a spherical object;
FIG. 6 is a schematic diagram showing deformation of EVA foam mat when the object to be measured is a cylindrical object;
The device comprises a robot 101, an image force touch detection device 102, a measured object 103, a base 201, an image sensor A202, an image sensor B203, an image sensor C204, an image sensor D205, a substrate silica gel pad 206, a film force sensor 207, a surface silica gel pad 208 and an EVA foam pad 209.
Detailed Description
The invention will now be described in further detail with reference to the accompanying drawings.
The application system of the image force touch detection device based on contact force information is shown in fig. 1, wherein a detection device 102 is used for acquiring a surface image of a detected object 103 and force sense information of the surface of the detected object, and the detection device 102 is arranged on a terminal flange plate of a robot 101 and fixedly connected through screws.
As shown in fig. 2 and 3, the detection device 102 mainly includes a device base 201, four image sensors (202, 203, 204, and 205), a substrate silicone pad 206, a film force sensor 207, a surface silicone pad 208, and an EVA sponge pad 209. The device base 201 is fixedly installed on the end flange of the robot 101, the device base 201 is in a square structure, a rectangular groove is formed in the center of each side, and the image sensors 202, 203, 204 and 205 are respectively installed in the rectangular grooves of the device base and are used for collecting surface image information of the measured object 103. A substrate silicone pad 206 is glued to the bottom surface of the device base 201 and a thin film force sensor 207 is glued to the substrate silicone pad 206. As shown in fig. 4, the film force sensor 207 is in a matrix form, and can detect contact force information of multiple points, a layer of surface silica gel pad 208 is glued on the surface of the film force sensor 207, and an eva sponge pad 209 is glued on the surface silica gel pad 208.
As shown in fig. 5 and 6, when the object to be measured 103 detected by the detecting means 102 is spherical, a hemispherical recess is formed on the surface of the EVA foam pad 209. When the object 103 to be measured is cylindrical, a semi-cylindrical recess is formed on the surface of the EVA foam pad 209; the pressure generated on the film force sensor 207 is also different.
The invention also provides an image force touch reproduction method based on the contact force information, which mainly comprises the following steps:
Step 1: the robot 101 is controlled to move to the upper side of the object 103 to be measured, because the visual angle of a single image sensor is limited, the surface image information of the object 103 to be measured needs to be acquired through four image sensors (202, 203, 204, 205), and four different images are fused into one image through an image fusion algorithm.
Step 2: the robot 101 is controlled to move towards the measured object 103, the surface shape of the measured object 103 can cause deformation of the EVA foam cushion 209, so that force is transmitted to the film force sensor 207, the force value sensed by the film force sensor 207 is obtained through embedded equipment such as a singlechip, the force value is related to the concave-convex condition of the surface of the measured object 103, and when the surface of the measured object 103 is convex, the corresponding value in the film force sensor 207 is larger, and conversely, the value is smaller.
Step 3: after the image sensor and the film force sensor 207 are fixed, the conversion relation between the matrix coordinate system of the film force sensor 207 and the two-dimensional image coordinate system acquired and fused by the image sensor can be obtained, and the deformation pressure values generated by different parts of the surface of the measured object 103 on the EVA foam cushion are corresponding to the parts of the surface of the measured object in the complete image by utilizing the conversion relation between the two coordinate systems.
Step 4: the two-dimensional image information of the object surface acquired by the image sensor is subjected to image processing algorithm to obtain image depth information, then the object surface pressure information acquired by the film force sensor 207 is combined, so that more accurate three-dimensional shape information of the surface of the measured object 103 is obtained, and finally a person perceives the shape of the surface of the measured object 103 through force feedback equipment (geomic Touch).
It should be noted that the terms like "upper", "lower", "left", "right", "front", "rear", and the like are also used for descriptive purposes only and are not intended to limit the scope of the invention in which the invention may be practiced, but rather the relative relationship of the terms may be altered or modified without materially altering the teachings of the invention.
The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above examples, and all technical solutions belonging to the concept of the present invention belong to the protection scope of the present invention. It should be noted that modifications and adaptations to the invention without departing from the principles thereof are intended to be within the scope of the invention as set forth in the following claims.
Claims (1)
1. An image force touch reproduction method of an image force touch detection device based on contact force information is characterized in that the detection device comprises a base (201), a substrate silica gel pad (206), a film force sensor (207), a surface silica gel pad (208) and an EVA foam pad (209);
The base (201) is fixedly arranged with the end flange of the robot (101), and a plurality of image sensors are arranged on the side edge of the base and used for collecting surface image information of the object (103) to be measured;
The substrate silica gel pad (206), the film force sensor (207), the surface silica gel pad (208) and the EVA foam pad (209) are sequentially laminated and fixed on the lower surface of the base (201), the EVA foam pad (209) senses the surface shape of the measured object (103) through contact deformation, deformation pressure is conducted onto the film force sensor (207), and the film force sensor (207) is used for detecting deformation pressure values generated on the EVA foam pad (209) by different parts of the surface of the measured object (103);
The base (201) is of a square structure, a convex groove-shaped structure is arranged at the central position of each side of the base, and the image sensor A (202), the image sensor B (203), the image sensor C (204) and the image sensor D (205) are respectively arranged in the groove-shaped structure of each side;
the film force sensor (207) is in a matrix form, and the detection range of the film force sensor comprises the deformation sensing range of the EVA foam cushion (209);
the method comprises the following steps:
step 1: controlling the robot (101) to move to the upper side of the detected object (103), collecting surface image information of the detected object (103) from different angles through a plurality of image sensors, and splicing the collected surface image information into a complete image capable of reflecting the surface information of the detected object (103) through an image fusion algorithm;
Step 2: controlling the robot (101) to move towards the detected object (103), enabling the EVA foam cushion (209) of the detection device to be in contact with the detected object (103) and generate deformation, and obtaining deformation pressure values of different parts detected by the film force sensor (207);
Step 3: acquiring the conversion relation between the coordinate system of the complete image and the coordinate system of the film force sensor (207) in the step 1, and corresponding deformation pressure values generated by different parts of the measured object (103) on the EVA foam cushion (209) to the surface parts of the measured object (103) in the complete image;
Step 4: obtaining image depth information from the complete image obtained in the step1 through an image processing algorithm, and combining deformation pressure values of different parts detected by a film force sensor (207) to obtain three-dimensional shape information of the surface of the detected object (103);
Step 5: and using the three-dimensional shape information of the surface of the measured object (103) to enable a person to sense the shape of the surface of the measured object (103) through the force feedback equipment.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111484857.1A CN114179082B (en) | 2021-12-07 | 2021-12-07 | Image force touch detection device and reproduction method based on contact force information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111484857.1A CN114179082B (en) | 2021-12-07 | 2021-12-07 | Image force touch detection device and reproduction method based on contact force information |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114179082A CN114179082A (en) | 2022-03-15 |
| CN114179082B true CN114179082B (en) | 2024-09-13 |
Family
ID=80603609
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111484857.1A Active CN114179082B (en) | 2021-12-07 | 2021-12-07 | Image force touch detection device and reproduction method based on contact force information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114179082B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102147661A (en) * | 2011-03-28 | 2011-08-10 | 南京航空航天大学 | Device and method for testing haptic display deformation model of flexible object |
| DE102013017007A1 (en) * | 2013-10-14 | 2015-04-16 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Robot with a end manipulator arm with end effector and method for determining a force and torque input to an end effector of a robot |
| CN106927079A (en) * | 2017-03-21 | 2017-07-07 | 长春理工大学 | A kind of industrial detonator crawl and packaging system and method based on machine vision |
Family Cites Families (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5261786B2 (en) * | 2007-12-11 | 2013-08-14 | 国立大学法人 筑波大学 | 3D tactile sensor and 3D tactile sensing method |
| JP2009288033A (en) * | 2008-05-28 | 2009-12-10 | Univ Of Tokyo | Optical touch sensor |
| JP5448423B2 (en) * | 2008-11-18 | 2014-03-19 | 東海ゴム工業株式会社 | Tactile sensor and manufacturing method thereof |
| US8139076B2 (en) * | 2009-09-04 | 2012-03-20 | Panasonic Corporation | System, method, and program for determining luminance values of multiple polygons which represent a surface of a subject |
| JP5660531B2 (en) * | 2010-08-12 | 2015-01-28 | 国立大学法人名古屋大学 | Shape measuring apparatus and shape measuring method |
| WO2014043037A1 (en) * | 2012-09-11 | 2014-03-20 | SynTouch, LLC | Compliant tactile sensor with fluid-filled, sponge-like material |
| JP6200361B2 (en) * | 2014-03-25 | 2017-09-20 | 株式会社フジクラ | Pressure-sensitive sensor and input device |
| CN104299261B (en) * | 2014-09-10 | 2017-01-25 | 深圳大学 | Three-dimensional imaging method and system for human body |
| US10488276B2 (en) * | 2015-03-24 | 2019-11-26 | National University Of Singapore | Resistive microfluidic pressure sensor |
| CN105653042B (en) * | 2016-02-29 | 2018-07-20 | 东南大学 | Pen type power-tactile representation device towards multimedia mobile terminal interactive application |
| US10048778B2 (en) * | 2016-05-31 | 2018-08-14 | Microsoft Technology Licensing, Llc | Force sensor apparatus |
| CN106092382B (en) * | 2016-07-20 | 2018-09-11 | 山东大学 | A kind of touch sensor and detection method based on elastomer three-dimensional deformation |
| CN106052914A (en) * | 2016-07-21 | 2016-10-26 | 清华大学 | Touch measurement sensor based on visual sense |
| US11340773B2 (en) * | 2016-08-25 | 2022-05-24 | Parade Technologies, Ltd. | Using 3D touch for tracking objects on a wet touch surface |
| CN112673338B (en) * | 2018-07-10 | 2024-11-22 | 小利兰·斯坦福大学托管委员会 | Capacitive and tactile sensors and related sensing methods |
| US11642796B2 (en) * | 2019-03-29 | 2023-05-09 | Rios Intelligent Machines, Inc. | Tactile perception apparatus for robotic systems |
| CN110162175B (en) * | 2019-05-16 | 2022-04-19 | 腾讯科技（深圳）有限公司 | Vision-based tactile measurement method, device, equipment and storage medium |
| CN111055279B (en) * | 2019-12-17 | 2022-02-15 | 清华大学深圳国际研究生院 | Multi-mode object grabbing method and system based on combination of touch sense and vision |
| CN113447172B (en) * | 2021-06-11 | 2023-08-18 | 北京纳米能源与系统研究所 | Wearable pressure sensor based on natural veins and manufacturing method |
| CN113625877A (en) * | 2021-08-16 | 2021-11-09 | 扬州大学 | Quantitative description method of force-tactile subjective perception results based on perceptual space analysis |
-
2021
- 2021-12-07 CN CN202111484857.1A patent/CN114179082B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102147661A (en) * | 2011-03-28 | 2011-08-10 | 南京航空航天大学 | Device and method for testing haptic display deformation model of flexible object |
| DE102013017007A1 (en) * | 2013-10-14 | 2015-04-16 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Robot with a end manipulator arm with end effector and method for determining a force and torque input to an end effector of a robot |
| CN106927079A (en) * | 2017-03-21 | 2017-07-07 | 长春理工大学 | A kind of industrial detonator crawl and packaging system and method based on machine vision |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114179082A (en) | 2022-03-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6021812B2 (en) | Apparatus for sensing using tiles, sensor with a series of plates, object identification and method for multi-touch surfaces | |
| CN103336634B (en) | Based on touching detection system and the method for adaptive layered structured light | |
| CN104391578B (en) | A kind of real-time gesture control method of 3-dimensional image | |
| JP7162079B2 (en) | A recording medium for recording a method, system and computer program for remotely controlling a display device via head gestures | |
| KR101518727B1 (en) | A stereoscopic interaction system and stereoscopic interaction method | |
| CN102622081A (en) | Method and system for realizing somatic sensory interaction | |
| CN102246201B (en) | Image processing device and image processing method | |
| CN115100742B (en) | Meta universe exhibition and demonstration experience system based on space-apart gesture operation | |
| CN109753153B (en) | Haptic interaction device and method for 360-degree suspended light field three-dimensional display system | |
| JP2015075429A (en) | Marker, evaluation method of marker, information processing apparatus, information processing method, and program | |
| CN111722769A (en) | Interaction method, interaction device, display equipment and storage medium | |
| CN112805660B (en) | System and method for human interaction with virtual objects | |
| CN106814963A (en) | A kind of human-computer interaction system and method based on 3D sensor location technologies | |
| CN111399634A (en) | Gesture-guided object recognition method and device | |
| WO2018028046A1 (en) | Virtual reality interaction system and method | |
| CN104349157A (en) | 3D display device and method thereof | |
| CN114179082B (en) | Image force touch detection device and reproduction method based on contact force information | |
| CN115617178B (en) | Method for completing key and function triggering by no contact between finger and vehicle | |
| CN111179341B (en) | Registration method of augmented reality equipment and mobile robot | |
| CN103440036B (en) | The display of 3-D view and interactive operation method and device | |
| JP2009288033A (en) | Optical touch sensor | |
| CN109426336A (en) | A kind of virtual reality auxiliary type selecting equipment | |
| CN112292656B (en) | Image display system, image display method, and computer-readable recording medium storing computer program | |
| CN114544052B (en) | Touch sensor, robot, method and device for achieving touch information acquisition | |
| Schlattmann et al. | Markerless 4 gestures 6 DOF real‐time visual tracking of the human hand with automatic initialization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |