JP6248780B2 - Pulse wave detection device, pulse wave detection method, and pulse wave detection program - Google Patents

Description

  The present invention relates to a pulse wave detection device, a pulse wave detection method, and a pulse wave detection program.

  As an example of a technique for detecting a change in blood volume, that is, a so-called pulse wave, a heartbeat measuring method for measuring a heartbeat from an image taken by a user has been proposed. In such a heartbeat measuring method, a face area is recognized from an image captured by a Web camera, and an average luminance value in the face area is calculated for each RGB component. In the heart rate measuring method, after applying ICA (Independent Component Analysis) to the time series data of the luminance average value for each RGB, FFT (Fast Fourier Transform) is applied to one of the three component waveforms after ICA. Apply. In addition, in the heart rate measurement method, the heart rate is estimated from the peak frequency obtained by FFT.

JP 2011-130996 A International Publication No. 2009/150793

“Advancements in Non-contact, Multiparameter Physiological Measurements Using a Webcam”, Vol.18, No.10, 2010

  However, as described below, the above-described technique has a limit in improving the detection accuracy of the pulse wave.

  That is, in the above heart rate measurement method, facial organ detection is used for recognition of a face region, but a good detection result is not always obtained from all images used for heart rate measurement. An area that is not a face may be misrecognized as a face. In this case, since the heart rate component is not included in the region erroneously recognized as the face, the heart rate detection accuracy is lowered. Thus, in the above heart rate measurement method, since the heartbeat detection accuracy depends on the face recognition accuracy, the heartbeat detection accuracy also decreases as the face recognition accuracy deteriorates.

  In one aspect, an object of the present invention is to provide a pulse wave detection device, a pulse wave detection method, and a pulse wave detection program that can improve the detection accuracy of a pulse wave.

  An aspect of the pulse wave detection device includes an acquisition unit that acquires an image captured by a camera, a scanning unit that scans a region of interest used for pulse wave detection in the acquired image, and an image A plurality of generation units for generating a plurality of first pulse wave waveforms from pixel values of a region of interest included in the combination for each combination in which the region of interest scanned for each image is combined between a plurality of images; The calculation unit that calculates the similarity between the first pulse wave waveform and the second pulse wave waveform set as a reference, and the highest similarity between the second pulse wave waveform and the second pulse wave waveform And a selection unit for selecting one pulse wave waveform.

  The detection accuracy of the pulse wave can be improved.

FIG. 1 is a block diagram illustrating a functional configuration of the pulse wave detection device according to the first embodiment. FIG. 2 is a diagram showing the three axes of pitch, roll, and yaw. FIG. 3 is a diagram illustrating an example of the relationship between time and sensor value. FIG. 4 is a diagram illustrating an example of the ROI. FIG. 5 is a diagram illustrating an example of a reference pulse wave waveform. FIG. 6 is a diagram illustrating an example of a face area estimation method. FIG. 7 is a diagram illustrating an example of the coupling. FIG. 8 is a diagram illustrating an example of the data management table. FIG. 9 is a diagram illustrating an example of the data management table. FIG. 10 is a diagram illustrating an example of a reference pulse wave waveform. FIG. 11 is a diagram illustrating an example of the data management table. FIG. 12 is a diagram illustrating an example of an input pulse wave waveform before interpolation. FIG. 13 is a diagram illustrating an example of an input pulse wave waveform after interpolation. FIG. 14 is a diagram showing an example of a pulse rate management table. FIG. 15 is a flowchart illustrating the procedure of the reference pulse wave generation process according to the first embodiment. FIG. 16 is a flowchart illustrating the procedure of the pulse wave detection process according to the first embodiment. FIG. 17 is a diagram illustrating an application example of ROI. FIG. 18 is a diagram illustrating an application example of ROI. FIG. 19 is a diagram illustrating an application example of ROI. FIG. 20 is a diagram illustrating an application example of ROI. FIG. 21 is a schematic diagram illustrating an example of a computer that executes a pulse wave detection program according to the first and second embodiments.

  Hereinafter, a pulse wave detection device, a pulse wave detection method, and a pulse wave detection program according to the present application will be described with reference to the accompanying drawings. Note that this embodiment does not limit the disclosed technology. Each embodiment can be appropriately combined within a range in which processing contents are not contradictory.

[Configuration of pulse wave detector]
First, the functional configuration of the pulse wave detection device according to the present embodiment will be described. FIG. 1 is a block diagram illustrating a functional configuration of the pulse wave detection device according to the first embodiment. The pulse wave detection device 10 shown in FIG. 1 uses a human body's pulse wave, that is, a human body pulse wave, without bringing a measuring instrument into contact with the human body under ordinary environmental light such as sunlight or room light. A pulse wave detection process for detecting a change in the volume of blood accompanying the pulsation of the heart is executed.

  Such a pulse wave detection device 10 can be implemented as one aspect by installing a pulse wave detection program in which the above-described pulse wave detection processing is provided as package software or online software in a desired computer. For example, the above pulse wave detection program is installed not only on mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System) but also on mobile terminal devices including tablet terminals and slate terminals. Accordingly, the mobile terminal device can function as the pulse wave detection device 10. Here, as an example, the following explanation will be given by exemplifying a case where the pulse wave detection device 10 is mounted as a mobile terminal device, but the pulse wave detection program is installed in a stationary terminal device such as a personal computer. Can also be installed.

  As shown in FIG. 1, the pulse wave detection device 10 includes a camera 11a, a gyro sensor 11b, an eye detection unit 12, a ROI (Region of Interest) setting unit 13, a reference waveform generation unit 14, and a reference waveform storage. Part 14a. Furthermore, the pulse wave detection device 10 includes an estimation unit 15, an ROI scanning unit 16, an input waveform generation unit 17, a similarity calculation unit 18, and a detection unit 19.

  The pulse wave detection device 10 may include various functional units included in a known computer in addition to the functional units illustrated in FIG. For example, when the pulse wave detection device 10 is implemented as a stationary terminal, it may further include an input / output device such as a keyboard, a mouse, and a display. Moreover, when the pulse wave detection apparatus 10 is mounted as a tablet terminal or a slate terminal, it may further include an acceleration sensor. When the pulse wave detection device 10 is mounted as a mobile communication terminal, the pulse wave detection device 10 further includes functional units such as an antenna, a wireless communication unit connected to the mobile communication network, and a GPS (Global Positioning System) receiver. It doesn't matter.

  The camera 11a is an imaging device on which an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) is mounted.

  As one embodiment, the camera 11a can be equipped with three or more light receiving elements such as R (red), G (green), and B (blue). As an implementation example of the camera 11a, a digital camera or a Web camera may be connected via an external terminal. As another implementation example, when a camera is mounted from the time of shipment, such as an in camera or an out camera, the camera can be used. Here, the case where the pulse wave detection device 10 is equipped with the camera 11a is illustrated, but the pulse wave detection device 10 does not necessarily have the camera 11a when an image can be acquired via a network or a storage device. It is not necessary.

  Here, in the following, it is assumed that a face is photographed as an example of a living body and an in-camera is used as an example of implementation of the camera 11a. In this way, when a face is photographed using the camera 11a, an image that can easily detect a pulse wave is displayed by the camera 11a through a display device (not shown) such as a touch panel display or a sound output from a sound output device (not shown). An image capturing operation can be guided so that an image is captured. For example, the above-described pulse wave detection program activates the camera 11a and causes the camera 11a to start photographing the subject accommodated in the photographing range of the camera 11a. At this time, the pulse wave detection program can also display the target position that reflects the user's nose of the pulse wave detection device 10 as an aim while displaying the image captured by the camera 11a on the display. As a result, it is possible to capture an image in which the user's nose is within the center of the imaging range among facial parts such as the user's eyes, ears, nose and mouth. And the camera 11a outputs the image by which the face was imaged to the function part of a back | latter stage.

  In addition, the above-described pulse wave detection program does not necessarily execute the above-described guidance, and can capture a user's face in the background of display by another program such as an operating system or an application program. For example, it is possible to photograph the face of a user who browses text, documents, still images, moving images, and the like displayed on the display. In this case, an image can be collected in daily life without making the user aware of shooting.

  The gyro sensor 11b is a sensor that detects an angular velocity, and is sometimes called a gyro or an angular velocity sensor.

  As an embodiment, a three-axis gyro sensor of pitch (X axis), roll (Y axis), and yaw (Z axis) can be adopted as the gyro sensor 11b. FIG. 2 is a diagram showing the three axes of pitch, roll, and yaw. As shown in FIG. 2, the gyro sensor 11b detects an angular velocity of rotation about the left and right sides of the screen, an angular velocity of rotation about the front and rear sides of the screen, and an angular velocity of rotation about the upper and lower sides of the screen. As an example, the unit of angular velocity measured by the gyro sensor 11b is “deg / s”, and the positive and negative signs of the angular velocity follow the right-handed system. Here, a case where a three-axis gyro sensor is employed has been illustrated, but a single-axis or two-axis gyro sensor may be employed.

  Here, the functional units other than the camera 11a and the gyro sensor 11b shown in FIG. 1 execute a first processing system that executes processing related to the reference pulse wave waveform and processing related to the input pulse wave waveform that is compared with the reference pulse wave waveform. The second processing system can be roughly divided.

  The above-mentioned “reference pulse waveform” refers to a pulse waveform used as a reference for the input pulse waveform. For example, the reference pulse wave waveform is generated from a series of images captured by the camera 11a when the pulse wave detection device 10 is in a stable state. On the other hand, the “input pulse wave waveform” refers to a pulse wave waveform to be compared with a reference pulse wave waveform. For example, the input pulse wave waveform is generated from a series of images input from the camera 11a. At this time, a plurality of input pulse wave waveforms are generated from the series of images. Although details will be described later, the position of the ROI used to generate the pulse wave waveform is different between at least one image included in the series of images among the input pulse wave waveforms.

  Hereinafter, each functional unit included in the first processing system will be described, and then each functional unit included in the second processing system will be described. Although FIG. 1 illustrates the case where both the first processing system and the second processing system are mounted on one apparatus, only one of the processing systems may be mounted.

  The first processing system includes an eye detection unit 12, an ROI setting unit 13, a reference waveform generation unit 14, a reference waveform storage unit 14a, and the like.

  The eye detection unit 12 is a processing unit that detects an eye from an image.

  As one embodiment, the eye detection unit 12 detects the coordinates of the center of the user's eye, for example, the pupil, by performing image processing such as template matching on the image captured by the camera 11a. At this time, the eye detection unit 12 can also perform eye detection after narrowing down the processing execution range to the face area by performing face detection on the image. Here, as an example, a case where eyes are detected is illustrated, but facial parts other than the eyes, for example, eyebrows, nose, mouth, and the like may be detected, or a face area may be detected. .

  The ROI setting unit 13 is a processing unit that sets an ROI, a so-called region of interest.

  As one embodiment, the ROI setting unit 13 determines whether or not the sensor value detected by the gyro sensor 11b is equal to or less than a threshold value. At this time, the ROI setting unit 13 can use the sensor values of all three axes of pitch, roll, and yaw for the determination, or can also use the sensor values of some axes for the determination. When using sensor values of a plurality of axes for determination, it is preferable to determine whether all sensor values are equal to or less than a threshold value using an AND condition. In addition, as an example of the threshold value used for the above determination, a value that can be regarded as a touch operation on the screen is set.

  When the sensor value is equal to or smaller than the threshold value, the ROI setting unit 13 further determines whether or not the period during which the sensor value equal to or smaller than the threshold value is detected continues for a predetermined period. Hereinafter, the period during which the sensor value below the threshold is detected may be referred to as a “stable period”. In addition, as an example of the predetermined period, a length that can include a cycle of one beat is set. Generally, the pulse rate is about 40 bpm (beat per minute) at the latest. One cycle of 40 bpm corresponds to 1.5 seconds. From this, as an example, the predetermined period can be set to 1.5 seconds. As a result, when the stable period lasts for 1.5 seconds, it can be assumed that the luminance change for one beat of the user is included in a series of images captured during the stable period. it can.

  Here, when the stable period continues for a predetermined period, the ROI setting unit 13 instructs the eye detection unit 12 to perform eye detection on a series of images captured during the stable period. As a result, when eye detection is performed on a series of images, the ROI setting unit 13 calculates the representative positions of the left and right pupils between the series of images. For example, the ROI setting unit 13 can obtain the average coordinates of the pupil obtained by averaging the coordinates of the pupil for each of the left and right eyes between a series of images as a position representing the pupil of each image. In addition to the average value, the coordinates of the mode value may be used as the representative position, or the median coordinates may be calculated as the representative position. Then, the ROI setting unit 13 sets an ROI to be applied to a series of images from the representative position of the pupil. For example, the ROI setting unit 13 calculates the distance L between the left and right pupils. In addition, as an example, the ROI setting unit 13 sets a rectangular region having the center position of both eyes as the center of gravity and the interval L between the pupils as the height and width as the ROI.

  FIG. 3 is a diagram illustrating an example of the relationship between time and sensor value. FIG. 4 is a diagram illustrating an example of the ROI. The vertical axis of the graph shown in FIG. 3 indicates the angular velocity amplitude, and the horizontal axis of the graph indicates time. In FIG. 3, the time change of the amplitude at the pitch angle is indicated by a solid line, the time change of the amplitude at the roll angle is indicated by a broken line, and the threshold value used for the determination is indicated by a dashed line.

  In the example shown in FIG. 3, in the section before time t1, even if the angular velocity of the pitch angle and roll angle temporarily falls below the threshold value, the pitch angle and roll until the predetermined period elapses. An angular velocity that exceeds the threshold for both angles is detected. In this case, there is a high possibility that the user performs a touch operation on the screen, or the user moves the pulse wave detection device 10 without looking at the screen. As a result, there is a high possibility that the face is not reflected in the image in the first place, or even if the face is reflected, the detection of the face or eyes fails due to shaking. Therefore, ROI setting is not executed in the section before time t1, and the generation of the reference pulse wave waveform is not executed.

  On the other hand, after the angular velocities of the pitch angle and the roll angle become equal to or less than the threshold at time t1, the angular velocities of both axes do not exceed the threshold for a predetermined period from time t1, that is, 1.5 seconds. Therefore, eye detection is performed on a series of images captured during the stable period from time t1 to time t2 when 1.5 seconds have elapsed from time t1 and time t2 has been reached. In response to this, the representative positions of the left and right pupils are calculated between a series of images. Thereafter, as shown in FIG. 4, the pupil interval L is calculated from the coordinates of the representative position IR of the right pupil and the representative position IL of the left pupil in the image 20, and the center position IC of both eyes is calculated. . In addition, a rectangular area having the center position IC of both eyes as the center of gravity and the distance L between the pupils as the height and width, and the bold rectangle shown in FIG. 4, is set as the ROI. By setting the ROI, it is possible to exclude a mouth or the like that easily includes body movement from the ROI. As a result, it can be expected to obtain a good reference pulse waveform from each ROI included in a series of images.

  The reference waveform generation unit 14 is a processing unit that generates a reference pulse wave waveform from the luminance change of the ROI set by the ROI setting unit 13.

  As one embodiment, the reference waveform generation unit 14 performs a predetermined statistical process on the pixel value of each pixel included in the ROI of the image for each image included in the series of images. For example, the reference waveform generation unit 14 averages the luminance value of the R component of each pixel in the ROI. In addition to the average, the median value and the mode value may be calculated. In addition to the average, an arbitrary average process such as a weighted average or a moving average may be executed. Moreover, it is good also as using the luminance value of G component or B component other than R component, and it does not matter as the luminance value of each wavelength component of RGB. Thereby, the luminance value of the R component representing the ROI is obtained for each image. Hereinafter, the luminance value of the R component representing ROI may be referred to as “representative luminance value of R component”. At this time, the time series data of the representative luminance value of the R component can be used as the reference pulse wave waveform. However, when the frame frequency of the camera 11a is low, the number of sampling points is small and the shape of the pulse wave waveform cannot be reproduced. Cases are also possible. Therefore, for example, the reference waveform generation unit 14 can also perform resampling by executing spline interpolation or the like on the time series data of the representative luminance value of the R component. A reference pulse wave waveform is generated by such resampling.

  FIG. 5 is a diagram illustrating an example of a reference pulse wave waveform. The vertical axis of the graph shown in FIG. 5 indicates amplitude, and the horizontal axis of the graph indicates time. The points plotted on the graph shown in FIG. 5 indicate the representative luminance values of the R component obtained from a series of images. As shown in FIG. 5, the representative luminance value of the R component obtained according to the frame frequency of the camera 11a is interpolated as a sinusoidal signal waveform having a predetermined sampling frequency by cubic spline interpolation as an example. After the resampled reference pulse wave signal is output, the reference pulse wave waveform is stored in the reference waveform storage unit 14a. For example, the representative luminance value of the R component after interpolation obtained by spline interpolation is registered in the reference waveform storage unit 14a at each interpolation time resampled by spline interpolation.

  Next, the second processing system will be described. The second processing system includes an estimation unit 15, an ROI scanning unit 16, an input waveform generation unit 17, a similarity calculation unit 18, a detection unit 19, and the like.

  The estimation unit 15 is a processing unit that estimates a face area using the gyro sensor 11b.

  As an embodiment, the estimation unit 15 estimates a face area after movement that moves between frames of a series of images each time an image is captured by the camera 11a for a predetermined period, for example, 1.5 seconds. For example, the estimation unit 15 estimates the moving direction of the face area from the positive and negative values of the pitch angle, roll angle, and yaw angle sensor values output by the gyro sensor 11b, and moves the face area from the magnitude of each sensor value. Estimate the amount. At this time, the estimation unit 15 can estimate the moving direction and the moving amount from the position of the face area detected when the reference pulse wave waveform is generated.

  FIG. 6 is a diagram illustrating an example of a face area estimation method. As shown in FIG. 6, when a user views an image displayed on the screen, the face and the screen surface are usually faced to each other. For example, when the reference pulse wave waveform is generated, that is, when the pulse wave detection device 10 is in a stable state, it can be estimated that the above-described normal posture is made. With this as a reference, the moving direction of the face region is estimated from the positive and negative values of the sensor values of the pitch angle and the roll angle. FIG. 6 illustrates the case where the sensor values of the pitch angle and the roll angle are used for estimating the movement direction of the face area, but the sensor value of the yaw angle can also be used.

  For example, when the angular velocity of the pitch angle is a positive value, that is, in the case of the + p angle shown in the figure, it can be seen that the user is browsing in a state where the screen is tilted away from the front. In this case, it can be estimated that the user's face is projected downward from the position of the face when captured by the in-camera at the time of facing. Further, when the angular velocity of the pitch angle is a negative value, that is, in the case of the -p angle shown in the figure, it can be seen that the user is browsing with the screen tilted toward the front than when facing the front. . In this case, it can be estimated that the face of the user is reflected upward from the position of the face when captured by the in-camera at the time of facing.

  Further, when the angular velocity of the roll angle is measured as a positive value, that is, in the case of the + r angle shown in the figure, it can be seen that the user is browsing with the screen tilted to the right than when facing the front. In this case, it can be estimated that the face of the user is reflected in the right direction from the position of the face when captured by the in-camera at the time of facing. In addition, when the negative angular velocity of the roll angle is measured, that is, in the case of the illustrated -r angle, it can be seen that the user is browsing with the screen tilted to the left rather than at the front. . In this case, it can be estimated that the user's face is reflected in the left direction from the position of the face when captured by the in-camera at the time of facing.

  From these correspondences, the movement direction of the face region can be estimated from the positive and negative values of the sensor values of the pitch angle and the roll angle, and the movement amount can be estimated by the amount of each sensor value.

  The ROI scanning unit 16 is a processing unit that scans the ROI.

  As one embodiment, the ROI scanning unit 16 moves the ROI within the face area for each face area estimated from the series of images by the estimation unit 15. At this time, as an example of the ROI, the ROI scanning unit 16 uses the ROI set by the ROI setting unit 13, that is, a rectangular region having the height and width of the pupil interval L, and follows a method such as raster scanning. The ROI can be moved within the face area.

  The input waveform generation unit 17 is a processing unit that generates an input pulse wave waveform.

As one embodiment, the input waveform generation unit 17 couples the ROI scanned for each face area by the ROI scanning unit 16 between a series of images. For example, when the number of the series of images is n and the number of ROIs scanned in the face area of one image included in the series of images is S, when coupling the ROI over all combinations, S ⁿ combinations are obtained. In this way, ROIs can be coupled across all combinations, but by coupling ROIs having the same relative position between the face regions included in a series of images, the number of couplings is S. Can also be reduced.

  FIG. 7 is a diagram illustrating an example of the coupling. FIG. 7 shows three extracted images included in the series of images. Reference numerals 30, 31, and 32 shown in FIG. 7 are all images included in a series of images. Of the images 30 to 32 shown in FIG. 7, the image 30 is taken at time t, the image 31 is taken at time t + 1, and the image 32 is taken at time t + 2. It was taken. Also, reference numerals 30A, 31A, and 32A illustrated in FIG. 7 indicate the estimation results of the face area estimated by the estimation unit 15. Further, the fill shown in FIG. 7 indicates an ROI having the same relative position between the face regions 30A to 32A. As shown in FIG. 7, when coupling ROIs scanned in the face areas 30 </ b> A to 32 </ b> A, ROIs having the same relative position between the face areas 30 </ b> A to 32 </ b> A can be coupled. As a result, the amount of calculation can be reduced in the subsequent processing as compared with the case of coupling the ROI over all combinations.

  Thereafter, every time the ROI is coupled, the input waveform generation unit 17 performs a predetermined statistical process on the pixel value of each pixel included in the ROI for each ROI included in the coupling combination. For example, the input waveform generation unit 17 averages the luminance value of the R component of each pixel in the ROI. In addition to the average, the median value and the mode value may be calculated. In addition to the average, an arbitrary average process such as a weighted average or a moving average may be executed. Moreover, it is good also as using the luminance value of G component or B component other than R component, and it does not matter as the luminance value of each wavelength component of RGB. As a result, the “representative luminance value of the R component” is obtained for each image. At this time, the time series data of the representative luminance value of the R component can be used as the input pulse wave waveform. However, as with the reference pulse wave waveform, spline interpolation or the like is applied to the time series data of the representative luminance value of the R component. It can also be resampled by executing. By this resampling, an input pulse wave waveform is generated.

  As described above, the input waveform generation unit 17 generates S input pulse wave waveforms by coupling ROIs having the same relative position between the face regions included in the series of images.

  The similarity calculation unit 18 is a processing unit that calculates the similarity between the reference pulse wave waveform and the input pulse wave waveform.

  As an embodiment, the similarity calculation unit 18 calculates a correlation coefficient between the reference pulse waveform and the input pulse waveform as an example of the similarity. For example, the correlation coefficient can be calculated according to the following equation (1). In the following equation (1), “x” indicates the representative luminance value of the R component of the input pulse wave waveform, and “y” indicates the representative luminance value of the R component of the reference pulse wave waveform. Further, “N” in the following equation (1) indicates the number of data of the input pulse waveform and the reference pulse waveform, that is, the number of sampling. Moreover, the superscript bar attached | subjected to "x" and "y" in following formula (1) points out the average value. As described above, the similarity calculation unit 18 calculates each similarity between the reference pulse waveform and the S input pulse waveforms. As a result, the similarity is obtained for each of the S input pulse wave waveforms.

  The detection unit 19 is a processing unit that detects a pulse wave from the input pulse wave waveform.

  As one embodiment, the detection unit 19 selects an input pulse wave waveform having the highest similarity calculated by the similarity calculation unit 18 among the input pulse wave waveforms generated by the input waveform generation unit 17. In addition, the detection unit 19 can output the input pulse wave waveform previously selected as one aspect of the detection result of the pulse wave, or outputs the pulse rate obtained from the input pulse wave waveform. You can also For example, as an example of a method for calculating the pulse rate, every time the amplitude value of the input pulse wave waveform is output, the peak detection of the input pulse wave waveform, for example, the detection of the zero cross point of the differential waveform is executed. At this time, when the peak of the input pulse wave waveform is detected by the peak detection, the detection unit 19 stores the sampling time in which the peak, that is, the maximum point is detected, in an internal memory (not shown). In addition, when the peak appears, the detection unit 19 can detect the pulse rate by obtaining a time difference from the local maximum point n times the predetermined parameter and dividing it by n. Here, the case where the pulse rate is detected by the peak interval is exemplified, but the peak is detected in the frequency band corresponding to the pulse wave by converting the input pulse wave waveform to the frequency component, for example, the frequency band of 40 bpm to 180 bpm. The pulse rate can also be calculated from the frequency taken.

  The pulse rate and pulse wave waveform obtained in this way can be output to an arbitrary output destination including a display device (not shown) included in the pulse wave detection device 10. For example, in the case where a diagnostic program for diagnosing the function of the autonomic nerve from fluctuations in pulse rate or pulse period or diagnosing a heart disease or the like from the pulse wave waveform is installed in the pulse wave detection device 10, the diagnostic program Can be the output destination. In addition, a server device that provides a diagnostic program as a Web service can be used as an output destination. Further, a terminal device used by a person concerned of the user who uses the pulse wave detection device 10, for example, a caregiver or a doctor, can be used as the output destination. This also enables monitoring services outside the hospital, for example, at home or at home. Needless to say, the measurement result and diagnosis result of the diagnostic program can also be displayed on the terminal devices of the persons concerned including the pulse wave detection device 10.

  The eye detection unit 12, the ROI setting unit 13, the reference waveform generation unit 14, the estimation unit 15, the ROI scanning unit 16, the input waveform generation unit 17, the similarity calculation unit 18, and the detection unit 19 are configured by a CPU (Central Processing). This can be realized by causing a pulse wave detection program to be executed by a unit) or MPU (micro processing unit). Each functional unit described above can also be realized by a hard wired logic such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

  Further, a semiconductor memory element or a storage device can be adopted as the reference waveform storage unit 14a or the internal memory. For example, examples of the semiconductor memory element include a flash memory, a dynamic random access memory (DRAM), and a static random access memory (SRAM). Further, examples of the storage device include storage devices such as a hard disk and an optical disk.

[Concrete example]
Subsequently, a specific example of pulse wave detection will be described with reference to FIGS. 8, FIG. 9 and FIG. 11 are diagrams showing an example of the data management table. FIG. 10 is a diagram illustrating an example of a reference pulse wave waveform. FIG. 12 is a diagram illustrating an example of an input pulse wave waveform before interpolation. FIG. 13 is a diagram illustrating an example of an input pulse wave waveform after interpolation. FIG. 14 is a diagram showing an example of a pulse rate management table. 8 to 14 exemplify the case where the frame rate of the camera 11a is set to 500 ms, the predetermined period is set to 1.5 seconds, and the generation of the reference pulse wave waveform and the detection of the pulse rate are executed for convenience of explanation. However, each parameter can be changed arbitrarily.

  Before explaining the processing contents of pulse wave detection, items included in the data management table will be explained. The data management tables shown in FIGS. 8, 9, and 11 are associated with items such as time, image ID, gyro sensor value, state change flag, pupil interval, ROI, average R value inside ROI, and face area. A table is adopted. In addition, although the data management table | surface of one user is illustrated here, a data management table | surface can also be produced for every user of the pulse-wave detection apparatus 10. FIG.

  As an example of the “time”, “hh: mm: ss” can be adopted. However, in the case of performing daily management, a date can also be added. Further, the time may be a local elapsed time measured by the pulse wave detection device 10. The “time” interval follows the frame rate of the camera 11a. “Image ID” indicates identification information for identifying an image. Here, it is assumed that each time an image is captured by the camera 11a, a sequence number, a so-called serial number, is assigned, but it is sufficient that each image can be identified, and any ID numbering form can be adopted. .

  The “gyro sensor value” refers to a sensor value measured by the gyro sensor 11b. For example, the angular velocity (deg) of the pitch angle and the roll angle is described as “(rol, pit)”. Further, the “state change flag” is a flag indicating whether or not the state of the pulse wave detection device 10 has changed. Here, as an example, when the sensor value of the gyro sensor 11b is less than a predetermined threshold, for example, (10, 10), “1” is set, while when it is equal to or greater than (10, 10). It is assumed that “0” is set.

  The “pupil interval” refers to the interval between the left and right pupils detected from the image captured by the camera 11a, and is represented by the number of pixels, for example. “ROI” refers to a region of interest used for pulse wave detection in an image. As described above, a rectangular region having the center position of both eyes as the center of gravity and the distance L between the pupils as the height and width. Is set. Here, as an example, a case where the ROI is defined by the coordinates of the upper left vertex of the ROI, the height of the ROI, and the width of the ROI is illustrated. The “average R value inside ROI” refers to the representative luminance value of the R component of each pixel inside the ROI. Further, the “face area” is an area estimated as the user's face in the image.

  Each time an image is captured by the camera 11a in a situation where such a data management table is prepared, the time when the image is captured, the image ID numbered for the image, and the gyro sensor value are displayed in the data management table. To be recorded. Then, the state change flag is set by comparing the gyro sensor value at each time with the threshold (10, 10). In the example shown in FIG. 8, the record of the image IDs “2”, “6”, and “8” in which the absolute value of the angular velocity of either the roll angle or the pitch angle measures 10 deg / sec or more has a state change While “0” is set in the flag, “1” is set in the state change flag for records of other image IDs. Note that the pupil interval calculated from each image by the ROI setting unit 13 is stored as it is in the pupil interval.

  When three state change flags “1” continue, that is, when the stable period continues for 1.5 seconds, the reference pulse wave waveform is targeted for three records where the state change flag “1” continues. Generation is started. In the example shown in FIG. 8, the pupil distance L is calculated from the average coordinates of the pupils obtained by averaging the coordinates of the pupils for the left and right eyes between a series of images with the image IDs “3” to “5”. The ROI is determined based on the center coordinates of both eyes. That is, the coordinates (50, 200) of the upper left vertex of the ROI, the ROI height “100”, and the ROI width “100” are set as the ROI. Thereafter, the representative luminance values “200”, “210”, and “205” of the R component are recorded for each ROI included in each of the series of images with the image IDs “3” to “5”, and the eye detection unit 12 The coordinates of the upper left vertex of the face area detected from each image and the height and width of the face area are recorded.

  Subsequently, spline interpolation is performed on the time-series data of the R component representative luminance values in the image IDs “3” to “5”. As a result, re-sampling is performed to time-series data of representative luminance values of the R component whose sampling interval is 0.02 seconds. The time series data of the representative luminance value of the R component resampled in this manner is recorded as the reference pulse wave waveform shown in FIG.

  Then, as shown in FIG. 10, after the reference pulse wave waveform is generated in the section of image IDs “3” to “5”, that is, the section of α, the section after image ID “6”, that is, the section of β. Processing for detecting a pulse wave is started. In the section of β, the height and width of the ROI are entered from the pupil interval obtained in the section of α, that is, the section of the image IDs “3” to “5” where the generation of the reference pulse wave waveform is executed.

  Subsequently, as shown in FIG. 11, the face area is estimated from the gyro sensor value in the section of β, for example, the section of the image IDs “6” to “8”. That is, the moving direction is estimated from the sign of the gyro sensor values of the image IDs “6” to “8”, and the moving amount from the previous frame is estimated based on the gyro sensor value. For example, in the example of the image ID “6”, the sign of the angular velocity of the roll angle is “positive”, while the sign of the angular velocity of the pitch angle is “negative”. Can be estimated as moving in the upper right direction with respect to the face area in the image ID “5”. Further, for example, if there is a change of 10 in the gyro sensor value, the face area is moved by 50 pixels (gyro sensor value: pixel = 1: 5). In this case, 100 (= 20 × 5) is added to the value of x with respect to the coordinates (160,280) of the upper left vertex of the face area in the image ID “5”, while −100 (−20 × 5) is subtracted from the value of y. As a result, the coordinates (260, 180) of the upper left vertex of the face area in the image ID “6” are calculated. For the height and width of the face movement, an average value acquired in the section α, that is, the section in which the pulse wave detection device 10 is in a stable state is used. The face area in the section of the image IDs “6” to “8” estimated in this way is recorded in the data management table.

  Subsequently, the ROI is moved within the face area for each face area in the section of the image IDs “6” to “8”. For example, the ROI is scanned every two pixels, and the representative luminance value of the R component is calculated from the ROI. Then, ROIs having the same relative position are coupled between the face areas included in the sections of the image IDs “6” to “8”. As a result, as shown in FIG. 12, time series data of representative luminance values of S R components is obtained. This is recorded as the input pulse wave waveform before interpolation. Thereafter, spline interpolation is performed on the input pulse wave waveform before interpolation. As a result, as shown in FIG. 13, the input pulse wave waveform after interpolation in which the sampling interval is interpolated to 0.02 seconds is recorded. When the pulse rate calculation is executed for each input pulse wave waveform of the input pulse wave waveform after interpolation, that is, all the pulse wave IDs of pulse wave IDs “01” to “0S” shown in FIG. As shown, the pulse rate for each of the pulse wave IDs “01” to “0S” is recorded in the pulse rate management table together with the correlation coefficient calculated with the reference pulse wave waveform. Thereafter, a pulse rate having a pulse wave ID having the highest correlation coefficient is selected from the pulse rates recorded in the pulse rate management table shown in FIG.

  Thus, ROI position correction can be realized by selecting the input pulse wave waveform having the highest correlation coefficient from a plurality of input pulse wave waveforms. For this reason, since the position correction of ROI can be realized using a plurality of input pulse wave waveforms regardless of the movement of the image and the erroneous detection of the eyes, the skin region can be stably acquired from the temporally continuous face image, The S / N ratio is improved when converting to an input pulse wave waveform. Therefore, the pulse wave detection accuracy can be improved. In addition, the face movement destination is estimated regardless of the image movement and eye misdetection caused by the operation of the pulse wave detection device 10, and the ROI is scanned by narrowing down the estimated face movement destination, thereby shortening the search time. it can.

[Process flow]
Subsequently, the flow of processing of the pulse wave detection device according to the present embodiment will be described. Here, (1) the reference waveform generation process executed by the pulse wave detection device 10 will be described, and then (2) the pulse wave detection process will be described.

(1) Reference Waveform Generation Processing FIG. 15 is a flowchart illustrating a procedure of reference pulse wave generation processing according to the first embodiment. This process can be executed when the pulse wave detection program is in an active state, and can also be executed when the pulse wave detection program is operating in the background. This process can also be executed in parallel with the pulse wave detection process described later.

  As shown in FIG. 15, the image captured by the camera 11a is acquired, and the angular velocity measured by the gyro sensor 11b when the image is captured is acquired (step S101 and step S102).

  Then, the ROI setting unit 13 determines whether or not the sensor value acquired in step S102 is equal to or less than a threshold value (step S103). If the sensor value is equal to or smaller than the threshold value (step S103 Yes), the ROI setting unit 13 further determines whether or not a period during which the sensor value equal to or smaller than the threshold value is detected, that is, a stable period has continued for a predetermined period (step). S104).

  Here, when the stable period continues for a predetermined period (step S104 Yes), the eye detection unit 12 performs eye detection on a series of images captured during the stable period (step S105). Thereafter, the ROI setting unit 13 calculates the representative position of the pupil for each of the left and right eyes between a series of images using the coordinates of the pupil obtained for each image in step S105, and the representative positions of the pupils of the left and right eyes The ROI to be applied to a series of images is set (step S106).

  Thereafter, the reference waveform generation unit 14 generates a reference pulse wave waveform by calculating a representative luminance value of the R component within the ROI included in the image for each image included in the series of images (step S107). Then, the reference pulse wave waveform is registered in the reference waveform storage unit 14a, and the process ends.

(2) Pulse Wave Detection Process FIG. 16 is a flowchart illustrating the procedure of the pulse wave detection process according to the first embodiment. This process can be executed when the pulse wave detection program is in an active state, and can also be executed when the pulse wave detection program is operating in the background. This process can be executed in parallel with the above-described reference pulse wave generation process.

  As shown in FIG. 16, an image captured by the camera 11a is acquired, and an angular velocity measured by the gyro sensor 11b when the image is captured is acquired (step S301 and step S302).

  When a predetermined period, for example, 1.5 seconds has elapsed since the last pulse wave detection (Yes at Step S303), the estimation unit 15 uses the sensor value measured over the predetermined period in Step S302. Thus, the face area of the image is estimated for each image included in the series of images (step S304).

  Subsequently, the ROI scanning unit 16 scans the ROI within the face area for each face area estimated from the series of images in step S304 (step S305). Thereafter, the input waveform generation unit 17 couples the ROI scanned in step S305, that is, the ROI having the same relative position (step S306). Then, the input waveform generation unit 17 generates an input pulse wave waveform by calculating a representative luminance value of the R component inside the ROI for each ROI included in the combination coupled in step S306 (step S307). .

  Thereafter, until the input pulse wave waveform is generated for all ROI combinations, for example, S combinations (No in step S308), the processes in steps S305 to S307 are repeated.

  When the input pulse waveform is generated for all combinations of ROIs (step S308 Yes), the similarity calculation unit 18 generates the reference pulse waveform stored in the reference waveform storage unit 14a and the input waveform generation unit 17. The degree of similarity is calculated with the input pulse wave waveform thus obtained (step S309).

  Thereafter, until the similarity is calculated for all input pulse wave waveforms (No in step S310), the process of step S309 is repeatedly executed. When the similarity is calculated for all the input pulse wave waveforms (Yes in step S310), the detection unit 19 selects the input pulse wave waveform having the highest similarity (step S311). Thereafter, the detection unit 19 detects the pulse rate from the input pulse wave waveform selected in step S311 (step S312).

  And it returns to said step S301 and repeats the process from step S301 to step S312.

[Effect of Example 1]
As described above, the pulse wave detection device 10 according to the present embodiment generates a plurality of input pulse wave waveforms from an image input from the camera 11a, and is generated when the terminal is stable among the plurality of input pulse wave waveforms. The input pulse waveform having the highest similarity with the reference pulse waveform is selected. Therefore, according to the pulse wave detection device 10 according to the present embodiment, it is possible to improve the detection accuracy of the pulse wave.

  Although the embodiments related to the disclosed apparatus have been described above, the present invention may be implemented in various different forms other than the above-described embodiments. Therefore, another embodiment included in the present invention will be described below.

[input signal]
In the first embodiment, an example in which an RGB signal is used as an input signal is illustrated. However, any type of signal and any number of signals may be used as an input signal as long as the signals have a plurality of different light wavelength components. it can. For example, an HSV signal may be used, two signals such as IR and NIR may be used, or three or more signals may be used.

[Application example of ROI]
In the first embodiment, a rectangular region having the center position of both eyes as the center of gravity and the distance L between the pupils as the height and width is exemplified as the ROI. However, an arbitrary shape and size are set for the ROI. be able to. 17 to 20 are diagrams showing application examples of ROI. For example, as shown in FIG. 17, the entire head from the jaw to the top of the head can be set as the ROI, and as shown in FIG. 18, a predetermined range above the eyebrows, for example, a range that is difficult to be applied to the bangs, etc. Can be set as ROI. Further, as shown in FIG. 19, the range defined by the nose to the eyebrows can be set as the ROI, and the range defined by the width of the nose can be set as the ROI. As shown in FIG. It can also be set as an ROI.

[Update of reference pulse waveform]
The reference pulse wave waveform is not fixed data and can be updated at an arbitrary timing. As an example, the latest reference pulse wave waveform can be updated by the processing shown in FIG. As another example, the input pulse wave waveform having the highest similarity in the process shown in FIG. 16 can be updated as the reference pulse wave waveform. In this case, the reference pulse wave waveform can be updated only when the similarity is equal to or higher than a predetermined threshold, for example, when the correlation coefficient is equal to or higher than 0.9.

[Other implementation examples]
In the first embodiment, the case where the pulse wave detection device 10 executes the above-described pulse wave detection processing in a stand-alone manner is illustrated, but it can also be implemented as a client server system. For example, the pulse wave detection device 10 may be implemented as a Web server that provides a pulse wave detection service, or may be implemented as a cloud that provides a pulse wave detection service by outsourcing. As described above, when the pulse wave detection device 10 operates as a server device, a mobile terminal device such as a smartphone or a mobile phone or an information processing device such as a personal computer can be accommodated as a client terminal. When an image showing the face of a subject is acquired from these client terminals via a network, a pulse wave detection process is executed, and a heartbeat detection result or a diagnosis result using the detection result is returned to the client terminal Can provide a pulse wave detection service.

[Pulse wave detection program]
The various processes described in the above embodiments can be realized by executing a prepared program on a computer such as a personal computer or a workstation. Therefore, in the following, an example of a computer that executes a pulse wave detection program having the same function as in the above embodiment will be described with reference to FIG.

  FIG. 21 is a schematic diagram illustrating an example of a computer that executes a pulse wave detection program according to the first and second embodiments. As illustrated in FIG. 21, the computer 100 includes an operation unit 110a, a speaker 110b, a camera 110c, a display 120, and a communication unit 130. Further, the computer 100 includes a CPU 150, a ROM 160, an HDD 170, and a RAM 180. These units 110 to 180 are connected via a bus 140.

  As shown in FIG. 21, the HDD 170 includes an eye detection unit 12, an ROI setting unit 13, a reference waveform generation unit 14, an estimation unit 15, an ROI scanning unit 16, an input waveform generation unit 17, and the like described in the first embodiment. A pulse wave detection program 170a that exhibits the same functions as the similarity calculation unit 18 and the detection unit 19 is stored in advance. For this pulse wave detection program 170a, each eye detection unit 12, ROI setting unit 13, reference waveform generation unit 14, estimation unit 15, ROI scanning unit 16, input waveform generation unit 17, and similarity calculation shown in FIG. Similarly to the constituent elements of the unit 18 and the detection unit 19, they may be appropriately integrated or separated. In other words, all data stored in the HDD 170 need not always be stored in the HDD 170, and only data necessary for processing may be stored in the HDD 170.

  Then, the CPU 150 reads the pulse wave detection program 170 a from the HDD 170 and develops it in the RAM 180. Accordingly, as shown in FIG. 21, the pulse wave detection program 170a functions as a pulse wave detection process 180a. The pulse wave detection process 180a expands various data read from the HDD 170 in an area allocated to itself on the RAM 180 as appropriate, and executes various processes based on the expanded various data. The pulse wave detection process 180a includes the eye detection unit 12, the ROI setting unit 13, the reference waveform generation unit 14, the estimation unit 15, the ROI scanning unit 16, the input waveform generation unit 17, and the similarity calculation unit 18 illustrated in FIG. And the process performed in the detection part 19, for example, the process shown in FIG.15 and FIG.16 is included. In addition, each processing unit virtually realized on the CPU 150 does not always require that all processing units operate on the CPU 150, and only a processing unit necessary for the processing needs to be virtually realized.

  Note that the pulse wave detection program 170a is not necessarily stored in the HDD 170 or the ROM 160 from the beginning. For example, each program is stored in a “portable physical medium” such as a flexible disk inserted into the computer 100, so-called FD, CD-ROM, DVD disk, magneto-optical disk, or IC card. Then, the computer 100 may acquire and execute each program from these portable physical media. In addition, each program is stored in another computer or server device connected to the computer 100 via a public line, the Internet, a LAN, a WAN, etc., and the computer 100 acquires and executes each program from these. It may be.

DESCRIPTION OF SYMBOLS 10 Pulse wave detection apparatus 11a Camera 11b Gyro sensor 12 Eye detection part 13 ROI setting part 14 Reference waveform generation part 14a Reference waveform memory | storage part 15 Estimation part 16 ROI scanning part 17 Input waveform generation part 18 Similarity calculation part 19 Detection part

Claims (5)

An acquisition unit for acquiring an image captured by the camera;
A scanning unit that scans a region of interest used for detecting a pulse wave in each of the plurality of acquired images;
A generator that generates a plurality of first pulse wave waveforms from pixel values of a region of interest included in the combination for each combination in which the region of interest scanned for each image is combined between the plurality of images;
A calculation unit for calculating a similarity between a plurality of generated first pulse wave waveforms and a second pulse wave waveform set as a reference;
A pulse wave detection device, comprising: a selection unit that selects the first pulse wave waveform having the highest degree of similarity with the second pulse wave waveform. A movement amount acquisition unit for acquiring a movement amount of the pulse wave detection device;
An extraction unit for extracting a biological region from the image;
Using the position of the living body region from the image in which the movement amount acquired by the movement amount acquisition unit is equal to or less than a predetermined threshold, and the movement amount acquired from each image when the plurality of images are captured, An estimation unit that estimates a living body region included in the image for each image;
The pulse wave detection device according to claim 1, wherein the scanning unit scans the region of interest within a living body region estimated by the estimation unit.   The second pulse wave waveform is generated from a plurality of images captured in a period when a period in which the movement amount acquired by the movement amount acquisition unit is a predetermined threshold value or less continues for a predetermined period. The pulse wave detection device according to claim 2. Computer
Get the image taken by the camera,
Scan the region of interest used for pulse wave detection in the acquired images for each of the plurality of images,
A plurality of first pulse wave waveforms are generated from pixel values of regions of interest included in the combination for each combination in which the region of interest scanned for each image is combined between the plurality of images,
A similarity is calculated between a plurality of generated first pulse wave waveforms and a second pulse wave waveform set as a reference,
A pulse wave detection method, comprising: performing a process of selecting a first pulse wave waveform having the highest degree of similarity with the second pulse wave waveform. On the computer,
Get the image taken by the camera,
Scan the region of interest used for pulse wave detection in the acquired images for each of the plurality of images,
A plurality of first pulse wave waveforms are generated from pixel values of regions of interest included in the combination for each combination in which the region of interest scanned for each image is combined between the plurality of images,
A similarity is calculated between a plurality of generated first pulse wave waveforms and a second pulse wave waveform set as a reference,
A pulse wave detection program for executing a process of selecting a first pulse wave waveform having the highest similarity with the second pulse wave waveform.

Images