JP2007088803A - Information processing device - Google Patents

Abstract

【Task】
When identifying a scene that the user is interested in from TV program scenes such as terrestrial broadcasts and satellite broadcasts, the scene that the user is interested in is based only on sensitivity information related to facial expressions such as eyes, pupils, and blinks. There is a problem that it is insufficient to extract.
[Solution]
In the media processing device of the present invention, user detection is performed based on the image information acquired by the image acquisition unit, and the face position of the user is estimated. The voice section is detected from the sound data recorded by the microphone array. When the sound source direction and the face position in the detected voice section coincide with each other, the utterance target is regarded as a media processing device. Then, it is determined that the utterance of the voice section is uttered because the user is interested in the video displayed on the media processing device, and the video is interested in the video section at the same time as the voice section. Judged as a section.
[Selection] Figure 2

Description

The present invention belongs to the field of media processing technology for identifying a scene in which a user is particularly interested from video information such as a television program.

2. Description of the Related Art Conventionally, as a technique for acquiring human sensitivity information, there is a technique for extracting sensitivity information related to facial expressions such as a line of sight, pupils, and blinks from a subject's video (for example, see Patent Document 1). In the present technology, the interest in the presented video is grasped from the sensibility information, and the video that the user is interested in is retrieved and presented based on the video determined to be interested.
In addition, there is a technology that includes a direction detection unit that detects a sound source direction of an input sound and a human detection unit that extracts a person from an input image, and determines that a speech is made by a speaker when a person is extracted from the sound source. (For example, Patent Document 2).

Japanese Patent Application Laid-Open No. 11-282721

JP 2003-189273 A

  Search for TV programs that the user may be interested in, infer what type of TV programs a particular person is interested in, and automatically select the TV programs that the person is interested in In realizing the system to be selected, in the above-described conventional technology, the user's interest is determined using a line of sight or the like. However, there are cases where information such as the line of sight alone is insufficient for grasping the true intention of the user.

  In this regard, it is considered that humans often scream and laugh when they are interested in the contents of TV programs. However, in the above-described conventional technology, a determination in consideration of such human characteristics is not made. Therefore, in the present application, a program presentation device is disclosed in which the system can discriminate a TV program that is interesting enough for a user to laugh out loud, a TV program that is interesting that a human will speak out and be drawn surprisingly, etc. With the goal.

  In the media processing device of the present invention, user detection is performed based on the image information acquired by the image acquisition unit, and the face position of the user is estimated. The voice section is detected from the sound data recorded by the microphone array. When the sound source direction and the face position in the detected voice section coincide with each other, the utterance target is regarded as a media processing device. Then, it is determined that the utterance of the voice section is uttered because the user is interested in the video displayed on the media processing device, and the video in which the user is interested in the video section of the same time as the voice section Judged as a section.

  According to the configuration of the present invention, it is possible to identify a scene in which the user faces a media processing device such as a television and is speaking out or laughing, that is, a scene in which the user is interested. it can.

  Hereinafter, typical embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a basic configuration diagram of the present invention. The media processing device 1 holds a camera 2 and a microphone array 10 in a casing. The media processing device 1 has a display device that presents content such as a television program to the user.

  In this embodiment, a microphone array is used, but a microphone or the like may be used as long as it is a sound acquisition device that acquires sound. The microphone array can obtain information on the direction of arrival of sound that cannot be obtained with a single microphone, and can improve the performance of the speech object determination unit 7 of the present invention.

  FIG. 2 is a block diagram of this embodiment. In FIG. 1, these processing units are integrated with the display unit, but the processing unit and the display unit are separate from each other, and may be separately connected by wire or wirelessly. The image captured by the camera 2 is sent to the image acquisition unit 4 and converted into digital image data. The image data converted into digital image data by the image acquisition unit 4 is sent to the user detection unit 5. The user detection unit 5 detects a face facing the front of the user from the transmitted image data using a face image recognition technique or a line-of-sight recognition technique. As the face image recognition technique and the line-of-sight recognition technique to be used, known techniques can be adopted. The face position estimation unit 11 connected to the user detection unit 5 estimates the position of the user's face in real space from the face image recognition result of the user detection unit 5. The microphone array 10 captures a plurality of channels of audio signals. A plurality of channels of audio signals captured by the microphone array 10 are sent to the audio acquisition unit 6 and converted into digital data of a plurality of channels.

  Next, the digital data is sent to the voice detection unit 12. The voice detection unit 12 performs voice segment detection processing based on voice power on one time-series data among time-series data of digital data of a plurality of channels, and detects a voice segment. In addition to the voice, noise such as human footsteps is mixed in the signal entering the microphone. While voice has the property that several s of sounds continue, noise often breaks in a relatively short time. By performing speech segment detection, it is possible to remove such noise that is interrupted in a short time and extract only speech in which several s of sounds are continuous. The voice section detection processing is performed from a frame in which the voice frame power changes more than a predetermined threshold (voice start end) to a frame in which the voice frame power changes to a value less than the predetermined threshold (voice end). Different threshold values are used for the threshold value used for detecting the voice start end and the threshold value used for detecting the voice end of single threshold voice segment detection processing or single threshold voice segment detection cut out as a voice segment 2 A stage threshold speech section detection process is used. Since voice has a higher power at the end than at the start of the voice section, it is possible to prevent the start from being missing from the detected voice section by reducing the threshold of the power at the end from the start. In addition, in the single threshold voice segment detection process and the two-stage threshold voice segment detection process, a paused voice segment detection process that allows a pause of a predetermined length or less to be included in the voice segment is also applicable. is there. The voice does not always have a certain level of power in one voice section, and there may be a pause of several hundred ms, but in such cases by performing a paused voice section detection process Even so, it is possible to appropriately detect the speech section.

  The sound source position detection unit 13 estimates the sound source direction of the most prevalent sound source in the voice section detected by the voice detection unit 12. A known technique can be used for the method of estimating the sound source direction. The utterance target determination unit 7 determines whether or not the sound source direction output from the sound source position detection unit 13 matches the user's face position output from the face position estimation unit 11. At this time, the position of the user's face and the sound source direction do not necessarily exactly match each other, and if the allowable error is set in advance and falls within the range, it is determined that the sound source direction and the face position match. I will do it. By doing so, the utterance target can be correctly estimated even if there is a slight error in the estimation of the face position or the sound source position. If the sound source direction matches the face position, it is determined that the utterance target in the voice section is the media processing device 1. When the sound source direction and the face position do not match, it is highly likely that the utterance target is not the media processing apparatus 1 but is interested in the video, and is not uttered. For example, when multiple people are talking while looking at each other's faces. Therefore, only when the sound source direction matches the face position, the utterance target is assumed to be the media processing apparatus 1, and the user is interested in the video by determining that the utterance has occurred because he / she is interested in the video. It is possible to extract the sections with high accuracy. The display content identification unit 14 detects the name of the program and the channel aired at the time when the utterance of the voice section detected by the voice detection unit 12 is generated. Furthermore, a video section including the time when the utterance was made is cut out. The video section may be cut out in units of programs, or may be cut out by adding the preceding and following video sections to the video section of the time when the utterance was made for a predetermined time length, or performing CM detection, A video section including the time when the speech of the voice section is made and the front and rear of the voice section may be cut out. The impression that the user has on the video is likely to change relatively slowly, not quickly, in a short time. On the other hand, the time when the user is interested in the video and speaks is considered to be relatively short. In other words, it seems that the user was interested in making a voice, not only in the video at the moment when the voice was made, but also in the video before that. In addition, since the user's interest changes relatively slowly, it is highly likely that the user is interested in the video after the voice is spoken. In other words, it is possible to cut out a video section that more accurately reflects the video section that the user is interested in by cutting out the video section including the video before and after that, not just the video when the user made a voice. Become.

  In the labeling unit 8, a label that distinguishes between the case where the speech object in the speech section is the media processing apparatus 1 and the case where the speech object in the speech section is not the media processing apparatus 1 is displayed on the display content identification unit 14 Is labeled for each video section output by. In addition, the storage section 9 stores the start time and end time of the video section, program name, channel information, and label information. People watch TV programs and laugh out loud if they find it interesting. Also, when you are concentrated and drawn into a TV program, you may be surprised to speak unconsciously. In this way, people often speak out when watching TV with interest. On the other hand, it is also assumed that the story is not related to the television program while watching the television with multiple people. By using the configuration of this embodiment, it is possible to identify the scene where the user is watching the display screen and making a voice, so that the video section in which the user is interested in the displayed program is not. It is possible to separate the video section.

  FIG. 3 is an example of a data structure for a video section in which the user is interested and stored in the storage unit 9. The data structure includes a user name or ID, a channel of an interested program, and a start time and an end time of an interested video section. When the display content identification unit 14 cuts out a video section in which the user is interested in a program unit, information specifying a program such as a program name may be added to the item of the data structure. When a plurality of users use it, it is desirable to record the user in association with the user identifier as shown in FIG. For user identification, a user interface such as a mouse, a keyboard, or a touch panel is attached to the present invention, and all user names that use the television are registered at the initial setting, and displayed on the screen of the media processing device 1 when watching the television. The user name can be selected, or it can be combined with a device that automatically performs user authentication using face image authentication technology or voice authentication technology. An embodiment in which the face image authentication technology is used for user authentication will be described later as a second embodiment of the present invention.

Note that the voice detection unit 12 may react only to a specific word that is considered to represent the user's interest (for example, “interesting” or the like) by using a word spot technology of voice recognition technology. By reacting only to a specific word, there is an effect that it is difficult to react to coughing. In the speech recognition dictionary, words that indicate that the user has a good impression of the video section, such as “interesting” or “interesting”, and words that indicate that the user has a bad impression, such as “dull” or “not interesting”. Register. In addition, an item related to the user impression is added to the data for each video section in FIG. 3, and a label that can distinguish whether the user is positive or negative, such as “good impression” and “bad impression” is described. May be. With such a configuration, it is possible not only to search for a video section having a good impression by the user, but also to search for a video section having a bad impression by the user. Further, not only binary values of “good impression” and “bad impression” but also labels such as “smear”, “good”, and “fun” may be added to the data relating to the video section. In this case, the word corresponding to each label such as “smudged” is registered in the speech recognition dictionary in advance, and how to read each word in the speech recognition dictionary so that it can be understood which label each word corresponds to. In addition to the above information, label information is also retained. With this configuration, each video section can be classified more finely according to the user's impression.
FIG. 4 is a block diagram of a second embodiment of the present invention. This embodiment is different from the embodiment shown in FIG. 4 in that a user authentication unit 15 is provided. A description of the same components as those described above will be omitted. The user authentication unit 15 compares the face information of the user registered in advance with the face image output by the user detection unit 5, and determines who the face image is. In the present embodiment, since user authentication is combined, it is possible to automatically authenticate the user, and it is possible to classify the video section in which the user is interested for each user.

  FIG. 5 is a block diagram of the third embodiment of the present invention. In the present embodiment, the robot 16 that is the second processing device controls the media processing device 1. The microphone array 10 and the camera 2-1 are attached to the housing of the robot 16. As described above, the microphone array 10 may be a single microphone. The camera 2-2 is attached to the media processing device 1. The computer 17 can perform signal processing using the control of the robot 16 and the media processing device 1 and the input signals of the camera 2-1, the camera 2-2, and the microphone array 10.

  FIG. 6 is a block diagram of a third embodiment of the present invention. In the present embodiment, the labeling unit 8, the storage unit 9, the display content identification unit 14, the image acquisition unit 4, the user detection unit 5, the voice acquisition unit 6, the speech target determination unit 7, the voice detection unit 12, and the sound source position estimation unit 13. The media processing device 1 serves as the user authentication unit 15, the face position estimation unit 11, and the voice analysis unit 18. Here, a part of the processing unit of the robot 16 may be responsible. In the following description, description of the same components as those in the above-described embodiment will be omitted.

  The image data captured by the camera 2-1 and the camera 2-2 is sent to the image acquisition unit 4, and processing is performed thereafter. In the utterance target determination unit 7, the microphone 2-1 and the microphone 2-2 capture whether or not the sound source direction output from the sound source position detection unit 13 and the user's face position output from the face position estimation unit 11 match. It is determined for each face in the image data. When the sound source direction and the face position are in a predetermined range, a result that “the utterance target is the robot 16” is returned for the face reflected in the camera 2-1. If the face is reflected in the camera 2-2, a determination result “the utterance target is the media processing device 1” is returned. If the sound source direction and face position are not within the predetermined range, nothing is returned. In the labeling unit 8, the display content is based on the determination of the utterance target part when the utterance target object in the voice section is the media processing apparatus 1 and when the utterance target object in the voice section is not the media processing apparatus 1. Label them so that they can be distinguished. Then, it is determined that the user is interested in the labeled video segment when the utterance target is the media processing device 1, and the video segment is stored in the video segment database for each user in the storage unit 9.

  Further, in the labeling unit 8, when the utterance target returned by the utterance target determination unit 7 is the robot 16, the voice section data is passed to the voice analysis unit 18. The voice analysis unit 18 analyzes the input voice section data using a known voice recognition technique, and converts it into a character string indicating the utterance content. When the utterance target is the robot 16, the utterance is considered to be a command to the robot 16, so it is necessary to recognize the utterance by speech, recognize what the command is, and respond to the command. As described above, the voice detection unit 12 may analyze the utterance content using the word spot technology of the voice recognition technology. In this case, the speech recognition dictionaries used by the speech detection unit 12 and the speech analysis unit 18 are different. The voice detection unit 12 uses a dictionary that lists words representing impressions / evaluations of a user's program such as “interesting”, while the voice analysis unit 18 uses a dictionary that already describes commands to a predetermined robot. The robot 16 and the media processing device 1 switch operations according to the utterance content. For example, if the utterance is “change TV”, the program to be displayed on the display device attached to the media processing device 1 is switched, and if the utterance is “turn to this”, the robot 16 is directed toward the user. Or turn to. How the media processing device 1 and the robot 16 switch the operation according to the utterance content is defined in advance in association with each word in the speech recognition dictionary.

  FIG. 7 is a usage example of the third embodiment of the present invention. The face of the user 19-1 faces the robot 16 but does not speak. The face of the user 19-2 faces the media processing device 1 and laughs out loud. The camera 2-1 shows the front face of the user 19-1, and the camera 2-2 shows the front face of the user 19-2. Since the user 19-2 is speaking, the utterance target determination unit 7 determines that the user's utterance target is a media processing device. The labeling unit 8 determines that the user is interested in the video section in which the face is directed to the media processing device, and only the video section in which the user is interested is stored in the video section of the user 19-2 in the storage unit 9. Save to database. In the present embodiment, it is determined for each utterance target whether the user utterance is a voice command to the robot or a voice uttered due to interest in viewing the video. In other words, voice recognition is not performed as a voice command of a robot for an utterance that is discriminated as a voice uttered due to interest in watching the video. For this reason, it is difficult for the robot to recognize the voice that was uttered as a voice command and to react erroneously because it was interested in watching the video. Similarly, a user's utterance uttered as a voice command to the robot is considered to be a voice uttered because he / she was interested in watching the video, and it is difficult to give an incorrect label to the video section. .

  FIG. 8 is a flowchart of a video section search system in which the user is interested using the data structure of the present application stored in the storage unit 9. The user inputs a user name and date via an instruction input unit using the GUI of the search system. The search system searches the storage unit 9 for a video section that matches the input user name and date. Then, the search results are displayed in a list on the GUI. The user can watch the video section by selecting the video section in the displayed list. In this search system, the date is specified and the configuration of the search system is shown. However, not only the date but also a TV channel name or a program name may be specified. If you want to review not only the programs that are being broadcast but also the interesting video scenes of the programs that you have seen in the past, it often happens, but by using this video section search system, past users can easily get interested. You can extract and review scenes with

  The embodiment disclosed in the present application is executed by reading a program that causes a computer to execute the present invention. In addition, it can be executed by cooperation with some hardware.

Basic configuration diagram of a typical embodiment of the present invention Example of block diagram of Embodiment 1 An example of the data structure of the video section that the user is interested in Example of block diagram of embodiment 2 Example of configuration diagram of Example 3 Example of block diagram of Example 3 Example of usage example of Example 3 An example of a flowchart for a video segment search system that the user is interested in

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Media processing apparatus, 2 ... Camera, 3 ... Microphone, 4 ... Image acquisition part, 5 ... User detection part, 6 ... Voice acquisition part, 7 ... Speech object Determination unit, 8 ... labeling unit, 9 ... storage unit, 10 ... microphone array, 11 ... face position estimation unit, 12 ... voice detection unit, 13 ... sound source position estimation unit, 14 ... Display content identification unit, 15 ... User authentication unit, 16 ... Robot, 17 ... Computer, 18 ... Voice analysis unit, 19 ... User

Claims (7)

An image acquisition unit for acquiring images via a camera;
An audio acquisition unit for acquiring sound via a microphone;
A face position estimation unit for detecting the front face of the user from the image;
A face position estimation unit for estimating the position of the front face;
A voice detection unit for detecting a user's utterance from the acquired sound;
A voice position estimation unit for estimating a sound source direction of the detected sound;
It is determined whether the estimated face position and the sound source direction are within a predetermined range, and the determination result that the content displayed on the display unit at the time of the face or sound detection is within the predetermined range is labeled Labeling and
An information processing apparatus comprising: a recording unit that records the labeling result. The information processing apparatus according to claim 1, wherein the labeling is performed by cutting out the content for each section from a commercial to a commercial. Furthermore, it has a user identification part,
The data processing apparatus according to claim 1, wherein the labeling is performed together with identification information of a user identified by the user identification unit. The information processing apparatus according to claim 3, wherein the user identification unit performs the user identification by comparing a face image acquired via the image acquisition unit with a registered image. The voice detection unit performs voice recognition of the detected voice, determines whether the voice is positive or negative with respect to the content from a result of the voice recognition,
5. The information processing apparatus according to claim 1, wherein the labeling unit also labels the result of the determination. Each having a camera and a microphone, connected to a display unit for displaying content and a terminal for controlling content display on the display unit;
An image acquisition unit for acquiring video via each of the cameras;
An audio acquisition unit for acquiring sound via each of the microphones;
A face position estimation unit for detecting the front face of the user from the image;
A face position estimation unit for estimating the position of the front face;
A voice detection unit for detecting a user's utterance from the acquired sound;
A voice position estimation unit for estimating a sound source direction of the detected sound;
Determining whether the estimated face position and the sound source direction are within a predetermined range;
For the information acquired from the display unit, labeling the determination result to the content displayed on the display unit at the time of the face or sound detection;
A recording unit for recording the labeling result;
An information processing apparatus comprising: a voice analysis unit that performs voice recognition on the sound acquired from the terminal determined to be within the predetermined range and gives a command instruction to the terminal based on a recognition result In addition, an instruction input unit is provided,
Receive designation of user, time, program channel name or program name via the instruction input unit,
The video of a section that is associated with the designated user, time, program channel name, or program name and is labeled is displayed on the display unit. An information processing apparatus according to any one of the above.

Images