CN105827931B - It is a kind of based on the audio-frequency inputting method and device taken pictures - Google Patents

Abstract

The embodiment of the invention provides a kind of based on the audio-frequency inputting method and device taken pictures, this method comprises: determining one or more target objects in the preview image data when camera collects preview image data；Sound field angle relative to loudspeaker is calculated to one or more of target objects；The deviation angle relative to loudspeaker is calculated to one or more of target objects；When receive take pictures indicate when, drive the speaker exports audio according to the sound field angle and the deviation angular orientation.The embodiment of the present invention exports the prompt acoustic bearing taken pictures to the target object in practical mapped imaged person, so that imaged person can not hear prompting sound under the factors such as environment is noisy, to improve the success rate taken pictures, avoid the probability taken pictures again, reduce the wasting of resources of electronic equipment, the efficiency taken pictures is improved, the cost taken pictures is reduced；In addition, the prompting sound taken pictures is not heard in the region other than orientation output audio generally, the influence to other people is reduced.

Description

A kind of audio output method and device based on photography

technical field

The present invention relates to the technical field of audio processing, and in particular, to a photography-based audio output method and a photography-based audio output device.

Background technique

With the development of science and technology, various electronic devices, especially mobile devices such as mobile phones and tablet computers, are increasingly used in people's work, study, daily communication and other aspects.

Most of electronic devices such as mobile phones and tablet computers are equipped with cameras, making photography an important application of electronic devices.

At present, in a photographing scene, a prompt sound is usually issued through a speaker to prompt the user and the person to be photographed when the photograph is taken.

However, due to factors such as noise in the environment, there may be cases where the person being photographed cannot hear the prompt sound. If the person being photographed does not hear the prompt sound clearly and moves a little, the picture will be blurred, especially when taking pictures of children, it is difficult to take pictures of children because they cannot attract the attention of children very well.

If the photo is blurred, it is necessary to take a photo again, which wastes the resources of the electronic device, and has low photo-taking efficiency and high cost.

Moreover, in scenarios such as a large number of people, the prompt sound of taking pictures may be transmitted to other people (not the person to be photographed), affecting others.

SUMMARY OF THE INVENTION

In view of the above problems, embodiments of the present invention are proposed to provide a photography-based audio output method and a corresponding photography-based audio output device that overcome the above problems or at least partially solve the above problems.

In order to solve the above problems, an embodiment of the present invention discloses a method for outputting audio based on photography, including:

When the camera captures the preview image data, determining one or more target objects in the preview image data;

calculating a sound field angle relative to the speaker for the one or more target objects;

calculating a deflection angle relative to the speaker for the one or more target objects;

When receiving a photographing instruction, the speaker is driven to output audio directionally according to the sound field angle and the deflection angle.

The embodiment of the present invention also discloses an audio output device based on photography, comprising:

a target object determination module, configured to determine one or more target objects in the preview image data when the camera captures the preview image data;

a sound field angle calculation module, configured to calculate the sound field angle relative to the speaker for the one or more target objects;

a deflection angle calculation module, configured to calculate the deflection angle relative to the speaker for the one or more target objects;

The audio directional output module is configured to drive the speaker to output audio directionally according to the sound field angle and the deflection angle when receiving a photographing instruction.

The embodiments of the present invention include the following advantages:

In the embodiment of the present invention, the sound field angle and deflection angle are calculated for one or more target objects in the preview image data, and the audio is directionally output according to the sound field angle and deflection angle when taking pictures, and the prompt sound of the photograph is directionally output to the target object at The actual mapping of the person to be photographed enables the person to be photographed to hear the prompt sound clearly in the noisy environment, thereby improving the success rate of taking pictures, avoiding the probability of re-taking pictures, reducing the waste of electronic equipment resources, and improving the efficiency of taking pictures. Reduce the cost of taking pictures; in addition, the area other than the directional output audio is generally inaudible to the prompt sound of taking pictures, reducing the impact on others.

Description of drawings

Fig. 1 is the step flow chart of a kind of audio output method embodiment based on photography of the present invention;

Fig. 2A and Fig. 2B are a kind of scene example diagrams of sound field angle and deflection angle of the present invention;

Fig. 3A and Fig. 3B are a kind of calculation example diagram of sound field angle of the present invention;

Fig. 4A to Fig. 4C are a kind of calculation example diagram of deflection angle of the present invention;

FIG. 5 is a structural block diagram of an embodiment of an audio output device based on photography according to the present invention.

Detailed ways

In order to make the above objects, features and advantages of the present invention more clearly understood, the present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

Referring to FIG. 1, a flowchart of steps of an embodiment of a method for outputting audio based on photography of the present invention is shown, which may specifically include the following steps:

Step 101, when the camera captures the preview image data, determine one or more target objects in the preview image data;

It should be noted that the embodiments of the present invention may be applied to various electronic devices, for example, mobile phones, tablet computers, personal digital assistants, wearable devices (such as glasses, watches, etc.), etc., which are not limited in the embodiments of the present invention .

The operating system of the electronic device may include Android (Android), IOS, Windows Phone, Windows, etc., and usually can support the operation of the camera and the speaker.

A camera is a piece of hardware on an electronic device, which can be used for taking pictures and shooting. It can be front-facing (in the same direction as the screen of the electronic device) or rear-facing (in the opposite direction to the screen of the electronic device). The example does not limit this.

In practical applications, the optical image generated by the lens (Lens) of the camera is projected onto the surface of the image sensor processor (Sensor), and then converted into an electrical signal, which is converted into an electrical signal after A/D (analog-to-digital conversion) conversion. It is a digital image signal. The digital image signal is compressed and converted into a specific image file format by a digital signal processing chip (DSP) or an encoding library, and is transmitted to the processor (Central Processing Unit, CPU) of the mobile device through the data bus. processing, it can be displayed on the display screen of the electronic device.

It should be noted that the preview is relative to photographing, and the preview image data is image data provided to the user for adjustment and selection before the photographing is stored, and is stored in the cache.

What the camera collects is a series of preview image data, that is, multiple frames of preview image data. In the embodiment of the present invention, while the camera continuously collects the preview image data, the calculation of the sound field angle and the deflection angle can be continuously performed until the user takes a picture. .

Of course, a selection control can also be provided, through which the user can choose whether to output audio in a direction when taking pictures. In this embodiment of the present invention, before determining one or more target objects in the preview image data, the state of the selection control can be judged , if the state of the selection control is to select the directional output audio, you can continue to perform the step of determining one or more target objects in the preview image data; if the state of the selection control is to output the audio in an undirected manner, you may not perform the step of determining the preview image data One or more target objects in the steps to reduce the resource consumption of the electronic device.

In a preferred embodiment of the present invention, step 101 may include the following sub-steps:

Sub-step S11, detecting the human face in the preview image data, and determining it as one or more target objects.

Generally speaking, when the camera collects preview image data, it can automatically focus through face detection. The so-called face detection may refer to calibrating the position and size of all faces from a frame of preview image data.

In this embodiment of the present invention, the successfully detected object may be identified as the target object, and the person to be photographed actually mapped by the target object may be a person.

Further, taking the Android (Android) system as an example, the Android (Android) system provides two APIs (Application Program Interface, application programming interface), android.media.FaceDetector and android.media.FaceDetector .Face, which implements face detection on a bitmap.

In another preferred embodiment of the present invention, step 101 may include the following sub-steps:

Sub-step S11, when receiving a focus operation instruction, determine that one or more faces in the preview image data corresponding to the focus operation instruction are one or more target objects;

In this embodiment of the present invention, the user can manually focus, and the focus operation instruction is triggered by clicking on the preview image data, selecting the focus frame, etc., and the camera can focus on the object selected by the user according to the focus operation instruction.

In this embodiment of the present invention, the object corresponding to the focus operation instruction may be identified as the target object, and the person to be photographed actually mapped by the target object may be a person, an animal, or a still life.

or,

Sub-step S12, when a cancellation operation instruction is received, cancel one or more target objects that have been determined.

When the person recognized by the camera's automatic focus is not the desired person to be photographed, etc., the user can trigger the cancellation operation instruction by clicking on the target object, etc., to cancel the target object.

In a specific implementation, the target object determined in the preview image data may be one or multiple (ie, two or more), which is not limited in this embodiment of the present invention.

Step 102, calculating the sound field angle relative to the speaker for the one or more target objects;

Speakers can be hardware that outputs audio, such as tiny piezoelectric thin-film ultrasonic sensors.

Assuming that when the speaker outputs audio directionally, the subject can hear the audio within a certain range, and other people outside the range generally cannot hear the audio, then the angle of the range relative to the speaker can be called the sound field angle.

That is, as shown in FIG. 2A and FIG. 2B , the sound field angle R may be the angle range within which the speaker 201 can hear audio when the speaker 201 outputs audio.

In a preferred embodiment of the present invention, step 102 may include the following sub-steps:

Sub-step S21, measuring the target distance between the speaker and the one or more target objects;

It should be noted that the target distance refers to the straight-line distance between the speaker and the entire target object, and does not necessarily refer to the straight-line distance between the speaker and a certain target object.

In a preferred example of the embodiment of the present invention, sub-step S21 may further include the following sub-steps:

Sub-step S211, when the target object is one, obtain the candidate distance between the camera and the target object as the target distance;

or,

Sub-step S212, when there are multiple target objects, obtain a plurality of candidate distances between the camera and the multiple target objects respectively;

Sub-step S213, calculating the target distance by using the plurality of candidate distances.

Since both the camera and the speaker are configured in the same electronic device, the gap between the two is generally small. Therefore, the candidate distance between the camera and the target object (the straight-line distance between the two), and the speaker and the target object The difference between the candidate distances (the straight-line distance between the two) is small, generally within the acceptable difference range.

And, the candidate distance between the camera and the target object is calculated from the preview image data. Therefore, in this example, in order to avoid adding additional hardware, the candidate distance between the camera and the target object can be replaced by the speaker and the target object. candidate distance between.

When the target object is single, the candidate distance between the camera and the target object can be directly set as the target distance between the speaker and the target object.

When there are multiple target objects, multiple candidate distances between the camera and multiple target objects can be used to calculate the target distance between the speaker and the target object, such as calculating the average value of multiple candidate distances, selecting multiple candidate distances The maximum value, selecting the minimum value among multiple candidate distances, etc., are not limited in this embodiment of the present invention.

Further, the candidate distance between the camera and the target object can be calculated in one or more of the following ways:

1. Stereoscopic vision.

It imitates the human stereo perception analysis method, observes the same target object with binocular or multi-eye cameras from different viewpoints, obtains two-dimensional images of the target object from different viewpoints, and calculates the positional deviation of the image pixels through the principle of triangulation. Obtain the 3D information of the target object.

2. Motion ranging method.

The bullet screen camera is used to obtain continuous two-dimensional images of the target object at different times or different spatial positions, and the distance and other parameters of the target object are calculated through the time or space changes of the target object in the two-dimensional image sequence.

3. Monocular ranging.

The ranging methods based on image processing in monocular ranging include: focus ranging (Depth from Focus, DFF) and defocus ranging (Depth from Defocus, DFD).

The focus ranging method is to shoot a series of image data by adjusting the optical writing parameters, find the clearest image data in these image data, and calculate the distance according to the shooting parameters of this image data and using the imaging principle of geometric optics.

The defocus ranging method is based on the principle that the greater the degree of defocusing of the object, the more blurred the image, and the use of two or three frames of image data captured under different optical parameters to determine the diffusion parameters of the defocusing point diffusion function, according to the defocusing diffusion parameters. The relationship with the distance to the target object is used for depth calculation.

Of course, the above calculation methods are only examples. When implementing the embodiments of the present invention, other calculation methods may be set according to actual conditions, which are not limited in the embodiments of the present invention. In addition, in addition to the above calculation methods, those skilled in the art may also adopt other calculation methods according to actual needs, which are not limited in this embodiment of the present invention.

In addition, in addition to multiplexing the candidate distance between the camera and the target object, the candidate distance between the speaker and the target object can also be directly measured by the active ranging method, that is, using a beam such as a laser or a light with a certain texture structure to find the target object, The distance of the object is determined by analyzing the texture deformation of the reflected light of the target object or measuring the propagation time of the speed of light, which is not limited in this embodiment of the present invention.

Sub-step S22, obtaining the audition range distance matched with the one or more target objects;

Assuming that the audio can be heard within a certain range when the audio is directionally output, the audio can generally not be heard outside the range, then the distance of this range is called the audition range distance.

That is, the distance of the audition range, which can be the distance of the range in which the directional output audio can be heard.

Applying the embodiment of the present invention, the matching audition range distance can be set in advance according to the target object. For example, the audition range of one target object is 35cm, and the audition range of two target objects is 45cm.

Of course, a suitable audition range distance can also be calculated according to the focal length, the distance between the target object and the preview image data, and so on, which is not limited in this embodiment of the present invention

Sub-step S23: Calculate the sound field angle according to the target distance and the audition range distance.

In the specific implementation, the target distance and the audition range distance can be used to calculate the sound field angle according to the trigonometric function relationship.

In one embodiment, an isosceles triangle is constructed with the target distance as the height and the audition range as the base, and the sound field angle is calculated according to the following trigonometric function relationship:

tanR/2=(K/2)/L

Among them, R is the sound field angle, K is the audition range distance, and L is the target distance.

Of course, in addition to the tangent tan function, other trigonometric function relationships can also be used to calculate the sound field angle, which is not limited in this embodiment of the present invention.

For example, as shown in FIG. 3A , when there is one target object, the measured candidate distance between the camera and the target object is L ₀ , that is, it can be considered that the target distance between the speaker 301 and the target object is L ₀ , and the audition range distance of one target object is L 0 . is K ₀ , then take the target distance L ₀ as the height and the audition range distance K ₀ as the base to construct an isosceles triangle, and calculate the sound field angle R ₀ according to the following trigonometric function relationship:

tanR ₀ /2=(K ₀ /2)/L ₀

For another example, as shown in FIG. 3B , when there are three target objects, the distribution of candidate distances between the camera and the target object is measured as L ₂ , L ₃ , and L ₄ , that is, the target distance between the speaker 301 and the target object L ₁ =( L ₂ +L ₃ +L ₄ )/3, the audition range distance of the three target objects is K ₁ , then take the target distance L ₁ as the height and the audit range distance K ₁ as the base to construct an isosceles triangle, according to the following triangle The functional relationship calculates the sound field angle R ₁ :

tanR ₁ /2=(K ₁ /2)/L ₁

Step 103, calculating the deflection angle relative to the speaker for the one or more target objects;

As shown in FIG. 2A and FIG. 2B , the deflection angle S may be the angle at which the person being photographed deviates from the forward direction of the speaker 201 actually mapped by the target object.

In a preferred embodiment of the present invention, step 103 may include the following sub-steps:

Sub-step S31, project the target object in the preview image into a preset coordinate system, and the coordinate system is constructed based on the position of the speaker;

Since the preview image data collected by the camera is generally proportional to the actual scene, the angle of the target object deviating from the speaker in the preview image data is generally the same as the deviation angle of the subject relative to the speaker mapped by the target object in practice. it's the same.

By applying the embodiment of the present invention, a coordinate system can be constructed based on the position of the speaker in advance.

It should be noted that the positions of the speaker and the sound outlet (a hole on the housing of the electronic device through which the audio from the speaker can be transmitted) are usually opposite, and the projection position of the speaker on the electronic device (that is, the position of the speaker on the electronic device The back of the device is used as a projection surface, and the position where the speaker is projected onto the projection surface) often coincides with the sound outlet. Therefore, a coordinate system can also be directly constructed based on the sound outlet.

Among them, the projection is a method in which the projection line passes through an object (such as a speaker), and is projected onto a selected projection surface, and a figure is obtained on this surface.

In this embodiment of the present invention, a coordinate system, such as a rectangular coordinate system, may be constructed with the speaker or the sound outlet as the origin.

Using the plane of the coordinate system as the projection plane, project the target object in the preview image into the coordinate system to calculate the deflection angle.

Sub-step S32, in the coordinate system, calculate the focus coordinates of the one or more target objects;

The focus coordinates can be the coordinates of the focus when the focus operation is performed on the target object.

In a preferred example of the embodiment of the present invention, sub-step S32 may further include the following sub-steps:

Sub-step S321, when the target object is one, search for the first coordinate of the upper left corner of the target object and the second coordinate of the lower right corner;

Sub-step S322, calculating the average value of the first coordinate and the second coordinate as the focus coordinate;

In this example, the target object is an area, and the midpoint of the area can be used as the focus coordinate.

or,

Sub-step S323, when the target object is multiple, find the third coordinate of the upper left corner of the leftmost target object, the fourth coordinate of the lower right corner, and the fifth coordinate of the upper left corner of the rightmost target object, The sixth coordinate of the lower right corner;

Sub-step S324: Calculate the average value of the third coordinate and the fourth coordinate, and the average value of the fifth coordinate and the sixth coordinate, respectively, as focus coordinates.

In this example, the target object is an area, and the midpoint of the area can be used as the focus coordinate.

If there are multiple target objects, the focus coordinates of the leftmost target object and the focus coordinates of the rightmost target object can be calculated, and there are two focus coordinates in total.

In sub-step S33, the deflection angle is calculated using the focus coordinates.

In a specific implementation, the focal coordinates can be used to calculate the deflection angle according to the trigonometric function relationship.

In a preferred example of the embodiment of the present invention, when there is one target object, a right-angled triangle can be constructed with the value of the X-axis coordinate of the focus coordinate and the value of the Y-axis coordinate as a right-angled side, and the bias is calculated according to the following trigonometric function relationship horn:

tanS=X ₀ /Y ₀

Among them, S is the deflection angle, X ₀ is the value of the X-axis coordinate of the focus coordinate, and Y ₀ is the value of the Y-axis coordinate of the focus coordinate.

Certainly, in addition to the tangent tan function, other trigonometric function relationships may also be used to calculate the deflection angle, which is not limited in this embodiment of the present invention.

In another preferred example of the embodiment of the present invention, sub-step S33 may further include the following sub-steps:

Sub-step S331, when there are multiple target objects, use the focus coordinates of the leftmost target object to calculate the first candidate deflection angle;

Sub-step S332, using the focus coordinates of the rightmost target object to calculate the second candidate deflection angle;

Sub-step S333, if the leftmost target object and the rightmost target object are located on both sides of the speaker, set the first characteristic angle as the deflection angle;

Wherein, the first characteristic angle is half of the difference between the first candidate deflection angle and the second candidate deflection angle;

Sub-step S334, if the leftmost target object and the rightmost target object are located on the same side of the speaker, set the second characteristic angle as the deflection angle;

Wherein, the second characteristic angle is half of the sum of the first candidate deflection angle and the second candidate deflection angle.

In this example, for each focus coordinate, a right-angled triangle can be constructed with the value of the X coordinate and the value of the Y coordinate of the focus coordinate as a right-angled side, and the first candidate deflection angle and the second candidate deflection angle are calculated according to the trigonometric function relationship, If the leftmost target object and the rightmost target object are located on both sides of the forward direction of the speaker, the deflection angle is:

S=(S ₁ -S ₂ )/2

If the leftmost target object and the rightmost target object are located on the same side (such as left or right) in the forward direction of the speaker, the deflection angle is:

S=(S ₁ +S ₂ )/2

Wherein, S is the deflection angle, S1 is the _first candidate deflection angle, and S2 is the _second candidate deflection angle.

It should be noted that the above calculation

For example, as shown in Fig. 4A, the XY coordinate system is constructed by taking the projection O of the speaker as a dot, and when there is one target object, the upper left corner of the target object is point A (X ₁ , Y ₁ ), and the lower right corner is the point B(X ₂ , Y ₂ ), then the focus coordinate U((X ₁ +X ₂ )/2, (Y ₁ +Y ₂ )/2), then the X-axis coordinate of the focus coordinate U (X ₁ +X ₂ )/2 and the Y-axis coordinates (Y ₁ +Y ₂ )/2 are right-angled sides, construct a right-angled triangle, and calculate the deflection angle S ₃ according to the following trigonometric function:

tanS ₃ =(X ₁ +X ₂ )/2/(Y ₁ +Y ₂ )/2;

For another example, as shown in FIG. 4B , the XY coordinate system is constructed by taking the projection O of the speaker as a dot, and when there are three target objects, the upper left corner of the leftmost target object is point C (X ₃ , Y ₃ ) , the lower right corner is point D(X ₄ , Y ₄ ), then the focus coordinates V((X ₃ +X ₄ )/2, (Y ₃ +Y ₄ )/2), the upper left corner of the rightmost target object is Point E(X ₅ , Y ₅ ), the lower right corner is point F (X ₆ , Y ₇ ), then the focus coordinates W((X ₅ +X ₆ )/2, (Y ₅ +Y ₆ )/2);

Then take the X-axis coordinates (X ₃ +X ₄ )/2 and Y-axis coordinates (Y ₃ +Y ₄ )/2 of the focus coordinate V as the right-angled sides, and the X-axis coordinates (X ₅ +X of the focus coordinate W) ₆ )/2 and the Y-axis coordinates (Y ₅ +Y ₆ )/2 are right-angled sides, construct a right-angled triangle, and calculate the first candidate deflection angle S ₅ and the second candidate deflection angle S ₆ according to the following trigonometric function relationship:

tanS ₅ =(X ₃ +X ₄ )/2/(Y ₃ +Y ₄ )/2;

tanS ₆ =(X ₅ +X ₆ )/2/(Y ₅ +Y ₆ )/2;

Assuming that S ₅ is 30° and S ₆ is 50°, the deflection angle S ₄ =(S ₆ -S ₅ )/2=10°, which means that the three target objects are deviated by 10° to the right of the speaker as a whole.

For another example, as shown in FIG. 4C , the XY coordinate system is constructed by taking the projection O of the speaker as a dot, and when there are three target objects, the upper left corner of the leftmost target object is point G (X ₇ , Y ₇ ) , the lower right corner is the point H(X ₈ , Y ₈ ), then the focus coordinates are M((X ₇ +X ₈ )/2, (Y ₇ +Y ₈ )/2), the upper left corner of the rightmost target object is point I(X ₉ , Y ₉ ), and the lower right corner is point J(X ₁₀ , Y ₁₀ ), then the focus coordinates are N((X ₉ +X ₁₀ )/2, (Y ₉ +Y ₁₀ )/2) ;

Then take the X-axis coordinates (X ₇ +X ₈ )/2 and Y-axis coordinates (Y ₇ +Y ₈ )/2 of the focus coordinate V as the right-angled sides, and the X-axis coordinates (X ₉ +X of the focus coordinate W) ₁₀ )/2 and the Y-axis coordinates (Y ₉ +Y ₁₀ )/2 are right-angled sides, construct a right-angled triangle, and calculate the first candidate deflection angle S ₈ and the second candidate deflection angle S ₉ according to the following trigonometric function relationship:

tanS ₈ =(X ₇ +X ₈ )/2/(Y ₇ +Y ₈ )/2;

tanS ₉ =(X ₉ +X ₁₀ )/2/(Y ₉ +Y ₁₀ )/2;

Assuming that S ₈ is 50° and S ₉ is 30°, the deflection angle S ₇ =(S ₈ +S ₉ )/2=40°, which means that the three target objects are deviated by 40° to the right of the speaker as a whole.

Step 104, when receiving the photographing instruction, drive the speaker to output audio directionally according to the sound field angle and the deflection angle.

In the specific implementation, the user can trigger the photographing instruction by clicking the photographing control, clicking on the preview image data, etc., and the camera performs the photographing process. The area emits a photo prompt sound to remind the person being photographed that the photo is being taken.

Directional output audio can generate highly directional audible sound (ie, audio directional) by utilizing the nonlinear propagation effect of ultrasonic waves in the air.

According to the nonlinear acoustic theory, two plane waves propagate nonlinearly in an inhomogeneous medium. When two series of electrical signals with frequencies f ₁ and f ₂ are input to the ultrasonic transducer (one of the components of the loudspeaker), the ultrasonic transducer passes through The mechanical vibration emits two series of ultrasonic waves with frequencies f ₁ and f ₂ into the air. When the two series of ultrasonic waves propagate in the air, there will be nonlinear interaction, thus generating fundamental frequencies f ₁ , f ₂ , sum frequency f ₁ +f ₂ , difference frequency f ₁ -f ₂ and various orders Complex sound waves including harmonics. Since the sound attenuation coefficient α is proportional to the square of the frequency, the ultrasonic signals f ₁ , f ₂ , f ₁ +f ₂ and the harmonics with higher frequencies will be quickly absorbed by the air, leaving the difference in the audio frequency range. The frequency signals f ₁ -f ₂ continue to propagate in the air.

Whether a sound wave has directivity is closely related to the ratio of the wavelength of the sound wave to the size of the sound source. When the wavelength of the sound wave is much larger than the size of the sound source, the sound wave has no directivity; when the wavelength of the sound wave is close to or much smaller than the size of the sound source, the sound wave will gradually show stronger and stronger directivity. Therefore, when the ultrasonic frequencies f ₁ and f ₂ are selected for the gift, the difference frequency signals f ₁ -f ₂ can be in the audible range, so as to generate sound waves through ultrasonic waves.

Further, in the parametric acoustic array theory, the ultrasonic transducer (one of the components of the loudspeaker) emits a strongly modulated ultrasonic wave into the air medium, and the ultrasonic wave propagates along its main axis direction (such as the sound field angle, the direction of the deflection angle) In the process of traveling, the audio signal is continuously modulated through nonlinear interaction, and these continuously demodulated audio waves are accumulated and superimposed, so that an end-fire virtual sound source array (end-fire vitual arrray) is realized in this way . This virtual sound source array is the so-called parametric acoustic array, and the parametric acoustic array makes the energy of the acoustic wave continuously strengthen in the direction of the sound wave. Due to the strong directivity of ultrasonic waves, this superposition enhancement effect will be very weak beyond the direction of the main propagation axis (such as the direction of the sound field angle and the deflection angle), which eventually makes the sound wave in the main propagation axis direction (such as the sound field angle, deflection angle). direction) has a strong directivity.

It should be noted that, for the method embodiment, for the sake of simple description, it is expressed as a series of action combinations, but those skilled in the art should know that the embodiment of the present invention is not limited by the described action sequence, because According to the embodiment of the present invention, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

Referring to FIG. 5, a structural block diagram of an embodiment of an audio output device based on photography of the present invention is shown, which may specifically include the following modules:

A target object determination module 501, configured to determine one or more target objects in the preview image data when the camera captures the preview image data;

a sound field angle calculation module 502, configured to calculate the sound field angle relative to the speaker for the one or more target objects;

a deflection angle calculation module 503, configured to calculate the deflection angle relative to the speaker for the one or more target objects;

The audio directional output module 504 is configured to drive the speaker to directional output audio according to the sound field angle and the deflection angle when receiving the photographing instruction.

In a preferred embodiment of the present invention, the target object determination module 501 may include the following sub-modules:

The first determination submodule is used to detect the human face in the preview image data, and determine it as one or more target objects.

In a preferred embodiment of the present invention, the target object determination module 501 may include the following sub-modules:

a second determination submodule, configured to, when receiving a focus operation instruction, determine that one or more faces in the preview image data corresponding to the focus operation instruction are one or more target objects;

or,

The cancellation sub-module is used to cancel one or more target objects that have been determined when a cancellation operation instruction is received.

In a preferred embodiment of the present invention, the sound field angle calculation module 502 may include the following sub-modules:

a target distance measurement submodule for measuring the target distance between the speaker and the one or more target objects;

an audition range distance acquisition sub-module, used for acquiring the audition range distance matched with the one or more target objects, where the audition range distance is the distance of the range where the audio can be heard;

The first calculation submodule is configured to calculate the sound field angle according to the target distance and the audition range distance.

In a preferred example of the embodiment of the present invention, the target distance measurement submodule may further include the following submodules:

The first acquisition submodule is used to acquire the candidate distance between the camera and the target object as the target distance when the target object is one;

or,

a second obtaining submodule, configured to obtain a plurality of candidate distances between the camera and the plurality of target objects when there are multiple target objects;

The second calculation submodule is configured to calculate the target distance by using the plurality of candidate distances.

In a preferred embodiment of the present invention, the deflection angle calculation module 503 may include the following sub-modules:

a projection sub-module for projecting the target object in the preview image data into a preset coordinate system, the coordinate system being constructed based on the position of the speaker;

a focus coordinate calculation submodule, configured to calculate the focus coordinates of the one or more target objects in the coordinate system;

The third calculation submodule is used for calculating the deflection angle by using the focus coordinates.

In a preferred example of the embodiment of the present invention, the focus coordinate calculation submodule may further include the following submodules:

The first search submodule is used to search for the first coordinate of the upper left corner of the target object and the second coordinate of the lower right corner of the target object when the target object is one;

a fourth calculation submodule, used for calculating the average value of the first coordinate and the second coordinate as the focus coordinate;

or,

The second search submodule is used to search for the third coordinate of the upper left corner of the leftmost target object, the fourth coordinate of the lower right corner of the leftmost target object, and the upper left corner of the rightmost target object when there are multiple target objects. The fifth coordinate, the sixth coordinate of the lower right corner;

The fifth calculation sub-module is configured to calculate the average value of the third coordinate and the fourth coordinate, and the average value of the fifth coordinate and the sixth coordinate, respectively, as the focus coordinate.

In a preferred example of the embodiment of the present invention, the third calculation submodule may further include the following submodules:

a first candidate deflection angle calculation submodule, configured to calculate the first candidate deflection angle by adopting the focus coordinates of the leftmost target object when there are multiple target objects;

The second candidate deflection angle calculation submodule is used to calculate the second candidate deflection angle by adopting the focus coordinates of the rightmost target object;

The first setting submodule is used to set the first characteristic angle as the deflection angle when the leftmost target object and the rightmost target object are located on both sides of the speaker;

The second setting submodule is used to set the second characteristic angle as the deflection angle when the leftmost target object and the rightmost target object are located on the same side of the speaker;

Wherein, the first characteristic angle is half of the difference between the first candidate deflection angle and the second candidate deflection angle; the second characteristic angle is the first candidate deflection angle and the second candidate deflection angle half of the sum of the angles.

As for the apparatus embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and reference may be made to the partial description of the method embodiment for related parts.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments may be referred to each other.

It should be understood by those skilled in the art that the embodiments of the embodiments of the present invention may be provided as a method, an apparatus, or a computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media having computer-usable program code embodied therein, including but not limited to disk storage, CD-ROM, optical storage, and the like.

Embodiments of the present invention are described with reference to flowcharts and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present invention. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing terminal equipment to produce a machine that causes the instructions to be executed by the processor of the computer or other programmable data processing terminal equipment Means are created for implementing the functions specified in the flow or flows of the flowcharts and/or the blocks or blocks of the block diagrams.

These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal equipment to operate in a particular manner, such that the instructions stored in the computer readable memory result in an article of manufacture comprising instruction means, the The instruction means implement the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing terminal equipment, so that a series of operational steps are performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby executing on the computer or other programmable terminal equipment The instructions executed on the above provide steps for implementing the functions specified in the flowchart or blocks and/or the block or blocks of the block diagrams.

Although preferred embodiments of the embodiments of the present invention have been described, additional changes and modifications to these embodiments may be made by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the embodiments of the present invention.

Finally, it should also be noted that in this document, relational terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply these entities or that there is any such actual relationship or sequence between operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or terminal device comprising a list of elements includes not only those elements, but also a non-exclusive list of elements. other elements, or also include elements inherent to such a process, method, article or terminal equipment. Without further limitation, an element defined by the phrase "comprises a..." does not preclude the presence of additional identical elements in the process, method, article or terminal device comprising said element.

A photography-based audio output method and a photography-based audio output device provided by the present invention have been described above in detail. In this paper, specific examples are used to illustrate the principles and implementations of the present invention. The above embodiments The description is only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation and application scope. However, the contents of this specification should not be construed as limiting the present invention.

Claims (12)

1. A photographing-based audio output method, comprising:

when the camera acquires preview image data, determining one or more target objects in the preview image data;

calculating an acoustic field angle relative to a speaker for the one or more target objects;

calculating a deflection angle relative to a speaker for the one or more target objects;

when a photographing instruction is received, driving an ultrasonic transducer in a loudspeaker, and outputting audio directionally according to the acoustic field angle and the deflection angle;

wherein the step of calculating a deviation angle with respect to the speaker for the one or more target objects comprises:

projecting the target object in the preview image data to a preset coordinate system, wherein the coordinate system is constructed based on the position of a loudspeaker;

calculating focus coordinates of the one or more target objects in the coordinate system; calculating a deviation angle by using the focal point coordinate;

wherein the step of calculating a deflection angle using the focus coordinates comprises:

when the target objects are multiple, calculating a first candidate deviation angle by using the focus coordinates of the leftmost target object;

calculating a second candidate deflection angle by using the focus coordinate of the rightmost target object;

if the target object on the leftmost side and the target object on the rightmost side are positioned on two sides of the loudspeaker, setting the first characteristic angle as a deflection angle;

if the leftmost target object and the rightmost target object are located on the same side of the loudspeaker, setting the second characteristic angle as a deflection angle;

the first characteristic angle is half of a difference between the first candidate deflection angle and the second candidate deflection angle; the second feature angle is half of a sum of the first candidate deflection angle and the second candidate deflection angle.

2. The method of claim 1, wherein the step of determining one or more target objects in the preview image data comprises:

and detecting the human face in the preview image data and determining the human face as one or more target objects.

3. The method of claim 1, wherein the step of determining one or more target objects in the preview image data comprises:

when a focusing operation instruction is received, determining one or more faces in the preview image data corresponding to the focusing operation instruction as one or more target objects;

or,

and when a cancel operation instruction is received, canceling the determined one or more target objects.

4. The method of claim 1, 2 or 3, wherein the step of calculating an acoustic field angle with respect to a speaker for the one or more target objects comprises:

measuring a target distance between a speaker and the one or more target objects;

acquiring audition range distances matched with the one or more target objects, wherein the audition range distances are distances of ranges in which audio can be heard;

and calculating an acoustic field angle according to the target distance and the listening range distance.

5. The method of claim 4, wherein the step of measuring a target distance between a speaker and the one or more target objects comprises:

when one target object is available, acquiring a candidate distance between a camera and the target object as a target distance;

or,

when a plurality of target objects are available, respectively acquiring a plurality of candidate distances between a camera and the plurality of target objects; and calculating the target distance by adopting the plurality of candidate distances.

6. The method of claim 1, wherein the step of calculating the focus coordinates of the one or more target objects comprises:

when one target object is available, searching a first coordinate of the upper left corner and a second coordinate of the lower right corner of the target object;

calculating an average value of the first coordinate and the second coordinate as a focus coordinate;

or,

when a plurality of target objects are available, searching a third coordinate of the upper left corner and a fourth coordinate of the lower right corner of the leftmost target object, and searching a fifth coordinate of the upper left corner and a sixth coordinate of the lower right corner of the rightmost target object;

and respectively calculating the average value of the third coordinate and the fourth coordinate, and the average value of the fifth coordinate and the sixth coordinate as the focal point coordinate.

7. An audio output device based on photographing, comprising:

the target object determining module is used for determining one or more target objects in the preview image data when the preview image data is acquired by the camera;

an acoustic field angle calculation module to calculate an acoustic field angle with respect to a speaker for the one or more target objects;

a deflection angle calculation module for calculating a deflection angle relative to the speaker for the one or more target objects;

the audio directional output module is used for driving a loudspeaker to directionally output audio according to the acoustic field angle and the deflection angle when receiving a photographing instruction;

wherein the deflection angle calculation module includes:

the projection submodule is used for projecting the target object in the preview image data to a preset coordinate system, and the coordinate system is constructed based on the position of a loudspeaker;

a focus coordinate calculation sub-module for calculating focus coordinates of the one or more target objects in the coordinate system;

the third calculation submodule is used for calculating a deviation angle by adopting the focal point coordinate;

wherein the third computation submodule comprises:

a first candidate deflection angle calculation sub-module, configured to calculate a first candidate deflection angle using the focus coordinate of the leftmost target object when there are multiple target objects;

the second candidate deflection angle calculation submodule is used for calculating a second candidate deflection angle by adopting the focus coordinate of the rightmost target object;

the first setting submodule is used for setting the first characteristic angle as a deflection angle when the leftmost target object and the rightmost target object are positioned at two sides of the loudspeaker;

the second setting submodule is used for setting the second characteristic angle as a deflection angle when the leftmost target object and the rightmost target object are positioned on the same side of the loudspeaker;

the first characteristic angle is half of a difference between the first candidate deflection angle and the second candidate deflection angle; the second feature angle is half of a sum of the first candidate deflection angle and the second candidate deflection angle.

8. The apparatus of claim 7, wherein the target object determination module comprises:

and the first determining submodule is used for detecting the human face in the preview image data and determining the human face as one or more target objects.

9. The apparatus of claim 7, wherein the target object determination module comprises:

the second determining submodule is used for determining one or more faces in the preview image data corresponding to the focusing operation instruction as one or more target objects when the focusing operation instruction is received;

or,

and the cancellation submodule is used for canceling the determined one or more target objects when a cancellation operation instruction is received.

10. The apparatus of claim 7, 8 or 9, wherein the acoustic field angle calculation module comprises:

a target distance measurement submodule for measuring a target distance between the speaker and the one or more target objects;

the audition range distance acquisition sub-module is used for acquiring audition range distances matched with the one or more target objects, wherein the audition range distances are distances of a range in which audio can be heard;

and the first calculation submodule is used for calculating an acoustic field angle according to the target distance and the listening range distance.

11. The apparatus of claim 10, wherein the target distance measurement sub-module comprises:

the first obtaining submodule is used for obtaining a candidate distance between a camera and the target object as a target distance when the number of the target objects is one;

or,

the second obtaining sub-module is used for respectively obtaining a plurality of candidate distances between the camera and the plurality of target objects when the plurality of target objects are provided;

a second calculation sub-module for calculating a target distance using the plurality of candidate distances.

12. The apparatus of claim 7, wherein the focus coordinate calculation submodule comprises:

the first searching submodule is used for searching a first coordinate of the upper left corner and a second coordinate of the lower right corner of the target object when the number of the target objects is one;

the fourth calculation submodule is used for calculating the average value of the first coordinate and the second coordinate to be used as a focus coordinate;

or,

the second searching submodule is used for searching a third coordinate of the upper left corner and a fourth coordinate of the lower right corner of the leftmost target object, and searching a fifth coordinate of the upper left corner and a sixth coordinate of the lower right corner of the rightmost target object when the target objects are multiple;

and the fifth calculating submodule is used for calculating the average value of the third coordinate and the fourth coordinate and the average value of the fifth coordinate and the sixth coordinate as the focal point coordinate.