CN118400584A - Method and device for controlling video content playback following user movements - Google Patents

Abstract

A method for controlling the playback of video content following user movements, comprising: displaying first video content on a display screen of a display device, wherein the first video content is video content in a first area of a screen of an original video, and the first area is smaller than the area occupied by the screen of the original video; when the relative position between the user and the display device changes, switching the first video content on the display screen to target video content, wherein the target video content is video content in a second area of the screen of the original video, and the second area is smaller than the area occupied by the screen of the original video. This method enables the played video content to change following the user's movements, so that the user can choose the angle and field of view of the video according to the part that the user is personally interested in, thereby increasing the sense of presence and perspective of the video program, simulating the feeling of being at the scene, and improving the user experience.

Description

Method and device for controlling video content playback following user movements

This application is a divisional application of the Chinese patent application filed with the State Intellectual Property Office of China on April 11, 2019, with application number 201980078423.6 and application name “Method and device for controlling the playback of video content by following user motion”, all contents of which are incorporated by reference in this application.

Technical Field

The present application relates to the field of video technology, and more particularly to a method and device for controlling the playback of video content following user movements.

Background technique

In recent years, television and various display technologies have developed rapidly, and manufacturers have also continuously developed new technologies to enhance the visual experience of viewers (or users). For video technology, they are even more pursuing immersive experience.

In order to create an immersive effect, the continuous development of video technology has made videos higher definition and more three-dimensional. Although the videos are higher definition and more three-dimensional, the video images played are edited and produced by the on-site director. The audience cannot watch the parts they are interested in according to their own choices like they would at the video recording site, resulting in a poor user experience.

Summary of the invention

The embodiments of the present application provide a method and device for controlling the playback of video content following the movement of a user, controlling the playback of video content according to the position of the user, increasing the user's sense of presence and perspective when watching the video, and improving the user experience.

In a first aspect, a method for controlling the playback of video content following user motion is provided, the method comprising:

The relative position of the user and the display device is obtained, where the relative position is obtained by the sensor when the display device plays a video; the target video content is determined in the video source according to the relative position, where the target video content is the video content in the video source that matches the user's field of vision when the user is at the relative position; the target video content is played on the display screen of the display device, so that the video content follows the user's movement and improves the user experience.

The method for controlling the playback of video content provided in the embodiment of the present application obtains the relative position of the user and the display device, and determines the target video content in the video source according to the relative position, wherein the target video content is the video content in the video source that matches the visual range of the user when the user is at the relative position, and plays the target video content on the display screen of the display device, thereby realizing that the playback video follows the movement of the user, so that the user can choose the angle and field of view of watching the video according to the part that the user is personally interested in, and presents the video content from different perspectives to the audience according to the relative position of the user and the display device, thereby increasing the sense of presence and perspective of the video program, simulating the feeling of being at the scene in person, and improving the user experience.

In combination with the first aspect, in a first possible implementation method of the first aspect, determining the target video content in the video source according to the relative position includes: determining the sight angle of the user at the relative position according to the relative position, the sight angle is used to reflect the sight range, the sight angle is the spatial angle of the user's sight, and the user's sight is the line connecting the eyes and the center point of the display screen when the user is in the relative position; determining the target video content from the video source according to the sight angle, the target video content is the video content that matches the user's sight angle.

In combination with the first possible implementation method of the first aspect, in a second possible implementation method of the first aspect, the line of sight angle includes a first angle and a second angle; the first angle is the angle between the user's line of sight and the y-axis when the line of sight of the user is mapped to the x-y plane, and the x-y plane is the plane formed by the x-axis and the y-axis; the second angle is the angle between the user's line of sight and the z-axis when the line of sight of the user is mapped to the y-z plane, and the y-z plane is the plane formed by the y-axis and the z-axis; wherein the y-axis is perpendicular to the plane where the display screen is located, the z-axis is perpendicular to the ground and parallel to the plane where the display screen is located, the x-axis and the z-axis form an x-z plane, the x-z plane is the plane where the display screen is located, the x-axis, the y-axis and the z-axis intersect at the origin of the coordinate axes, and the origin of the coordinate axes is the center point of the display screen.

In combination with the second possible implementation method of the first aspect, in a third possible implementation method of the first aspect, the target video content is determined from the video source according to the line of sight angle, including: determining the target video content from the video source according to the first angle and the second angle, the target video content being the target video content, wherein the angle between the projection of the line connecting the center point of the target video content and the center point of the display screen on the x-y plane and the y-axis is the first angle, and the angle between the projection of the line connecting the center point of the target video content and the center point of the display screen on the y-z plane and the z-axis is the second angle.

In combination with the second possible implementation method of the first aspect, in a fourth possible implementation method of the first aspect, the relative position is the distance L1 between the user and the center point of the display screen, the video source includes a video shooting distance L2 and a video content viewing angle range, and L2 is the distance from the video content of the video source to the center point of the display screen; determining the target video content in the video source according to the relative position includes: determining the target video content according to the line of sight angle, L1, L2, the viewing angle range and the size of the display screen.

In combination with the fourth possible implementation method of the first aspect, in a fifth possible implementation method of the first aspect, the viewing angle range includes a horizontal display viewing angle and a vertical display viewing angle, the horizontal display viewing angle is a display range angle of the video content of the video source mapped to the x-y plane, and the vertical display viewing angle is a display range angle of the video content of the video source mapped to the y-z plane; the size of the display screen includes a width D of the display screen and a height H of the display screen; the target video content is determined according to the line of sight angle, L1, L2, the viewing angle range and the size of the display screen, including: determining the display range of the target video content on the x-y plane according to the first angle, L1, L2, the horizontal display viewing angle and the width D of the display screen; determining the display range of the target video content on the y-z plane according to the second angle, L1, L2, the vertical display viewing angle and the height H of the display screen.

In combination with the first aspect, in a sixth possible implementation of the first aspect, determining the target video content in the video source according to the relative position includes: determining a first intersection point of the video source with a line connecting the user's eyes and a center point of the display screen when the user is at the relative position; and taking the video content in the video source with the first intersection point as the center point and having the same aspect ratio as the display screen as the target video content.

In combination with the first aspect, or any of the foregoing possible implementations of the first aspect, in a seventh possible implementation of the first aspect, determining the target video content in the video source according to the relative position includes: determining the target video content according to the relative position and the speed of the user's movement.

In combination with the first aspect, or any of the above possible implementations of the first aspect, in an eighth possible implementation of the first aspect, the method further includes: obtaining action information of the user; and determining target video content based on the relative position and the action information.

In combination with the eighth possible implementation method of the first aspect, in a ninth possible implementation method of the first aspect, the target video content is determined based on the relative position and the action information, including: determining the first video content in the video source based on the relative position; determining the presentation rule of the pre-stored video content based on the action information, the presentation rule corresponding to the action information; determining the target video content based on the first video content and the presentation rule.

In combination with the eighth or ninth possible implementation of the first aspect, in a tenth possible implementation of the first aspect, the action information includes at least one of information of a telescope gesture, information of a gesture of pushing both hands forward, and information of a gesture of moving both hands backward.

In combination with the ninth or tenth possible implementation method of the first aspect, in the eleventh possible implementation method of the first aspect, the target video content is determined according to the first video content and the presentation rule, including: enlarging or reducing the first video content with the center point of the first video content as the center to determine the target video content.

In combination with the first aspect, or any of the above possible implementation methods of the first aspect, in a twelfth possible implementation method of the first aspect, determining the target video content in the video source according to the relative position includes: determining a cropping or scaling strategy of the video source according to the relative position to obtain the target video content.

In combination with the first aspect, or any of the above possible implementations of the first aspect, in a thirteenth possible implementation of the first aspect, the video source includes at least one of the following: a video with a resolution greater than or equal to 4K, a video with a viewing angle greater than or equal to 140 degrees, or a long strip video.

In combination with the first aspect, or any of the above-mentioned possible implementation methods of the first aspect, in a fourteenth possible implementation method of the first aspect, in an initial state, a preset second video content is displayed on the display screen, and the second video content is part of the video content of the video source.

In a second aspect, a device for controlling the playback of video content following user movement is provided, the device comprising a processor and a transmission interface; the transmission interface is used to receive the user's position obtained by a sensor when the user is watching a video; the processor is used to determine target video content in a video source according to the user's position, the target video content being video content in the video source that matches the user's line of sight at the said position; the transmission interface is also used to transmit the target video content to a display screen so that the display screen plays the target video content, thereby achieving the goal of video content following the user's movement and improving the user experience.

It should be understood that when the device is a chip, the processor and the transmission interface both belong to the chip, or it can be said that the transmission interface is an interface used by the processor to send and receive data.

In combination with the second aspect, in a first possible implementation method of the second aspect, the processor is specifically used to: determine the line of sight angle of the user at the position according to the user's position, the line of sight angle is used to reflect the line of sight range, the line of sight angle is the spatial angle of the user's line of sight, and the user's line of sight is the line connecting the eyes and the center point of the display screen when the user is at the position; determine the target video content from the video source according to the line of sight angle, the target video content is the video content that matches the user's line of sight angle.

In combination with the first possible implementation method of the second aspect, in a second possible implementation method of the second aspect, the line of sight angle includes a first angle and a second angle; the first angle is the angle between the user's line of sight and the y-axis when the line of sight is mapped to the x-y plane, and the x-y plane is the plane formed by the x-axis and the y-axis; the second angle is the angle between the user's line of sight and the z-axis when the line of sight is mapped to the y-z plane, and the y-z plane is the plane formed by the y-axis and the z-axis; wherein the y-axis is perpendicular to the plane where the display screen is located, the z-axis is perpendicular to the ground and parallel to the plane where the display screen is located, the x-axis and the z-axis form an x-z plane, the x-z plane is the plane where the display screen is located, the x-axis, the y-axis and the z-axis intersect at the origin of the coordinate axes, and the origin of the coordinate axes is the center point of the display screen.

In combination with the second possible implementation method of the second aspect, in a third possible implementation method of the second aspect, the processor is specifically used to: determine the target video content from the video source according to the first angle and the second angle, the projection of the line connecting the center point of the target video content and the center point of the display screen on the x-y plane having an angle of the first angle with the y-axis, and the projection of the line connecting the center point of the target video content and the center point of the display screen on the y-z plane having an angle of the second angle with the z-axis is the target video content.

In combination with the second possible implementation method of the second aspect, in a fourth possible implementation method of the second aspect, the position of the user and the display device is the distance L1 between the user and the center point of the display screen, the video source includes a video shooting distance L2 and a video content viewing angle range, and L2 is the distance from the video content of the video source to the center point of the display screen; the processor is specifically used to: determine the target video content according to the line of sight angle, L1, L2, the viewing angle range and the size of the display screen.

In combination with the fourth possible implementation method of the second aspect, in a fifth possible implementation method of the second aspect, the viewing angle range includes a horizontal display viewing angle and a vertical display viewing angle, the horizontal display viewing angle is a display range angle of the video content of the video source mapped to the x-y plane, and the vertical display viewing angle is a display range angle of the video content of the video source mapped to the y-z plane; the size of the display screen includes a width D of the display screen and a height H of the display screen; the processor is specifically used to: determine the display range of the target video content on the x-y plane according to the first angle, L1, L2, the horizontal display viewing angle and the width D of the display screen; determine the display range of the target video content on the y-z plane according to the second angle, L1, L2, the vertical display viewing angle and the height H of the display screen.

In combination with the second aspect, in a sixth possible implementation of the second aspect, the processor is specifically used to: determine a first intersection point of a line connecting the user's eyes and the center point of the display screen and a video source when the user is at the said position; and use video content in the video source with the first intersection point as the center point and having the same aspect ratio as the display screen as target video content.

In combination with the second aspect, or any of the foregoing possible implementations of the second aspect, in a seventh possible implementation of the second aspect, the processor is specifically used to: determine the target video content according to the user's position and the speed of the user's movement.

In combination with the second aspect, or any of the above-mentioned possible implementation methods of the second aspect, in an eighth possible implementation method of the second aspect, the transmission interface is also used to receive the user's motion information acquired by the sensor; the processor is also used to determine the target video content based on the user's position and motion information.

In combination with the eighth possible implementation method of the second aspect, in a ninth possible implementation method of the second aspect, the processor is specifically used to: determine the first video content in the video source according to the user's position; determine the presentation rule of the pre-stored video content according to the action information, the presentation rule corresponding to the action information; determine the target video content according to the first video content and the presentation rule.

In combination with the eighth or ninth possible implementation of the second aspect, in a tenth possible implementation of the second aspect, the action information includes at least one of information of a telescope gesture, information of a gesture of pushing both hands forward, and information of a gesture of moving both hands backward.

In combination with the ninth or tenth possible implementation of the second aspect, in an eleventh possible implementation of the second aspect, the processor is specifically used to: zoom in or out the first video content with the center point of the first video content as the center to determine the target video content.

In combination with the second aspect, or any of the above possible implementations of the second aspect, in a twelfth possible implementation of the second aspect, the processor is specifically used to: determine a cropping or scaling strategy of the video source according to the user's position to obtain target video content.

In combination with the second aspect, or any of the above possible implementations of the second aspect, in a thirteenth possible implementation of the second aspect, the video source includes at least one of the following: a video with a resolution greater than or equal to 4K, a video with a field of view greater than or equal to 140 degrees, or a long strip video.

In combination with the second aspect, or any of the above-mentioned possible implementations of the second aspect, in a fourteenth possible implementation of the second aspect, the transmission interface is also used to transmit the preset second video content to the display screen so that the display screen plays the second video content.

In combination with the second aspect, or any of the foregoing possible implementations of the second aspect, in a fifteenth possible implementation of the second aspect, the device includes a display screen.

In combination with the second aspect, or any of the foregoing possible implementations of the second aspect, in a sixteenth possible implementation of the second aspect, the device includes a camera, and the camera includes an image sensor.

In a third aspect, a device for controlling the playback of video content following user motion is provided, the device comprising:

A receiving unit is used to receive the user's position obtained by a sensor when the user is watching a video; a determining unit is used to determine the target video content in the video source according to the user's position, and the target video content is the video content in the video source that matches the user's visual range at the said position; a sending unit is used to transmit the target video content to the display screen so that the display screen plays the target video content, thereby realizing that the video content follows the user's movement and improving the user experience.

In conjunction with the third aspect, in a first possible implementation manner of the third aspect, the determining unit is specifically configured to:

Determine the sight angle of the user at the position according to the user's position. The sight angle is used to reflect the sight range. The sight angle is the spatial angle of the user's sight. The user's sight is the line connecting the eyes and the center point of the display screen when the user is at the position; determine the target video content from the video source according to the sight angle, and the target video also contains video content that matches the user's sight angle.

In combination with the first possible implementation method of the third aspect, in a second possible implementation method of the third aspect, the line of sight angle includes a first angle and a second angle; the first angle is the angle between the user's line of sight and the y-axis when the line of sight is mapped to the x-y plane, and the x-y plane is the plane formed by the x-axis and the y-axis; the second angle is the angle between the user's line of sight and the z-axis when the line of sight is mapped to the y-z plane, and the y-z plane is the plane formed by the y-axis and the z-axis; wherein the y-axis is perpendicular to the plane where the display screen is located, the z-axis is perpendicular to the ground and parallel to the plane where the display screen is located, the x-axis and the z-axis form an x-z plane, the x-z plane is the plane where the display screen is located, the x-axis, the y-axis and the z-axis intersect at the origin of the coordinate axes, and the origin of the coordinate axes is the center point of the display screen.

In combination with the second possible implementation method of the third aspect, in a third possible implementation method of the third aspect, a determination unit determines the target video content from the video source according to the first angle and the second angle, and the projection of a line connecting the center point of the target video content and the center point of the display screen on the x-y plane includes an angle of the first angle with the y-axis, and the projection of a line connecting the center point of the target video content and the center point of the display screen on the y-z plane includes an angle of the second angle with the z-axis is the target video content.

In combination with the second possible implementation method of the third aspect, in a fourth possible implementation method of the third aspect, the position of the user and the display device is the distance L1 between the user and the center point of the display screen, the video source includes the video shooting distance L2 and the video content viewing angle range, L2 is the distance from the video content of the video source to the center point of the display screen; the determination unit is specifically used to: determine the target video content according to the line of sight angle, L1, L2, the viewing angle range and the size of the display screen.

In combination with the fourth possible implementation method of the third aspect, in a fifth possible implementation method of the third aspect, the viewing angle range includes a horizontal display viewing angle and a vertical display viewing angle, the horizontal display viewing angle is a display range angle of the video content of the video source mapped to the x-y plane, and the vertical display viewing angle is a display range angle of the video content of the video source mapped to the y-z plane; the size of the display screen includes the width D of the display screen and the height H of the display screen; the determination unit is specifically used to: determine the display range of the target video content on the x-y plane according to the first angle, L1, L2, the horizontal display viewing angle and the width D of the display screen; determine the display range of the target video content on the y-z plane according to the second angle, L1, L2, the vertical display viewing angle and the height H of the display screen.

In combination with the third aspect, in a sixth possible implementation of the third aspect, the determination unit is specifically used to: determine a first intersection point of a line connecting the user's eyes and the center point of the display screen and a video source when the user is at the said position; and use video content in the video source with the first intersection point as the center point and having the same aspect ratio as the display screen as target video content.

In combination with the third aspect, or any of the above possible implementations of the third aspect, in a seventh possible implementation of the third aspect, the determination unit is specifically used to: determine the target video content according to the user's position and the speed of the user's movement.

In combination with the third aspect, or any of the above-mentioned possible implementation methods of the third aspect, in an eighth possible implementation method of the third aspect, the receiving unit is further used to receive the user's motion information acquired by the sensor; the determination unit is further used to determine the target video content based on the user's position and motion information.

In combination with the eighth possible implementation method of the third aspect, in a ninth possible implementation method of the third aspect, the determination unit is specifically used to: determine the first video content in the video source according to the user's position; determine the presentation rule of the pre-stored video content according to the action information, the presentation rule corresponding to the action information; determine the target video content according to the first video content and the presentation rule.

In combination with the eighth or ninth possible implementation of the third aspect, in the tenth possible implementation of the third aspect, the action information includes at least one of information of a telescope gesture, information of a gesture of pushing both hands forward, and information of a gesture of moving both hands backward.

In combination with the ninth or tenth possible implementation method of the third aspect, in the eleventh possible implementation method of the third aspect, the determination unit is specifically used to: enlarge or reduce the first video content with the center point of the first video content as the center to determine the target video content.

In combination with the third aspect, or any of the above-mentioned possible implementation methods of the third aspect, in a twelfth possible implementation method of the third aspect, the determination unit is specifically used to: determine a cropping or scaling strategy of the video source according to the user's position to obtain target video content.

In combination with the third aspect, or any of the above possible implementations of the third aspect, in a thirteenth possible implementation of the third aspect, the video source includes at least one of the following: a video with a resolution greater than or equal to 4K, a video with a field of view greater than or equal to 140 degrees, or a long strip video.

In combination with the third aspect, or any of the above-mentioned possible implementation methods of the third aspect, in the fourteenth possible implementation method of the third aspect, the sending unit is also used to transmit the preset second video content to the display screen so that the display screen plays the second video content.

In a fourth aspect, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed on a computer or a processor, the computer or the processor executes the method in the first aspect or any possible implementation of the first aspect.

According to a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer or a processor, enables the computer or the processor to execute the method in the first aspect or any possible implementation manner of the first aspect.

Based on the provided method and device for controlling the playback of video content following the movement of the user, the relative position of the user and the display device is obtained, and the target video content is determined in the video source according to the relative position, the target video content is the video content in the video source that matches the visual range of the user at the relative position, and the target video content is played on the display screen of the display device, so that the playback video follows the movement of the user, so that the user can choose the angle and field of view of the video according to the part that the user is personally interested in, and the video content of different perspectives is presented to the audience according to the relative position of the user and the display device, thereby increasing the sense of presence and perspective of the video program, simulating the feeling of being on the scene, and improving the user experience.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1a is a schematic diagram of video content in a video source provided by an embodiment of the present application;

FIG1b is a schematic diagram of video content in another video source provided in an embodiment of the present application;

FIG2 is a schematic diagram of a visual range provided in an embodiment of the present application;

FIG3 is a schematic diagram of a user's visual range after the user moves, provided in an embodiment of the present application;

FIG4 is a schematic diagram of a viewing angle provided in an embodiment of the present application;

FIG5 is a schematic diagram of determining target video content in the horizontal direction according to an embodiment of the present application;

FIG6 is a schematic diagram of determining target video content in a vertical direction according to an embodiment of the present application;

FIG7 is a schematic diagram of another method for determining target video content in the horizontal direction provided by an embodiment of the present application;

FIG8 is a schematic diagram of another method for determining target video content in a vertical direction provided by an embodiment of the present application;

FIG9 is a schematic diagram of the structure of a display device provided in an embodiment of the present application;

FIG10 is a schematic structural diagram of another display device provided in an embodiment of the present application;

FIG11 is a schematic diagram of a flow chart of controlling the playback of video content provided in an embodiment of the present application;

FIG12 is a flow chart of a method for controlling video playback following user motion provided by an embodiment of the present application;

FIG13 is a schematic diagram of a visual range provided in an embodiment of the present application;

FIG14 is a schematic diagram of another visual range provided in an embodiment of the present application;

FIG15 is a schematic diagram of a visual range provided in an embodiment of the present application;

FIG16 is a schematic diagram of a visual range provided in an embodiment of the present application;

FIG17 is a schematic diagram of a visual range provided in an embodiment of the present application;

FIG18 is a flow chart of a method for controlling the playback of video content following user motion provided by an embodiment of the present application;

FIG19a is a schematic diagram of a visual range provided in an embodiment of the present application;

FIG19b is a schematic diagram of another visual range provided in an embodiment of the present application;

FIG20 is a schematic diagram of a device structure for controlling the playback of video content following user motion provided by an embodiment of the present application;

FIG21 is a schematic diagram of a device structure for controlling the playback of video content following user motion provided by an embodiment of the present application;

FIG. 22 is a schematic diagram of the structure of a device for controlling the playback of video content following user motion provided in an embodiment of the present application.

Detailed ways

In the development of display technology, in order to enable viewers (also known as users) to achieve an immersive experience, the development of video technology mainly has the following development trends:

1. Clearer. From high-definition (HD) to full-high-definition (FHD), to 4K resolution, and then to 8K resolution, the increase in display screen (or display) resolution brings clearer video images. However, as a display screen that always needs to be viewed at a certain distance, the further increase in resolution has increasingly limited effect on improving video effects, and the market penetration rate is relatively low.

2. More 3D. To improve the video viewing experience, there are currently a variety of 3D video solutions, including 3D display with glasses, naked-eye 3D, 360-degree video, and virtual reality (VR) glasses. These solutions require wearing special equipment, and the long-term user experience is poor, and it is difficult for multiple people to experience it together. In addition, these solutions have been available for many years, but they are still niche applications and difficult to popularize.

3. Customization. For live video programs, some operators provide functions such as video on demand, playback, and time shift, but they are essentially program on demand and lack creativity.

On the contrary, many new technologies have emerged in the field of TV games. For example, "body sensing" devices represented by Microsoft Xbox Kinect have enriched the way of playing games. "Body sensing" devices recognize the user's position and movement to control the game, getting rid of the traditional game controller. However, the application scenarios of many "body sensing" devices are limited to the field of games. There is no application combined with video, and there is no technology to use related devices to improve the video experience.

Later, the technology of controlling the TV through user actions was gradually developed. The camera captures the user's action information, such as gestures: swinging left and right, controlling the TV to change channels or increase or decrease the volume. This technology replaced the remote control operation. However, the implementation of this technology is actually no different from using a remote control. It just associates the user's actions with a few buttons on the remote control, and does not improve the user experience.

There is also a technology for watching 360-degree videos through mobile phones. This technology is to install a gyroscope in the mobile phone, and use the built-in gyroscope to sense the change in the direction of the mobile phone. When the gyroscope senses that the mobile phone changes its angle, the video area displayed on the mobile phone screen moves accordingly, so that users can watch 360 degrees from multiple angles. However, this technology must rely on the built-in gyroscope of the mobile phone. In a different scenario, such as a display device such as a TV that cannot change its direction, this technology cannot be used.

No matter how the current technology develops, the display devices such as TVs only present a flat screen to the user, similar to a painting. No matter where the user stands, the screen the user sees is always the same. Even if special video programs such as 3D videos are derived, users standing in different positions in front of the TV or other display devices still see the same program content. Because the program content or video screen seen is the program content or video screen edited and produced by the director, users cannot watch the program content or video screen of interest according to their own choices. For example, in the stadium box, users can see almost the entire stadium through the window. Users can freely choose the part of interest and control the viewing angle and field of view. Current TVs and other display devices cannot provide users with a similar experience of watching live.

Therefore, in order to enable users to have an immersive user experience and to freely choose the video content to watch, the embodiments of the present application provide a method and device for controlling the playback of video content following the user's movements, which can be applied to a variety of scenarios, such as program broadcasts (such as sports game broadcasts), advertising machines, and other scenarios.

In the embodiment of the present application, an ultra-wide-angle or panoramic camera is first used to record an ultra-high-definition video with a resolution greater than or equal to 4K, such as an ultra-high-definition video with a resolution of 4K or an ultra-high-definition video with a resolution of 8K. The ultra-high-definition video recorded by this solution can view the video content of the entire scene as shown in Figure 1a, while the traditionally recorded video can only view part of the video content of the entire scene as shown in Figure 1b.

When the ultra-high-definition video starts to be played on the display screen, the device only presents (or displays) the video content within the regular angle field of view to the user. For example, if the original video is in the 8K resolution range, only the video in the middle 4K resolution range is displayed by default, as shown in Figure 2, and the wide-angle part of the edge is not displayed.

Among them, the visual field is also called the field of view, which is the spatial range that can be seen when the eyes are fixed on a point. The visual field is divided into static visual field and dynamic visual field; static visual field refers to the spatial range that can be seen when the eyes look at the object directly in front of the person's head and eyeballs are fixed; dynamic visual field refers to the spatial range that can be seen when the eyes move. Static visual field and dynamic visual field are usually expressed in angles. In this embodiment, the visual field is called the line of sight, and the line of sight is the line connecting the user's eyes and the center point of the display screen, as shown in Figure 2.

When the device detects that the user has moved, that is, when the device detects that the user watching the video has changed the viewing position, the device determines the video content that the user is watching after the movement in the video source according to the relative position of the user device and the viewing angle of the user's line of sight and the display device, and plays the video content on the display screen of the display device. Among them, the viewing angle is used to reflect the viewing range, and the viewing angle is the spatial angle of the user's line of sight.

It should be noted that the video content within the user's field of view may not be the same video content of the video source before and after the user changes the viewing position, as shown in Figures 2 and 3, where Figure 2 can be considered as the video content watched by the user before moving the position, and Figure 3 is the field of view of the user after moving the position.

The method of following the user's movement to control the playback of video content produces an effect in which the visual range moves with the user's movement, so that the played video content moves with the user's movement, simulating the effect of on-site viewing, making the user feel immersive, and improving the user experience.

As for traditional videos, because they are edited and produced by the director, when the video is played on a display device, only part of the video content of the entire scene can be seen. Even if the user changes the viewing position, he can only see the same video content.

In one embodiment, the device includes a sensor, for example, the device includes a camera, the sensor is located in the camera of the device, and the sensor can be an image sensor, an ultrasonic sensor, or an infrared sensor in the camera, etc., for obtaining the relative position of the user and the display device. In one embodiment, the camera can obtain the position of a specific user among multiple users; in one embodiment, the camera can obtain the positions of multiple users at the same time. After the device obtains the relative position of the user through the camera, it calculates the line of sight of each user at its relative position and the line of sight of the display device; wherein the line of sight of the user is the line connecting the user's eyes and the center point of the display screen, as shown in FIG4.

In Figure 4, the center point of the display screen is called the origin, the axis passing through the origin and perpendicular to the plane where the display screen is located is called the y-axis, the axis passing through the origin, perpendicular to the ground and parallel to the plane where the display screen is located is called the z-axis, and the axis passing through the origin, parallel to the plane where the display screen is located and perpendicular to the y-axis and the z-axis respectively is called the x-axis; the plane formed by the x-axis and the y-axis is called the x-y plane (also called the horizontal plane, the direction indicated by the x-axis is called the horizontal direction), the plane formed by the y-axis and the z-axis is called the y-z plane (also called the vertical plane, the direction indicated by the z-axis is called the vertical direction), and the plane formed by the x-axis and the z-axis is called the x-z plane.

The device calculates the sight angle of the user at the relative position according to the relative position of the user, and the sight angle includes angle a (also called the first angle) and angle b (also called the second angle), as shown in Figure 4. The device calculates the angle a between the user's sight mapped to the x-y plane and the y-axis, and the angle b between the user's sight mapped to the y-z plane and the z-axis; wherein angle a is also called the horizontal angle, and angle b is also called the vertical angle.

In one embodiment, when a user moves relative to a display device, the device obtains the relative position of the user and the display device through a sensor, calculates the sight angle of the user at the relative position, determines the target video content in the video source according to the sight angle, and the target video is the video content to be played on the display screen after the user moves. The target video is played on the display screen of the display device, as shown in FIG3 , so that the user can choose the angle and field of view of watching the video according to the part that the user is personally interested in, and presents the video content of different perspectives to the audience according to the relative position of the user and the display device, thereby increasing the sense of presence and perspective of the video program, and playing the video content following the user's movement, so that the user has a sense of being at the scene when watching the video, simulating the feeling of watching on the spot, and improving the user experience.

The device determines target video content in a video source according to a sight angle, including: the device determines the target video content from the video source according to a first angle and a second angle, the projection of a line connecting a center point of the target video content and a center point of a display screen on an x-y plane having an angle of first angle with the y-axis, as shown in FIG5 , and the projection of a line connecting a center point of the target video content and a center point of the display screen on a y-z plane having an angle of second angle with the z-axis (as shown in FIG6 ) is the target video content.

FIG5 is a schematic diagram of determining the target video content in the horizontal direction. FIG6 is a schematic diagram of determining the target video content in the vertical direction. In FIG5 and FIG6, the axis passing through the center point of the display screen of the display device and perpendicular to the plane where the display screen is located is called the y-axis; the axis perpendicular to the y-axis and parallel to the plane where the display screen is located is called the z-axis; the axis perpendicular to the y-axis and the z-axis respectively and parallel to the plane where the display screen is located is called the x-axis, and the x-axis, y-axis and z-axis intersect at the center point of the display screen; the direction indicated by the x-axis is called the horizontal direction, and the direction indicated by the z-axis is called the vertical direction; the plane formed by the x-axis and the y-axis is called the x-y plane; the plane formed by the y-axis and the z-axis is called the y-z plane.

The angle between the projection of the user's line of sight on the y-z plane and the z-axis is the second angle, namely angle b. As shown in FIG5 , the angle between the line from the center point of the display screen to the center point of the target video content and the z-axis is also the second angle, namely angle b.

The angle formed by the projection of the user's line of sight on the x-y plane and the y-axis is the first angle, that is, angle a. As shown in Figure 6, the angle formed by the line connecting the center point of the display screen to the center point of the target video content and the y-axis is also the first angle, that is, angle a.

In other words, the line connecting the user's line of sight to the center point of the video content is projected onto the x-y plane, and the angle between the line and the y-axis is the first angle of the video content; and the line connecting the user's line of sight to the center point of the video content is projected onto the y-z plane, and the angle between the line and the z-axis is the second angle of the video content is determined as the target video content.

In one embodiment, as shown in FIG. 7 and FIG. 8 , the relative position of the user and the display device also includes the distance L1 from the user to the center point of the display screen, and the sight angle of the user viewing the target video content, including angle a and angle b, which are the same as angle a and angle b in FIG. 5 and FIG. 6 . The video content in the video source includes the shooting distance of the video content when shooting the video content, that is, the distance L2 from the video content in the video source to the center point of the display screen, and the video content viewing angle range, also known as the viewing angle. The viewing angle range includes the display range angle of the video content of the video source mapped to the x-y plane, that is, the horizontal angle range A (as shown in FIG. 7 ), also known as the horizontal display viewing angle, and the display range angle mapped to the y-z plane, that is, the vertical angle range B (as shown in FIG. 8 ), also known as the vertical display viewing angle. Among them, the size of the display screen includes the width of the display screen as D and the height of the display screen as H.

As shown in FIGS. 7 and 8 , the device determines the target video content in the video source according to the relative position of the user and the display device, including: the device determines the target video content according to the sight angle, the distance L1, the distance L2, the viewing angle range and the size of the display screen. The specific process is:

As shown in Figure 7, the device determines the user's field of vision in the x-y plane, that is, the horizontal field of vision range, based on the angle b, the distance L1, and the width D of the display device, and then determines the target video content in the x-y plane based on the horizontal field of vision range, the distance L2 and the horizontal angle range A.

As shown in Figure 8, the device determines the user's field of view in the y-z plane, that is, the vertical field of view range, based on the angle a, the distance L1 and the width of the display device, and then determines the target video content in the y-z plane based on the vertical field of view range, the distance L2 and the vertical angle range B.

That is, the device determines the display range of the target video content in the x-y plane according to the angle b, the distance L1, the distance L2, the horizontal display viewing angle and the width D of the display screen; the device determines the display range of the target video content in the y-z plane according to the second angle, L1, L2, the vertical display viewing angle and the height H of the display screen.

The device determines that three-dimensional video content composed of target video content in the x-y plane and target video content in the y-z plane is the target video content.

In one embodiment, as shown in FIG. 5 and FIG. 6 , the apparatus determines the target video content in the video source according to the relative position of the user and the display device, including:

Determine a first intersection point between a line connecting the user's eyes and the center point of the display screen and the video source when the user is in the relative position; and use video content in the video source with the first intersection point as the center point and having the same aspect ratio as the display screen as the target video content.

The aspect ratio refers to the ratio of the width to the height of the video content. In one embodiment, the display screen may be at a certain distance from the video source, and the aspect ratio of the target video content displayed on the display screen is the same as the aspect ratio of the target video content determined in the video source, that is, the target video content displayed on the display screen is in a certain proportional relationship with the target video content determined in the video source.

An embodiment of the present application also provides a method for controlling the playback of video content by following the user's movements, wherein the device obtains the user's motion information, such as gesture information, such as a telescope gesture, opening both hands outward, closing both hands inward, making a camera gesture with both hands, pointing one hand forward, and other gestures through a camera; and the field of view of the video is changed accordingly according to the specific motion information.

The following describes the device provided by the present application example. In one embodiment, as shown in FIG9 , the device may be a display device 100, including a camera 110, a display screen 120, a decoder 130, and a processor 140. The camera 110 may be arranged on the display device, for example, the display device may be a mobile phone, and the camera is arranged on the mobile phone. The camera 110 includes an image sensor.

In one embodiment, the camera 110 may also be used as a supporting device of the display device 100. In this case, as shown in FIG. 10, the camera 110 and the display device may be connected via a USB or other high-speed bus.

The display device 100 may be a device that supports videos with a resolution greater than or equal to 4K, such as a television that decodes and plays videos with a resolution of 4K or 8K, a smart terminal (such as a mobile phone, a tablet computer), or other device with a display and a camera.

When the video is playing, the camera 110 is used to obtain the user's position or motion information. The motion information can be gesture information, such as a telescope gesture, a gesture of opening both hands outward, a gesture of closing both hands inward, a gesture of making a camera with both hands, a gesture of pointing one hand forward, and so on.

The display screen 120 is used to display the video content to be displayed.

In one embodiment, as shown in FIG10 , the display device 100 may further include a video signal source interface 150 for receiving a video code stream from a video signal source. The display device 100 may further include a wired network interface or a wireless network module 160 for connecting to a network via a wired or wireless method, so that the display device 100 can obtain a video code stream from the network via a wired or wireless method. The display device 100 may further include a peripheral device interface 170 for connecting to a peripheral device, such as a USB flash drive, so that the display device 100 can obtain a video code stream from the peripheral device.

The decoder 130 is used to decode the video code stream received by the display device 100 from the video signal source, network or storage device. The processor 140 is used to process the video content to be displayed according to the user's position or action information obtained by the camera 110. For example, when the processor 140 determines that the user is moving, the processor 140 determines to control the field of view and controls the display screen 120 to display the video content within the field of view; for another example, the processor 140 controls the field of view according to the user's action information obtained by the camera 110, such as the action information is a telescope gesture, the processor 140 enlarges the video content, the field of view becomes narrower, and controls the display screen 120 to display the enlarged video content.

The method flow of controlling the display device 100 to play video content is shown in Figure 11. In Figure 11, the solid line is the data flow, and the dotted line is the control flow.

The display device can obtain the video code stream from the network, or the video signal source (such as a device that can receive video signals such as radar), or the storage device, and send it to the decoder, that is, step 1: the decoder obtains the video code stream; step 2: the decoder decodes the video code stream to obtain the video source; step 3: the decoder sends the video source to the memory, and the memory stores it, or it can also be stored in the video buffer. When the camera of the display device obtains the user's position or action information, step 4: the camera sends the obtained user position or action information to the processor; step 5: the processor determines the video cropping or scaling scheme according to the user's position or action information; step 6: obtain the video source from the memory; step 7: the processor crops or scales the video source according to the position information or action information, so that the cropped or scaled video content is the video content to be displayed; step 8: the processor sends the cropped or scaled video content to the display screen; step 9: display the cropped or scaled video content on the display screen.

In one embodiment, as shown in FIG. 9 , the display device includes a memory 180 , which is used to store a video code stream.

The following describes a specific solution for controlling the playback of video content following user motion provided in an embodiment of the present application.

FIG12 is a flow chart of a method for controlling the playback of video content following user motion provided by an embodiment of the present application. As shown in FIG12 , the method is executed by a device, and the method may include the following steps:

S201, obtaining a relative position between a user and a display device, where the relative position is obtained by a sensor when the display device plays a video.

In one embodiment, when playing a video, the device obtains the relative position of the user and the display device through a sensor in the camera.

In one embodiment, the sensor may be an image sensor in a camera, an ultrasonic sensor, or an infrared sensor.

S202, determining target video content in the video source according to the relative position, where the target video content is video content in the video source that matches the implementation range of the user at the relative position.

In one embodiment, as shown in Figures 5 and 6, the device determines the sight angle of the user when the user is in a relative position, the sight angle is used to reflect the sight range, the sight angle is the spatial angle of the user's sight, and the user's sight is the line connecting the eyes and the center point of the display screen when the user is in the relative position; the device determines the target video content from the video source according to the sight angle, and the target video content is the video content that matches the user's sight angle.

As shown in FIG5 and FIG6, the sight angle includes angle a and angle b. The device determines the target video content from the video source according to the sight angle, including: the device determines the target video content from the video source according to angle a and angle b, specifically: the device determines that the angle between the projection of the line connecting the center point of the video content and the center point of the display screen on the x-y plane and the y-axis is angle a, and the angle between the projection of the line connecting the center of the video content and the center point of the display screen on the y-z plane and the z-axis is angle b, and the video content is the target video content. For a specific description, please refer to the description of FIG5 and FIG6, which will not be repeated here for brevity.

In one embodiment, the device determines the target video content in the video source according to the relative position of the user and the display device, including: the device determines the target video content according to the sight angle, the distance L1, the distance L2, the viewing angle range and the size of the display screen. The specific process is: the device determines the display range of the target video content on the x-y plane according to the angle b, the distance L1, the distance L2, the horizontal display viewing angle and the width D of the display screen; the device determines the display range of the target video content on the y-z plane according to the second angle, L1, L2, the vertical display viewing angle and the height H of the display screen. For a specific description, please refer to the description of Figures 7 and 8. For the sake of brevity, it will not be repeated here.

In one embodiment, as shown in Figures 5 and 6, the device determines the target video content in the video source according to the relative position, including: the device determines the first intersection of the line connecting the user's eyes and the center point of the display screen and the video source when the user is in the relative position, and uses the video content in the video source centered on the first focus and having the same aspect ratio as the display screen as the target video content, that is, the device determines the center point of the target video content, and the target video content has the same aspect ratio as the display device.

In one embodiment, the device determines target video content from a video source according to the relative position of a user and a display device, including: the device determines a cropping strategy of the video source according to the relative position of the user and the display device, and crops the target video content from the video source according to the cropping strategy to obtain the target video content.

The video source may be at least one of a video with a resolution greater than or equal to 4K, a video with a viewing angle greater than or equal to 140 degrees, or a long video. It should be understood that the viewing angle greater than or equal to 140 degrees means that the horizontal angle range of the video source in FIG. 5 and/or the vertical angle range of the video source in FIG. 6 are both greater than or equal to 140 degrees.

S203: Play the target video content on a display screen of a display device.

In one embodiment, after the device determines the target video content, it sends the target video content to a display screen of a display device, and plays the target video content on the display screen.

In one embodiment, the video source may be at least one of a video with a resolution of 4K or higher, a video with a viewing angle of 140 degrees or higher, or a long video strip, wherein the long video strip is mainly suitable for scenes that move horizontally but remain stationary vertically, such as an advertising video on a wall.

In one embodiment, such as a sports game broadcast, the video source is a video with a resolution greater than or equal to 4K and a viewing angle greater than or equal to 140 degrees. As shown in FIG13 , when the display screen of the display device plays the video content, the video output range of the display screen displays part of the video content in the entire video source.

The device obtains the relative position of the user and the display device in real time or periodically, and accordingly determines the target video content in the video source in real time or periodically. When the device obtains the relative position of the user and the display, and the relative position of the user and the display device changes relative to that shown in FIG13, as shown in FIG14, when the user moves to the left relative to the display device, the device determines the target video content from the video source according to the relative position of the user and the display device after the movement. The manner in which the device determines the target video content is shown in FIG5, FIG6 or FIG7, FIG8, and is not repeated here for the sake of brevity. The target video content determined by the device is shown in FIG14. In FIG14, the user moves to the left relative to the display device, and the user's line of sight moves to the right relative to the user's line of sight in FIG13, so as to determine the target video content, and finally displays the determined target video content on the display screen, achieving the effect of the video content following the user's movement.

In one embodiment, the video source is a long video strip, such as a video played on an advertising machine, as shown in FIG. 15 , FIG. 16 and FIG. 17 , in which only a portion of the horizontal video content is displayed.

In Fig. 15, the dashed box is a portion of the video content displayed by default. The device obtains the relative position of the user and the display device in real time or periodically, and determines the target video content from the video source according to the relative position.

When the video source is a long strip of video, in one embodiment, the device determines the target video content according to the relative position, including: the device determines the user's line of sight angle at the relative position, including angle a and angle b, as shown in Figure 4, the device determines the target video content from the video source according to the intersection of angle a and angle b, specifically: the device determines that the projection of the line connecting the center point of the video content and the center point of the display screen on the x-y plane includes an angle a with the y-axis, and the projection of the line connecting the center point of the video content and the center point of the display screen on the y-z plane includes an angle b with the z-axis as the target video content, as shown in Figures 5 and 6. The specific description is the same as that of Figures 5 and 6, and will not be repeated here for the sake of brevity.

In one embodiment, the device determines the target video content in the video source according to the relative position of the user and the display device, including: the device determines the target video content according to the sight angle, the distance L1, the distance L2, the viewing angle range and the size of the display screen. The specific process is: the device determines the display range of the target video content on the x-y plane according to the angle b, the distance L1, the distance L2, the horizontal display viewing angle and the width D of the display screen; the device determines the display range of the target video content on the y-z plane according to the second angle, L1, L2, the vertical display viewing angle and the height H of the display screen. For a specific description, please refer to the description of Figures 7 and 8. For the sake of brevity, it will not be repeated here.

In one embodiment, the device determines the target video content according to the relative position, including: the device determines the first intersection point of the line connecting the user's eyes and the center point of the display screen and the video source when the user is in the relative position, and uses the video content in the video source with the first intersection point as the center point and the same aspect ratio as the display screen as the target video content.

In one embodiment, the device determines the target video content in the video source according to the relative position, including: the device determines the target video content according to the relative position of the user and the display device, and the speed of the user's movement.

When the device determines that the user moves relative to the display device, such as moving from the position shown in Figure 15 to the position shown in Figure 16, the device determines the target video content from the video source based on the relative positions of the user and the display device, or the device determines the target video content based on the relative positions of the user and the display device and the moving speed of the user from the position shown in Figure 15 to Figure 16, so that the played video content is based on the user's movement and is always at the closest distance to the user, thereby improving the user experience.

In one embodiment, FIG16 can also be used as a schematic diagram of video content playback before the user moves in FIG17. When the device determines the user's moving position, that is, when moving from the position shown in FIG16 to the position shown in FIG17, the device determines the target video content from the video source according to the relative position of the user and the display device, or the device determines the target video content according to the relative position of the user and the display device, and the moving speed of the user from the position of FIG16 to the position of FIG17, so that the played video content is based on the user's movement and is always at the closest distance to the user, thereby improving the user experience.

When the video source is a long video strip, it can be used for display applications, such as an advertising machine. The video screen moves with the user, which can attract the user's attention, prolong the time that the video content (such as advertising) is close to the user, and improve the commercial value.

Optionally, in one embodiment, the method further includes: in the initial state, that is, when the display screen just plays the video content, the display screen displays the preset video content (also called the second video content), as shown in FIG. 2 , the display screen displays the middle 4K resolution video content; or, as shown in FIG. 15 , the display screen displays a horizontal portion of the video content.

Optionally, in one embodiment, as shown in FIG18 , the method may further include:

S204: Obtain user action information.

In one embodiment, the sensor of the device obtains the user's motion information. The motion information may include gesture information, such as telescope gesture, hands open outward, hands closed inward, hands making a camera gesture, one hand forward, and other gestures.

S205: The apparatus determines target video content according to the relative position and action information between the user and the display device.

In one embodiment, the device determines the target video content based on the relative position and action information of the user and the display device, including: the device determines the first video content in the video source according to the relative position; the device determines the presentation rule of the stored video content according to the action information, wherein the presentation rule corresponds to the action information; the device determines the target video content according to the first video content and the presentation rule.

In one embodiment, the device determines the target video content according to the first video content and the presentation rule, including: the device zooms in or out the first video content with the center point of the first video content as the center to determine the target video content.

In one embodiment, when the action information is a telescope gesture, the device zooms in or out the currently viewed video content according to the telescope gesture according to a preset zoom-in or zoom-out ratio factor, and obtains the target video content through a zooming strategy.

For example, the video content currently being watched by the user is shown in FIG19a . When the camera of the display device obtains the user's telescope gesture, the device zooms in on the currently watched video content and the field of view becomes narrower, as shown in FIG19b .

In one embodiment, whether the device specifically zooms in or out the currently viewed video content according to the telescope gesture can be set in advance according to user needs, and is not limited in this embodiment.

For another example, when the action information is a gesture of opening both hands outward, the device enlarges the video content, narrows the field of vision, and displays the enlarged video content on the display screen.

When the action information is a gesture of closing both hands inward, the device zooms out the video content, widens the field of vision, and displays the zoomed-out video on the display screen, as well as other video content within the field of vision after the field of vision has been widened.

When the action information is a gesture of both hands making a camera posture, the device captures the current video content and displays the process of capturing the current video content on the display screen.

When the action information is a single palm forward gesture, the device keeps the currently playing video content still, records the picture of the current video content, and displays the process of recording the current video content on the display screen, and so on.

It should be noted that the display device shown in FIG. 18 obtains the user's motion information and controls the playback of video content based on the motion information. It can be used alone, or it can be combined with the display device in FIG. 12 to obtain the relative position of the user and the display device to control the playback of video content.

An embodiment of the present application provides a device for controlling the playback of video content following user movements. As shown in FIG. 20 , the device 300 includes a receiving unit 310 , a determining unit 320 , and a sending unit 330 .

The receiving unit 310 is used to receive the user's position obtained by the sensor when the user watches a video.

The determination unit 320 is configured to determine target video content in the video source according to the user's position, where the target video content is video content in the video source that matches the user's visual range at the position.

The sending unit 330 is used to transmit the target video content to the display screen so that the display screen plays the target video content, thereby realizing that the video content follows the user's movement and improving the user experience.

In one embodiment, the determination unit 320 is specifically used to: determine the sight angle of the user at the position according to the user's position, the sight angle is used to reflect the sight range, the sight angle is the spatial angle of the user's sight, and the user's sight is the line connecting the eyes and the center point of the display screen when the user is at the position; determine the target video content from the video source according to the sight angle, and the target video also contains video content that matches the user's sight angle.

The sight angle includes a first angle and a second angle. In one embodiment, the first angle is the angle between the user's sight line and the y-axis when the user's sight line is mapped to the x-y plane, and the x-y plane is the plane formed by the x-axis and the y-axis; the second angle is the angle between the user's sight line and the z-axis when the user's sight line is mapped to the y-z plane, and the y-z plane is the plane formed by the y-axis and the z-axis; wherein the y-axis is perpendicular to the plane where the display screen is located, the z-axis is perpendicular to the ground and parallel to the plane where the display screen is located, the x-axis and the z-axis form the x-z plane, and the x-z plane is the plane where the display screen is located, and the x-axis, the y-axis and the z-axis intersect at the origin of the coordinate axis, and the origin of the coordinate axis is the center point of the display screen.

In one embodiment, the determination unit 320 determines the target video content from the video source according to the first angle and the second angle, and the video content whose angle between the projection of the line connecting the center point of the target video content and the center point of the display screen on the x-y plane and the y-axis is the first angle, and whose angle between the projection of the line connecting the center point of the target video content and the center point of the display screen on the y-z plane and the z-axis is the second angle is the target video content.

In one embodiment, the position of the user and the display device is the distance L1 between the user and the center point of the display screen, the video source includes the video shooting distance L2 and the video content viewing angle range, L2 is the distance from the video content of the video source to the center point of the display screen; the determination unit 320 is specifically used to: determine the target video content according to the line of sight angle, L1, L2, the viewing angle range and the size of the display screen.

Among them, the viewing angle range includes a horizontal display viewing angle and a vertical display viewing angle, the horizontal display viewing angle is the display range angle of the video content of the video source mapped to the x-y plane, and the vertical display viewing angle is the display range angle of the video content of the video source mapped to the y-z plane; the size of the display screen includes the width D of the display screen and the height H of the display screen; the determination unit 320 is specifically used to: determine the display range of the target video content on the x-y plane according to the first angle, L1, L2, the horizontal display viewing angle and the width D of the display screen; determine the display range of the target video content on the y-z plane according to the second angle, L1, L2, the vertical display viewing angle and the height H of the display screen.

In another embodiment, the determination unit 320 is specifically used to: determine a first intersection point of a line connecting the user's eyes and the center point of the display screen and the video source when the user is at the position; and use the video content in the video source with the first intersection point as the center point and having the same aspect ratio as the display screen as the target video content.

In another embodiment, the determination unit 320 is specifically configured to determine the target video content according to the user's position and the speed of the user's movement.

In one embodiment, the receiving unit 310 is further used to receive the acquired action information of the user; the determining unit 320 is further used to determine the target video content according to the user's position and action information.

In one embodiment, the determination unit 320 is specifically used to: determine the first video content in the video source according to the user's position; determine the presentation rule of the pre-stored video content according to the action information, the presentation rule corresponds to the action information; determine the target video content according to the first video content and the presentation rule.

In one embodiment, the motion information includes at least one of information of a telescope gesture, information of a gesture of pushing both hands forward, and information of a gesture of moving both hands backward.

In one embodiment, when the camera acquires the telescope gesture information, the determination unit 320 zooms in or out the fourth video content according to the telescope gesture information.

When the camera obtains the gesture information of opening both hands outward, the determination unit 320 zooms in the video content, narrows the field of view, and displays the zoomed video content on the display screen.

When the camera obtains the gesture information of the hands closing inward, the determination unit 320 reduces the video content, widens the field of view, and displays the reduced video content on the display screen.

When the camera obtains the gesture information of both hands making a camera gesture, the determination unit 320 captures the current video content and displays the screenshot process on the display screen.

When the camera obtains the single palm forward gesture information, the determination unit 320 keeps the video content still, records the picture of the current video content, and displays the picture of the current video content on the display screen, and so on.

In one embodiment, the determination unit 320 is specifically configured to: zoom in or out the first video content with the center point of the first video content as the center, so as to determine the target video content.

In one embodiment, the determination unit 320 is specifically configured to determine a cropping or scaling strategy of a video source according to a user's position to obtain target video content.

In one embodiment, the video source includes at least one of the following: a video with a resolution greater than or equal to 4K, a video with a field of view greater than or equal to 140 degrees, or a long strip video.

In one embodiment, the sending unit 330 is further configured to transmit the preset second video content to the display screen so that the display screen plays the second video content.

The functions of each functional unit in the device can be realized through the steps executed by the device in the embodiments shown in Figures 12 and 18, and the corresponding technical effects can be achieved. Therefore, the specific working process of the device provided in the embodiment of the present invention will not be repeated here.

The embodiment of the present application provides a device for playing video content following user motion control, as shown in FIG21 , the device 400 includes a transmission interface 410 and a processor 420. It should be understood that the transmission interface 410 is an interface in the device for receiving or sending data, and the transmission interface may also be a part of the processor 420, and the transmission interface and the processor belong to the same chip.

The transmission interface 410 is used to receive the user's position acquired by the sensor when the user watches a video.

The processor 420 is used to determine target video content in the video source according to the user's position, where the target video content is the video content in the video source that matches the user's visual range when at the position.

The transmission interface 410 is also used to transmit the target video content to the display screen, so that the display screen plays the target video content, thereby realizing that the video content follows the user's movement and improving the user experience.

In one embodiment, the processor 420 is specifically used to: determine the sight angle of the user at the position according to the user's position, the sight angle is used to reflect the sight range, the sight angle is the spatial angle of the user's sight, and the user's sight is the line connecting the eyes and the center point of the display screen when the user is at the position; determine the target video content from the video source according to the sight angle, the target video content is the video content that matches the user's sight angle.

The sight angle includes a first angle and a second angle. In one embodiment, the first angle is the angle between the user's sight line and the y-axis when the user's sight line is mapped to the x-y plane, and the x-y plane is the plane formed by the x-axis and the y-axis; the second angle is the angle between the user's sight line and the z-axis when the user's sight line is mapped to the y-z plane, and the y-z plane is the plane formed by the y-axis and the z-axis; wherein the y-axis is perpendicular to the plane where the display screen is located, the z-axis is perpendicular to the ground and parallel to the plane where the display screen is located, the x-axis and the z-axis form the x-z plane, and the x-z plane is the plane where the display screen is located, and the x-axis, the y-axis and the z-axis intersect at the origin of the coordinate axis, and the origin of the coordinate axis is the center point of the display screen.

In one embodiment, the processor 420 is specifically used to: determine the target video content from the video source according to the first angle and the second angle, the video content whose angle between the projection of the center point of the target video content and the center point of the display screen on the x-y plane and the y-axis is the first angle, and whose angle between the projection of the center point of the target video content and the center point of the display screen on the y-z plane and the z-axis is the second angle is the target video content.

In one embodiment, the position of the user and the display device is the distance L1 between the user and the center point of the display screen, the video source includes the video shooting distance L2 and the video content viewing angle range, L2 is the distance from the video content of the video source to the center point of the display screen; the processor 420 is specifically used to: determine the target video content according to the line of sight angle, L1, L2, the viewing angle range and the size of the display screen.

Among them, the viewing angle range includes a horizontal display viewing angle and a vertical display viewing angle, the horizontal display viewing angle is the display range angle of the video content of the video source mapped to the x-y plane, and the vertical display viewing angle is the display range angle of the video content of the video source mapped to the y-z plane; the size of the display screen includes the width D of the display screen and the height H of the display screen; the processor 420 is specifically used to: determine the display range of the target video content on the x-y plane according to the first angle, L1, L2, the horizontal display viewing angle and the width D of the display screen; determine the display range of the target video content on the y-z plane according to the second angle, L1, L2, the vertical display viewing angle and the height H of the display screen.

In another embodiment, the processor 420 is specifically used to: determine a first intersection point of a line connecting the user's eyes and the center point of the display screen and a video source when the user is at the position; and use video content in the video source with the first intersection point as the center point and having the same aspect ratio as the display screen as target video content.

In another embodiment, the processor 420 is specifically configured to determine the target video content according to the user's position and the speed of the user's movement.

In one embodiment, the transmission interface 410 is further used to receive acquired motion information of the user; the processor 420 is further used to determine the target video content according to the position and motion information of the user.

In one embodiment, the processor 420 is specifically used to: determine the first video content in the video source according to the user's position; determine the presentation rule of the pre-stored video content according to the action information, the presentation rule corresponds to the action information; determine the target video content according to the first video content and the presentation rule.

In one embodiment, the motion information includes at least one of information of a telescope gesture, information of a gesture of pushing both hands forward, and information of a gesture of moving both hands backward.

In one embodiment, when the camera acquires the telescope gesture information, the processor 420 zooms in or out the fourth video content according to the telescope gesture information.

When the camera obtains gesture information of both hands opening outward, the processor 420 zooms in on the video content, narrows the field of view, and displays the zoomed-in video content on the display screen.

When the camera obtains the gesture information of the hands closing inward, the processor 420 reduces the video content, widens the field of view, and displays the reduced video content on the display screen.

When the camera obtains the gesture information of the two hands making a camera gesture, the processor 420 captures the current video content and displays the screenshot process on the display screen.

When the camera obtains the gesture information of a single palm facing forward, the processor 420 keeps the video content still, records the picture of the current video content, and displays the picture of the current video content on the display screen, and so on.

In one embodiment, the processor 420 is specifically configured to: zoom in or out the first video content with the center point of the first video content as the center to determine the target video content.

In one embodiment, the processor 420 is specifically configured to determine a cropping or scaling strategy of a video source according to a user's position to obtain target video content.

In one embodiment, the video source includes at least one of the following: a video with a resolution greater than or equal to 4K, a video with a field of view greater than or equal to 140 degrees, or a long strip video.

In one embodiment, the transmission interface 410 is further used to transmit the preset second video content to the display screen, so that the display screen plays the second video content.

Optionally, in one embodiment, the apparatus 400 further includes a memory 430 for storing information such as the relative position of the user and the display device.

In one embodiment, as shown in FIG. 22 , the device 400 may further include a display screen 440 .

In one embodiment, as shown in FIG. 22 , the device 400 may further include a camera 450 , which includes a sensor.

The functions of each functional device in the device can be realized through the steps performed by the device in the embodiments shown in Figures 12 and 18, and the corresponding technical effects can be achieved. Therefore, the specific working process of the device provided in the embodiments of the present application will not be repeated here.

An embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a computer or a processor, the computer or the processor executes the method of each step in Figures 12 and 18.

The embodiment of the present application also provides a computer program product containing instructions, which, when executed on a computer or a processor, enables the computer or the processor to execute the method of each step in FIG. 12 and FIG. 18 .

The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (36)

1. A method of playing video content following user motion control, the method comprising:

displaying first video content on a display screen of a display device, wherein the first video content is video content in a first area in a picture of an original video, and the first area is smaller than an area occupied by the picture of the original video;

And under the condition that the relative position between the user and the display equipment is changed, switching the first video content on the display screen into target video content, wherein the target video content is video content in a second area in the picture of the original video, and the second area is smaller than the area occupied by the picture of the original video.

2. The method of claim 1, wherein there is an overlapping region between the first region and the second region.

3. The method according to claim 1 or 2, wherein the target video content is video content in a picture of the original video that matches a line-of-sight range of the user at the relative position.

4. A method according to any of claims 1-3, wherein the relative position is obtained by a sensor when the display device plays video.

5. The method of any of claims 1-4, further comprising, prior to switching the first video content on the display screen to the target video content:

And determining the target video content from the original video according to the relative position.

6. The method of claim 5, wherein said determining the target video content in the original video based on the relative position comprises:

Determining a sight angle of the user at the relative position according to the relative position, wherein the sight angle is used for reflecting the sight range, the sight angle is a space angle of the sight of the user, and the sight of the user is a connecting line of eyes and a central point of the display screen when the user is at the relative position;

And determining the target video content from the original video according to the sight angle, wherein the target video content is the video content matching the sight angle of the user.

7. The method of claim 6, wherein the line of sight angle comprises a first angle and a second angle;

the first angle is an included angle formed by the sight line of the user and a y-axis when the sight line of the user is mapped to the x-y plane, and the x-y plane is a plane formed by the x-axis and the y-axis;

the second angle is an included angle formed by the sight line of the user and a z-axis when the sight line of the user is mapped to the y-z plane, and the y-z plane is a plane formed by the y-axis and the z-axis;

The display screen comprises a display screen, a y axis, a z axis, an x axis, a y axis, a z axis, a coordinate axis origin and a central point, wherein the y axis is perpendicular to the plane where the display screen is located, the z axis is perpendicular to the ground and parallel to the plane where the display screen is located, the x axis and the z axis form an x-z plane, the x-z plane is the plane where the display screen is located, the x axis, the y axis and the z axis intersect at the coordinate axis origin, and the coordinate axis origin is the central point of the display screen.

8. The method of claim 7, wherein said determining said target video content from said original video based on said line of sight angle comprises:

And determining the target video content from the original video according to the first angle and the second angle, wherein the included angle between the projection of the central point of the target video content and the central point of the display screen on the x-y plane and the y axis is the first angle, and the video content with the included angle between the projection of the central point of the target video content and the central point of the display screen on the y-z plane and the z axis is the second angle is the target video content.

9. The method of claim 7, wherein the relative position is a distance L1 of the user from a center point of a display screen, the original video includes a video capturing distance L2 and a video content viewing angle range, and L2 is a distance from video content of the original video to the center point of the display screen;

the determining the target video content in the original video according to the relative position comprises the following steps:

and determining the target video content according to the line of sight angle, the L1, the L2, the view angle range and the size of the display screen.

10. The method of claim 9, wherein the view range comprises a horizontal display view angle that is a display range angle at which video content of the original video is mapped onto the x-y plane and a vertical display view angle that is a display range angle at which video content of the original video is mapped onto the y-z plane; the size of the display screen comprises the width D of the display screen and the height H of the display screen;

determining the target video content according to the line of sight angle, the L1, the L2, the view angle range and the size of the display screen, wherein the determining comprises the following steps:

Determining a display range of the target video content on the x-y plane according to the first angle, the L1, the L2, the horizontal display view angle and the width D of the display screen;

And determining the display range of the target video content on the y-z plane according to the second angle, the L1, the L2, the vertical display view angle and the height H of the display screen.

11. The method of claim 5, wherein said determining said target video content in the original video based on said relative position comprises:

determining a first intersection point of a connecting line of eyes of the user and a central point of the display screen and the original video when the user is at the relative position;

And taking the video content which takes the first intersection point as a center point and has the same aspect ratio as the display screen in the original video as the target video content.

12. The method according to any one of claims 5-11, wherein said determining said target video content in the original video based on said relative position comprises:

And determining the target video content according to the relative position and the speed of the user movement.

13. The method according to any one of claims 5-12, further comprising:

Acquiring action information of the user;

And determining the target video content according to the relative position and the action information.

14. The method of claim 13, wherein said determining said target video content based on said relative position and said motion information comprises:

Determining first video content in the original video according to the relative position;

Determining a presentation rule of pre-stored video content according to the action information, wherein the presentation rule corresponds to the action information;

and determining the target video content according to the first video content and the presentation rule.

15. The method of claim 13 or 14, wherein the motion information comprises at least one of telescope gesture information, two-hand forward push gesture information, two-hand backward move gesture information.

16. The method according to claim 14 or 15, wherein said determining said target video content according to said first video content and said rendering rules comprises:

the first video content is enlarged or reduced with the center point of the first video content as the center to determine the target video content.

17. The method according to any one of claims 5-16, wherein said determining target video content in the original video based on said relative position comprises:

And determining a cropping or scaling strategy of the original video according to the relative position so as to obtain the target video content.

18. The method of any of claims 5-17, wherein the original video comprises at least one of: video with a resolution of 4K or more, video with a viewing angle of 140 degrees or more, or strip video.

19. An apparatus for playing video content following user motion control, said apparatus comprising a transmission interface;

The transmission interface is used for transmitting first video content to a display screen of display equipment so as to display the first video content on the display screen of the display equipment, wherein the first video content is video content in a first area in a picture of an original video, and the first area is smaller than the area occupied by the picture of the original video;

The transmission interface is further configured to transmit a target video content to the display screen under a condition that a relative position between a user and the display device changes, so as to switch a first video content on the display screen to the target video content, where the target video content is a video content in a second area in a picture of the original video, and the second area is smaller than an area occupied by the picture of the original video.

20. The apparatus of claim 19, wherein there is an overlapping region between the first region and the second region.

21. The apparatus according to claim 19 or 20, wherein the target video content is video content in a picture of the original video that matches a line-of-sight range of the user at the relative position.

22. The apparatus of any of claims 19-21, wherein the relative position is obtained by a sensor while the display device is playing video.

23. The apparatus according to any one of claims 19-22, wherein the apparatus further comprises:

and the processor is used for determining the target video content from the original video according to the relative position before switching the first video content on the display screen to the target video content.

24. The apparatus of claim 23, wherein the processor is specifically configured to:

determining a sight angle of the user at the position according to the relative position, wherein the sight angle is used for reflecting the sight range, the sight angle is a space angle of the sight of the user, and the sight of the user is a connecting line of eyes and a central point of the display screen when the user is at the position;

And determining the target video content from the video source according to the sight angle, wherein the target video content is the video content matched with the sight angle of the user.

25. The apparatus of claim 24, wherein the line of sight angle comprises a first angle and a second angle; the first angle is an included angle formed by the sight line of the user and a y-axis when the sight line of the user is mapped to the x-y plane, and the x-y plane is a plane formed by the x-axis and the y-axis; the second angle is an included angle formed by the sight line of the user and a z-axis when the sight line of the user is mapped to the y-z plane, and the y-z plane is a plane formed by the y-axis and the z-axis; the display screen comprises a display screen, a y axis, a z axis, an x axis, a y axis, a z axis, a coordinate axis origin and a central point, wherein the y axis is perpendicular to the plane where the display screen is located, the z axis is perpendicular to the ground and parallel to the plane where the display screen is located, the x axis and the z axis form an x-z plane, the x-z plane is the plane where the display screen is located, the x axis, the y axis and the z axis intersect at the coordinate axis origin, and the coordinate axis origin is the central point of the display screen.

26. The apparatus of claim 25, wherein the processor is specifically configured to:

And determining the target video content from the video source according to the first angle and the second angle, wherein the included angle between the projection of the central point of the target video content and the central point of the display screen on the x-y plane and the y axis is the first angle, and the included angle between the projection of the central point of the target video content and the central point of the display screen on the y-z plane and the video content of which the included angle between the central point of the target video content and the z axis is the second angle is the target video content.

27. The apparatus of claim 25, wherein the location is a distance L1 of the user from a center point of a display screen, the video source comprises a video capture distance L2 and a video content viewing angle range, and the L2 is a distance of video content of the video source from the center point of the display screen; the processor is specifically configured to:

and determining the target video content according to the line of sight angle, the L1, the L2, the view angle range and the size of the display screen.

28. The apparatus of claim 27, wherein the range of viewing angles comprises a horizontal display viewing angle that is a display range angle at which video content of the video source is mapped onto the x-y plane and a vertical display viewing angle that is a display range angle at which video content of the video source is mapped onto the y-z plane; the size of the display screen comprises the width D of the display screen and the height H of the display screen; the processor is specifically configured to:

Determining a display range of the target video content on the x-y plane according to the first angle, the L1, the L2, the horizontal display view angle and the width D of the display screen;

And determining the display range of the target video content on the y-z plane according to the second angle, the L1, the L2, the vertical display view angle and the height H of the display screen.

29. The apparatus of claim 23, wherein the processor is specifically configured to:

determining a first intersection point of a connecting line of eyes of the user and a central point of the display screen and the video source when the user is at the position;

And taking the video content which takes the first intersection point as a center point and has the same aspect ratio as the display screen in the video source as the target video content.

30. The apparatus according to any one of claims 23-29, wherein the processor is specifically configured to:

And determining the target video content according to the position and the speed of the user movement.

31. The apparatus of any one of claims 23-30, wherein,

The transmission interface is also used for receiving the acquired action information of the user;

the processor is further configured to determine the target video content based on the location and the motion information.

32. The apparatus of claim 31, wherein the processor is specifically configured to:

determining a first video content in the video source according to the location;

determining a presentation rule of pre-stored video content according to the action information, wherein the presentation rule corresponds to the action information;

and determining the target video content according to the first video content and the presentation rule.

33. The apparatus of claim 31 or 32, wherein the motion information comprises at least one of telescope gesture information, two-hand forward push gesture information, two-hand backward move gesture.

34. The apparatus of claim 32 or 33, wherein the processor is specifically configured to:

the first video content is enlarged or reduced with the center point of the first video content as the center to determine the target video content.

35. A computer readable storage medium having instructions stored therein which, when executed on a computer or processor, cause the computer or processor to perform the method of any of claims 1-18.

36. A computer program product comprising instructions which, when run on a computer or processor, cause the computer or processor to perform the method of any of claims 1-18.