CN115278193A - Panoramic video distribution method, device, device and computer storage medium - Google Patents

Abstract

The application discloses a panoramic video distribution method, a panoramic video distribution device, panoramic video distribution equipment and a computer storage medium. The method comprises the steps that under the condition that panoramic video transmission connection is established with a terminal, playing parameters of the terminal are received, wherein the playing parameters comprise position information, time information and view information; according to the position information and the time information, determining an initial panoramic image frame from the panoramic video; cutting the initially selected panoramic image frame according to the visual field information to obtain a target image frame; and distributing the target image frame to the terminal. According to the method and the device, the playing parameters of the terminal are matched with the panoramic video, the matched panoramic image frame is finally cut, the cut target image frame is distributed to the terminal, the user operation can be responded in time, and the target image frame is obtained through cutting, so that the smoothness of the distribution of the panoramic video can be guaranteed.

Description

Panoramic video distribution method, device, equipment and computer storage medium

technical field

The present application belongs to the technical field of video distribution, and in particular relates to a panoramic video distribution method, device, equipment and computer storage medium.

Background technique

Panoramic video is a kind of video that is shot in 360 degrees in all directions with a three-dimensional camera, and is the content bearer of the application of virtual reality (Virtual Reality, VR) technology. Existing panoramic video usually adopts the distribution method of field of view (FOV) transmission. The panoramic video is pre-divided into several FOV partitions, and then the panoramic background stream is assisted in splicing. It is often difficult to adapt to user operations in time, resulting in unsmooth panoramic video distribution.

Contents of the invention

Embodiments of the present application provide a panoramic video distribution method, device, device, and computer storage medium to solve the technical problem that panoramic video distribution is not smooth enough.

In the first aspect, the embodiment of the present application provides a panoramic video distribution method, the method includes:

In the case of establishing a panoramic video transmission connection with the terminal, receiving the playback parameters of the terminal, wherein the playback parameters include position information, time information and field of view information;

determining a primary panoramic image frame from the panoramic video according to the position information and the time information;

clipping the primary panoramic image frame according to the field of view information to obtain a target image frame;

Distributing the target image frame to the terminal.

In one embodiment, the panoramic video includes N panoramic sub-videos, each of which is associated with a shooting point, and each of the panoramic sub-videos includes at least one panoramic image frame, where N is greater than or an integer equal to 1,

The determining the primary panoramic image frame from the panoramic video according to the position information and the time information includes:

Determining a first panoramic sub-video matching the position information from the panoramic video according to the preset correspondence between the position information and the shooting point;

According to the preset corresponding relationship between time information and panoramic image frames, a primary panoramic image frame matching the time information is determined from at least one panoramic image frame included in the first panoramic sub-video.

In one embodiment, before the playback parameters of the receiving terminal, the method further includes:

Obtain N panoramic sub-videos in the panoramic video, and point information associated with each of the panoramic sub-videos, where the panoramic sub-videos are videos taken at each shooting point in the panoramic video;

Acquiring a frame sequence of each panoramic sub-video, the frame sequence including at least one panoramic image frame, and a playback progress corresponding to each said panoramic image frame;

The panoramic sub-video and the point information, as well as the panoramic image frame and the playback progress are stored in association.

In one embodiment, after the frame sequence of each panoramic sub-video is acquired, the method further includes:

Acquiring pixel coordinates of each pixel point of each panoramic image frame in a preset image coordinate system;

The pixel point and the pixel coordinates are stored associatively.

In one embodiment, the field of view information includes field of view coordinates and an initial field of view angle, and according to the field of view information, the primary panoramic image frame is clipped to obtain a target image frame, including:

Determining a center point from the primary panoramic image frame according to the field of view coordinates, where the center point is a pixel point whose pixel coordinates match the field of view coordinates;

Based on the center point and the initial field of view angle, the primary panoramic image frame is cropped to obtain a target image frame.

In one embodiment, the playback parameters further include a scaling ratio, and the primary panorama image frame is clipped based on the center point and the initial field of view angle to obtain a target image frame, including:

Obtaining a target viewing angle according to the scaling ratio and the initial viewing angle;

Based on the center point and the target field of view angle, the primary panoramic image frame is cropped to obtain a target image frame.

In one embodiment, the initial field of view is obtained based on the terminal according to the field of view of the human eye, and the scaling ratio is obtained based on the operation information collected by the terminal's sensor, or based on the first Enter to get.

In the second aspect, the embodiment of the present application provides a panoramic video distribution device, which includes:

A receiving module, configured to receive playback parameters of the terminal when a panoramic video transmission connection is established with the terminal, wherein the playback parameters include position information, time information, and field of view information;

A determining module, configured to determine a primary panoramic image frame from the panoramic video according to the position information and the time information;

A clipping module, configured to clip the primary panoramic image frame according to the field of view information to obtain a target image frame;

A distribution module, configured to distribute the target image frame to the terminal.

In a third aspect, an embodiment of the present application provides an electronic device, including:

a processor and a memory storing computer program instructions;

The above method is implemented when the processor executes the computer program instructions.

In a fourth aspect, an embodiment of the present application provides a computer storage medium, where computer program instructions are stored on the computer storage medium, and the above method is implemented when the computer program instructions are executed by a processor.

The panoramic video distribution method, device, device, and computer storage medium of the embodiments of the present application can receive playback parameters such as the location information, time information, and field of view information of the terminal when establishing a panoramic video transmission connection with the terminal, and first according to the location information. and time information, determine the primary panoramic image frame from the panoramic video, and then cut the primary panoramic image frame according to the field of view information to obtain the target image frame that is finally distributed to the terminal. In this embodiment of the present application, the playback parameters of the terminal are matched with the panoramic video, and finally the matched panoramic image frame is trimmed, and the trimmed target image frame is distributed to the terminal, and the target image frame can be cropped in response to user operations in a timely manner, effectively avoiding It solves the serious problem of cutting and partitioning the panoramic video and subsequent splicing, so as to ensure the smoothness of panoramic video distribution.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the accompanying drawings that need to be used in the embodiments of the present application. Additional figures can be derived from these figures.

FIG. 1 is a schematic flow diagram of a panoramic video distribution method provided by an embodiment of the present application;

Fig. 2 is a schematic diagram of the point information of the panoramic sub-video in the embodiment of the present application;

Fig. 3 is a schematic diagram of the coordinates of each pixel of the panoramic image frame in the embodiment of the present application;

FIG. 4 is a schematic diagram of a target image frame in an embodiment of the present application;

FIG. 5 is a schematic structural diagram of a panoramic video distribution device provided in another embodiment of the present application;

Fig. 6 is a schematic structural diagram of an electronic device provided by another embodiment of the present application.

Detailed ways

The characteristics and exemplary embodiments of various aspects of the application will be described in detail below. In order to make the purpose, technical solution and advantages of the application clearer, the application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only intended to explain the present application rather than limit the present application. It will be apparent to one skilled in the art that the present application may be practiced without some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or order between them. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the statement "comprising..." does not exclude the presence of additional same elements in the process, method, article or device comprising said element.

In order to solve the problems in the prior art, embodiments of the present application provide a panoramic video distribution method, device, device, and computer storage medium. The panoramic video distribution method provided by the embodiment of the present application is firstly introduced below.

Fig. 1 shows a schematic flowchart of a panoramic video distribution method provided by an embodiment of the present application. The panoramic video distribution method can be applied to any scene involving panoramic video distribution, for example, it can be applied to a scene where a terminal such as a smart phone, a computer or a smart TV requests to play a panoramic video, and can also be applied to a virtual reality (Virtual Reality, VR) The scene is not specifically limited here; but to simplify the description, the following description will mainly take the VR scene as an example.

As shown in Figure 1, the panoramic video distribution method includes:

Step S101, in the case of establishing a panoramic video transmission connection with the terminal, receiving the playback parameters of the terminal, wherein the playback parameters include position information, time information and field of view information;

Step S102, according to the location information and time information, determine the primary selected panoramic image frame from the panoramic video;

Step S103, according to the field of view information, crop the primary panoramic image frame to obtain the target image frame;

Step S104, distributing the target image frame to the terminal.

In this embodiment, establishing a panoramic video transmission connection with the terminal in step S101 may mean that the server interacts with the terminal, the terminal requests to play the panoramic video, and the server distributes the panoramic video to the terminal in response to the terminal's request.

Wherein, the server may be any device that stores panoramic video-related data, for example, it may be a content delivery network (Content Delivery Network, CDN) device. To simplify the description, the server is a CDN device as an example for illustration below.

Wherein, the terminal may be a device capable of processing panoramic video data, such as a smart phone, a computer, a smart TV, a VR head-mounted display device, and the like. Specifically, the terminal may include a sensor, a remote control, a button or a mouse, etc., which may be used to obtain playback parameters such as position information and field of view information; Get playback time.

Playing parameters may include position information, time information and field of view information. The location information may be the spatial location of the user acquired by the control device. For example, the user wears a VR headset and enters the first space of the VR scene. At this time, the location information corresponds to the first space, that is, the playback parameter indicates the required The panoramic video data associated with the first space is played.

The time information can be a virtual control of the time dimension of the panoramic video, for example, it can be the playback time axis of the panoramic video, and the playback time axis of the panoramic video can also be changed according to operations such as fast forward and rewind. At this time, a specific panoramic image frame may be mapped according to the playback time axis of the panoramic video, that is, the playback parameter indicates that a panoramic image frame matching the current playback time axis needs to be played.

The visual field information may be the user's visual field range, that is, the area that the user's visual field can see. For example, the user looks forward in the first space of the VR space. At this time, with the user's orientation as a reference, the user's visual range may be a partial area directly in front of the first space.

In step S102, the primary panoramic image frame may be determined from the panoramic video according to the location information and time information. As mentioned above, the panoramic video may include one or more panoramic sub-videos with different spatial associations.

If there is one panorama sub-video, it means that the panorama video has only one space, and the position information corresponds to this space, so the panorama sub-video matches the position information. If there are multiple panoramic sub-videos, the space to which the user belongs can be determined according to the location information, and the panoramic sub-video associated with the space is the panoramic sub-video matching the location information.

The first panoramic sub-video matching the position information may be determined first according to the position information, and then the primary panorama image frame matching the time information in the first panoramic sub-video is determined according to the time information. It is also possible to first determine all the panoramic image frames in the panoramic video that match the time information according to the time information, and then match all the panoramic image frames according to the position information to obtain the panoramic image frames corresponding to the space to which the position information belongs as the primary panoramic image frame .

In step S103, the primary panoramic image frame may be cropped according to the field of view information to obtain a target image frame. In this embodiment, after the primary panoramic image frame is determined, the primary panoramic image frame is not directly distributed to the terminal at this time, and the primary panoramic image frame can be cut according to the field of view information, that is, the image frame area within the user's visual range Cropped from the panoramic image frame, the image frame area is used as the target image frame.

In step S104, the target image frame is distributed to the terminal. The CDN device can directly distribute the target image frame to the terminal, so that the target image frame can be displayed on the display screen of the terminal.

The panoramic video distribution method provided by the embodiment of the present application can receive playback parameters such as location information, time information, and field of view information of the terminal when establishing a panoramic video transmission connection with the terminal, and first select the panoramic video from the panoramic video according to the location information and time information. The primary panoramic image frame is determined, and then the primary panoramic image frame is clipped according to the field of view information to obtain a target image frame that is finally distributed to the terminal. In this embodiment of the present application, the playback parameters of the terminal are matched with the panoramic video, and finally the matched panoramic image frame is trimmed, and the trimmed target image frame is distributed to the terminal, and the target image frame can be cropped in response to user operations in a timely manner, effectively avoiding It solves the serious problem of cutting and partitioning the panoramic video and subsequent splicing, so as to ensure the smoothness of panoramic video distribution.

Optionally, in one embodiment, the panoramic video can include N panoramic sub-videos, each of which can be associated with a shooting point, and each panoramic sub-video includes at least one panoramic image frame, where N is greater than or an integer equal to 1,

Step S102, according to the location information and time information, determine the primary panoramic image frame from the panoramic video, which may include:

According to the corresponding relationship between the preset position information and the shooting point, determine the first panoramic sub-video matching the position information from the panoramic video;

According to the preset corresponding relationship between time information and panoramic image frames, a primary panoramic image frame matching the time information is determined from at least one panoramic image frame included in the first panoramic sub-video.

In this embodiment, the panoramic video may include N panoramic sub-videos, where N is an integer greater than or equal to 1, and each panoramic sub-video may be associated with a shooting point. For example, a panoramic video may be composed of panoramic sub-videos in N different spaces, and specifically, each space may be shot by a shooting point.

One of the shooting points may include a plurality of surrounding light-sensitive sensors, for example, one shooting point may include three 120° wide-angle cameras for shooting, so as to obtain a panoramic sub-video of the space. A shooting point can also be a rotating camera, or other equipment capable of shooting panoramic video.

Each panoramic sub-video includes at least one panoramic image frame, for example, the panoramic sub-video may include "panoramic image frame 1", "panoramic image frame 2", "panoramic image frame 3" ... "panoramic image frame n".

According to the preset corresponding relationship between the position information and the shooting point, the first panoramic sub-video matching the position information is determined from the panoramic video. The preset correspondence between location information and shooting points may be that one shooting point corresponds to one piece of location information, and the location information may be a specific location point, or within a location range.

According to the correspondence between the position information and the shooting point, the shooting point matching the position information can be determined, and the panoramic sub-video associated with the shooting point is the first panoramic sub-video matching the position information.

According to the preset corresponding relationship between time information and panoramic image frames, a primary panoramic image frame matching the time information is determined from at least one panoramic image frame included in the first panoramic sub-video. The preset correspondence between time information and panoramic image frames may be that one panoramic image frame corresponds to one piece of time information.

For example, "panoramic image frame 1" corresponds to time T1, "panoramic image frame 2" corresponds to time T2, "panoramic image frame 3" corresponds to time T3... "panoramic image frame n" corresponds to time Tn.

According to the corresponding relationship between the time information and the panoramic image frame, the "panoramic image frame 1", "panoramic image frame 2", "panoramic image frame 3" ... "panoramic image frame" included in the first panoramic sub-video determined from the previous step In the image frame n", the primary panoramic image frame matching the time information is determined.

For example, if the time information is T3, then the corresponding "panoramic image frame 3" may be determined as the primary panorama image frame matching the time information.

In this embodiment, the matching first panoramic sub-video is first determined according to the position information, and then the primary panoramic image frame is determined from the first panoramic sub-video according to the time information, and the target image is obtained by cutting according to the determined primary panoramic image frame frame, and then distribute the target image frame to the terminal. It can respond to user operations in a timely manner according to playback parameters, and obtain target image frames, thereby ensuring the smoothness of panoramic video distribution.

Optionally, in one embodiment, before receiving the playback parameters of the terminal, the panoramic video distribution method may further include:

Obtain N panoramic sub-videos in the panoramic video, and point information associated with each panoramic sub-video, where the panoramic sub-video is a video taken at each shooting point in the panoramic video;

Obtain the frame sequence of each panoramic sub-video, the frame sequence includes at least one panoramic image frame, and the playback progress corresponding to each panoramic image frame;

Associated storage of panoramic sub-video and point information, as well as panoramic image frames and playback progress.

In this embodiment, N panoramic sub-videos in the panoramic video and point information associated with each panoramic sub-video may be acquired. For example, a space coordinate system can be established for panoramic video, as shown in FIG. 2 , the coordinate origin of the space coordinate system can be a certain shooting point, so as to obtain the coordinates of other shooting points in the space coordinate system. The coordinates of the shooting point in the spatial coordinate system may be used as point information associated with the panorama sub-video.

Alternatively, it is also possible to obtain the panoramic sub-video shot at each shooting point, and then use the coordinates of the central point of the panoramic sub-video in the spatial coordinate system as the point information associated with the panoramic sub-video.

In order to simplify the description, the coordinates of the shooting point in the spatial coordinate system are used as the point information associated with the panoramic sub-video as an example for description.

For example, the coordinates of "shooting point 1" in the space coordinate system are (x1, y1, z1), then the point information associated with the panoramic sub-video shot at "shooting point 1" is (x1, y1, z1); The coordinates of "shooting point 2" in the space coordinate system are (x2, y2, z2), then the point information associated with the panoramic sub-video shot at "shooting point 2" is (x2, y2, z2)..." The coordinates of "shooting point n" in the space coordinate system are (xn, yn, zn), then the point information associated with the panoramic sub-video shot at "shooting point n" is (xn, yn, zn).

The frame sequence of each panoramic sub-video may also be obtained, the frame sequence includes at least one panoramic image frame, and the playback progress corresponding to each panoramic image frame. For example, the frame sequence of each panorama sub-video may be acquired according to video playing time or sequence of video editing, and the frame sequence may include at least one panorama image frame.

Wherein, t may be used to represent the playback progress corresponding to each panoramic image frame. For example, t1 may represent the panoramic image frame at time t1, or may directly represent the t1th panoramic image frame.

Associated storage of panoramic sub-video and point information, as well as panoramic image frames and playback progress. Specifically, the panoramic sub-video and point information corresponding to the panoramic video may be stored, and on this basis, each panoramic image frame in the panoramic sub-video and the playback progress corresponding to each panoramic image frame may be stored.

For example, the point information and playback progress of the panorama sub-video taken at "shooting point 1" at the moment t1 are stored as (x1, y1, z1, t1); the panorama sub-video shot at "shooting point 1" The point information and playback progress of the panoramic image frame at the moment t2 are stored as (x1, y1, z1, t2); The progress is stored as (x2, y2, z2, t1)... The point information and playback progress of the panoramic image frame of the panoramic sub-video shot at "shooting point n" at time tn are stored as (xn, yn, zn, tn) .

In the embodiment of the present application, after obtaining the point information associated with the panoramic sub-video and the playback progress corresponding to the panoramic image frame, the panoramic sub-video and the point information, as well as the panoramic image frame and the playback progress can be associated and stored, and the panoramic video Indexing is performed to facilitate the subsequent rapid search of the corresponding primary panoramic image frame according to the index information, thereby ensuring the smoothness of panoramic video distribution. The index information can be used to indicate the above point information and playback progress.

In an example, the location information of the terminal can be based on user operations or based on a preset location relationship, to obtain an initial location coordinate (X, Y, Z), and the initial location coordinate (X, Y, Z) matches an initial point information, for example, the matching initial point information may be (x2, y2, z2).

The location information may change according to the user operations collected by the terminal's sensors, for example, the user's actions collected by the gyroscope in the VR device, the user moves the location, and thus the location coordinates change. It can also change the position coordinates according to virtual space actions such as "forward, backward, up, down" input by remote control, mouse, etc. If the position coordinates change, the matching point information may also change.

The time information of the terminal may also be based on a preset time axis, and the time T is obtained, and the time T matches a playback progress, for example, the matched playback progress may be t1. The time information can also change according to commands such as "fast forward and fast rewind" input by the remote control, mouse, etc., and the matching playback progress may also change.

For example, if the terminal's initial position information and time information (X, Y, Z, T) match point information and playback progress (x2, y2, z2, t1), then (x2, y2, z2, t1) The corresponding panoramic image frame is used as the primary panoramic image frame. With the change of position and time axis, at the next moment, the point information and playback progress matched by the terminal's position information and time information (X, Y, Z, T) become (x1, y1, z1, t2 ), then the panoramic image frame corresponding to (x1, y1, z1, t2) can be used as the primary panoramic image frame.

Optionally, in one embodiment, after obtaining the frame sequence of each panoramic sub-video, the panoramic video distribution method may also include:

Obtain the pixel coordinates of each pixel point of each panoramic image frame in the preset image coordinate system;

Associatively store pixel points and pixel coordinates.

In this embodiment, the pixel coordinates of each pixel point in each panoramic image frame in the preset image coordinate system may also be obtained. Wherein, the preset image coordinate system may be a two-dimensional coordinate system established for each panoramic image frame.

As shown in Figure 3, the starting point of the physical orientation of the shooting point, such as the north direction, can be selected to establish the image coordinate system. Of course, the starting point of other directions can also be selected to establish the image coordinate system, which is not limited here. In this way, the panoramic image frame is flattened in the image coordinate system, and the pixel coordinates of each pixel point in the image coordinate system are obtained.

For example, the pixel coordinates of the first pixel point can be expressed as (a1, b1), the pixel coordinates of the second pixel point can be expressed as (a2, b2)... the pixel coordinates of the nth pixel point can be expressed as (an ,bn).

Associatively store pixel points and pixel coordinates. Specifically, each pixel point in each panoramic image frame in each panoramic sub-video may be associated and stored in combination with point information, playback progress, and pixel coordinates.

For example, the point information, playback progress and pixel coordinates of the first pixel in the panoramic image frame of the panorama sub-video captured by "shooting point 1" at time t1 are stored as (x1, y1, z1, t1, a1, b1); the point information, playback progress and pixel coordinates of the second pixel point in the panorama image frame of the panorama sub-video taken by "shooting point 2" at time t1 are stored as (x2, y2, z2, t1, a2 , b2)... The point information, playback progress and pixel coordinates of the nth pixel in the panoramic image frame of the panorama sub-video captured by "shooting point n" at time tn are stored as (xn, yn, zn, tn , an, bn).

The embodiment of the present application can also obtain the pixel coordinates of each pixel point in the preset image coordinate system, and associate and store each pixel point and pixel coordinates, and further index the panoramic video, which further facilitates the follow-up according to the index The information quickly finds the specific pixel points in the corresponding primary panoramic image frame, and according to the pixel point, the primary panoramic image frame can be quickly processed, such as cropping to obtain the target image frame, thereby further ensuring the smoothness of panoramic video distribution, and Effectively save transmission resources in the process of panoramic video distribution. The index information can also be used to indicate the above-mentioned pixel coordinates.

Optionally, the field of view information may include field of view coordinates and an initial field of view angle. According to the field of view information, the primary panoramic image frame is clipped to obtain the target image frame, which may include:

According to the coordinates of the field of view, a center point is determined from the primary panoramic image frame, and the center point is a pixel point whose pixel coordinates match the coordinates of the field of view;

Based on the center point and the initial field of view, the primary panoramic image frame is cropped to obtain the target image frame.

In this embodiment, the coordinates of the field of view can be used to represent the focal point of the human eye. For example, the focus point of the human eye can be tracked in real time according to the eye tracking sensor, and the coordinates (A, B) of the focus point can be saved as the coordinates of the field of view.

According to the coordinates of the field of view, a center point is determined from the primary panoramic image frame, and the center point may be a pixel point whose pixel coordinates match the coordinates of the field of view. For example, if the pixel coordinates matched by field of view coordinates (A, B) are (a3, b3), then the pixel point corresponding to the pixel coordinates in the primary panorama image frame (a3, b3) can be used as the center point, that is, the primary selection The third pixel in the panoramic image frame is the center point.

The initial field of view F can be preset according to the actual situation. For example, generally speaking, the perception of the human eye’s field of view is generally less than 120° in the horizontal direction and less than 60° in the vertical direction. The initial field of view can be preset as horizontal 120°, 60° vertically.

Or, in order to adapt to related operations such as quick head shaking and nodding, etc., the range of the initial field of view can be expanded in combination with the performance of the terminal. For example, the initial field of view can be preset to 150° horizontally and 80° vertically, so When the user is shaking his head rapidly, nodding his head, etc., the initial field of view is still within the range of the human eye.

In addition, the initial viewing angle may also be set by the terminal based on capturing differences in viewing ranges of different users. The initial viewing angle may also include a resolution, and the resolution may be a preset fixed value, or may be set to a different value based on user selection.

Based on the center point and the initial field of view, the primary panoramic image frame is cropped to obtain the target image frame. To simplify the description, take the initial field of view of 120° horizontally and 60° vertically as an example, take the center point of the primary panoramic image frame as the cropping center, crop 120° horizontally and 60° vertically, and the obtained area is the target image frame .

The embodiment of the present application can quickly obtain the center point and cropping range of the primary panoramic image frame according to the field of view coordinates and the initial field of view angle, and distribute the target image frame to the terminal after clipping, thereby ensuring the smoothness of panoramic video distribution. And effectively save the transmission resources in the panoramic video distribution process.

In one example, the rotational movement of the user's real head or body can be grasped by the gyroscope in the VR device worn by the user, and the spatial transformation of the user's field of view can be represented forward or backward, resulting in a change in the coordinates of the field of view . Or, according to virtual space actions such as "rotate left and right, move up and down" input by a mouse, a remote control, etc., a field of view transformation under the user's virtual position can be formed, thereby resulting in a change of the coordinates of the field of view. When the coordinates of the field of view change, its center point in the primary panoramic image frame may also change accordingly.

Optionally, in one embodiment, the playback parameters may also include scaling, based on the center point and the initial field of view, the primary panorama image frame is cropped to obtain the target image frame, which may include:

According to the zoom ratio and the initial field of view, the target field of view is obtained;

Based on the center point and the target field of view, the primary panoramic image frame is cropped to obtain the target image frame.

In this embodiment, the playback parameter may also include a scaling ratio, and the target viewing angle may be obtained according to the scaling ratio and the initial viewing angle. For example, when the zoom ratio is a magnification factor, in the primary panorama image frame, the area seen by the user becomes smaller, that is, the target field of view is smaller than the initial field of view; otherwise, when the zoom ratio is a zoom factor In some cases, in the primary panoramic image frame, the area seen by the user becomes larger, that is, the target field of view is larger than the initial field of view.

As shown in FIG. 4 , for ease of understanding, the target image frame is described as a rectangle. When the user chooses to zoom in by 2 times, the initial field of view will be reduced by 2 times both horizontally and vertically, that is, the target field of view can be 1/4 of the initial field of view.

Based on the center point and the target field of view, the primary panoramic image frame is cropped to obtain the target image frame. Taking the target field of view as 1/4 of the initial field of view as an example, the center point of the primary panoramic image frame is used as the cropping center, and the target field of view is used as the cropping range to obtain the target image frame. The target image frame is 1/4 of the clipping range area with the initial field of view.

In this embodiment, the final target image frame can be acquired in combination with the scaling ratio, so as to ensure the smoothness of panoramic video distribution and effectively save transmission resources in the panoramic video distribution process.

Optionally, in an embodiment, the initial field of view may be obtained based on the terminal according to the field of view of the human eye, and the scaling ratio may be obtained based on the operation information collected by the terminal's sensor, or may be obtained based on the first input received by the terminal.

In this embodiment, the initial viewing angle may be obtained based on the terminal according to the viewing range of human eyes. For example, after a user wears a VR device, the VR device can analyze the user's human eye field of view through an eye tracking sensor or other sensors, so as to set an initial field of view based on the analyzed data.

Among them, the angle range obtained by analysis can be directly used as the initial field of view angle, and the angle range obtained by analysis can also be added with a preset angle value as the initial field of view angle to eliminate possible error bands in the analysis coming impact.

The zoom ratio can be obtained based on the operation information collected by the terminal's sensors. For example, the gyroscope on the VR device senses the user's "zoom in, zoom out" action. Or it may be obtained based on the first input received by the terminal, for example, the user may perform a zoom-out or zoom-in input through a remote controller or a mouse.

In one embodiment, after the terminal is turned on, the user selects the panoramic video to be played, and generates a corresponding initial playback uniform resource locator (Uniform Resource Locator, URL), which is represented by URL1 below, and URL1 only represents the selection of specific panoramic video content, not Characterizes the playback parameters of the terminal.

The terminal plays parameters according to the user's initial position, the user's comprehensive field of view angle, the user's field of view coordinates, zoom ratio, and time dimension (that is, position information, time information, field of view information, zoom ratio, etc., where the field of view information includes field of view coordinates and field of view angle ), namely (X, Y, Z, T, A, B, F, L) and other information locally process URL1 to form URL2 (X, Y, Z, T, A, B, F, L), where URL2 (X, Y, Z, T, A, B, F, L) represent playback parameters of the terminal.

The terminal sends a play request to the CDN system according to URL2(X, Y, Z, T, A, B, F, L). After being scheduled by its own scheduling mechanism, the CDN system finally directs the playback request to a specific CDN device that provides services for the terminal. The specific scheduling process of the CDN system can be realized by the scheduling method of the existing video storage server system, which will not be repeated here.

After the CDN device receives the playback request URL2 (X, Y, Z, T, A, B, F, L), it first

(X, Y, Z, T) to find the matching index information (x, y, z, t); find the corresponding primary panoramic image frame according to the (x, y, z, t) information (that is, according to the position information and Time information, determine the primary panorama image frame from the panorama video).

The CDN device further finds the corresponding center point (a, b) in the primary panorama image frame according to the playback request URL2 (A, B, F, L), combines F and L to perform distribution picture calculation, and completes the primary panorama The clipping of the image frame (that is, according to the field of view coordinates, determine the center point from the primary panoramic image frame, and the center point is the pixel point that the pixel coordinates match with the field of view coordinates; based on the center point and the initial field of view angle, the primary panoramic image The image frame is cropped to obtain the target image frame).

After the CDN device completes the material processing corresponding to the target, it only distributes the cropped target image frame to the terminal for display (that is, distributes the target image frame to the terminal).

FIG. 5 shows a schematic structural diagram of a panoramic video distribution device provided by another embodiment of the present application. For ease of description, only parts related to the embodiment of the present application are shown.

With reference to Fig. 5, panoramic video distribution device comprises:

The receiving module is configured to receive playback parameters of the terminal when a panoramic video transmission connection is established with the terminal, wherein the playback parameters include position information, time information and field of view information;

A determination module is used to determine the primary panoramic image frame from the panoramic video according to the position information and time information;

The cutting module is used to cut the primary panoramic image frame according to the field of view information to obtain the target image frame;

The distribution module is used for distributing the target image frame to the terminal.

Optionally, the panoramic video includes N panoramic sub-videos, and each panoramic sub-video is associated with a shooting point, and each panoramic sub-video includes at least one panoramic image frame, wherein N is an integer greater than or equal to 1, and the determination module can include:

The first determination unit is configured to determine the first panoramic sub-video matching the position information from the panoramic video according to the preset correspondence between the position information and the shooting point;

The second determining unit is configured to determine a primary panoramic image frame matching the time information from at least one panoramic image frame included in the first panoramic sub-video according to a preset correspondence between time information and panoramic image frames.

Optionally, the device may also include:

The first obtaining module is used to obtain N panoramic sub-videos in the panoramic video, and point information associated with each panoramic sub-video, where the panoramic sub-video is a video taken at each shooting point in the panoramic video;

The second obtaining module is used to obtain the frame sequence of each panoramic sub-video, the frame sequence includes at least one panoramic image frame, and the playback progress corresponding to each panoramic image frame;

The first storage module is used for associating storage of panoramic sub-video and point information, as well as panoramic image frames and playback progress.

Optionally, the device may also include:

The third obtaining module is used to obtain the pixel coordinates of each pixel point of each panoramic image frame in the preset image coordinate system;

The second storage module is used for associatively storing pixel points and pixel coordinates.

Optionally, the field of view information includes field of view coordinates and initial field of view angle, and the cropping module may include:

The third determining unit is used to determine a center point from the primary panoramic image frame according to the coordinates of the field of view, and the center point is a pixel point whose pixel coordinates match the coordinates of the field of view;

The clipping unit is configured to clip the primary panoramic image frame based on the center point and the initial field of view to obtain the target image frame.

Optionally, the playback parameters also include scaling, and the cropping unit may include:

Acquiring subunits, used to obtain the target field of view according to the zoom ratio and the initial field of view;

The cropping subunit is configured to crop the primary panoramic image frame based on the center point and the target field of view to obtain the target image frame.

Optionally, the initial field of view is obtained based on the terminal based on the field of view of the human eye, and the scaling ratio is obtained based on the operation information collected by the terminal's sensor, or based on the first input received by the terminal.

It should be noted that the information interaction and execution process between the above-mentioned devices/units are based on the same concept as the method embodiment of the present application, and are devices corresponding to the above-mentioned panoramic video distribution method. All implementation methods in the above-mentioned method embodiment are In the embodiments applicable to the device, its specific functions and technical effects brought by it can be referred to the method embodiments for details, and will not be repeated here.

Those skilled in the art can clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used for illustration. In practical applications, the above-mentioned functions can be assigned to different functional units, Completion of modules means that the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated into one processing unit, or each unit may exist separately physically, or two or more units may be integrated into one unit, and the above-mentioned integrated units may adopt hardware It can also be implemented in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the present application. For the specific working process of the units and modules in the above system, reference may be made to the corresponding process in the foregoing method embodiments, and details will not be repeated here.

FIG. 6 shows a schematic diagram of a hardware structure of an electronic device provided by another embodiment of the present application.

The device may comprise a processor 601 and a memory 602 storing computer program instructions.

When the processor 601 executes the computer program, the steps in any of the foregoing method embodiments are implemented.

Exemplarily, the computer program can be divided into one or more modules/units, and one or more modules/units are stored in the memory 602 and executed by the processor 601 to complete the present application. One or more modules/units may be a series of computer program instruction segments capable of accomplishing specific functions, and the instruction segments are used to describe the execution process of the computer program in the device.

Specifically, the processor 601 may include a central processing unit (CPU), or an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or may be configured to implement one or more integrated circuits in the embodiments of the present application.

Memory 602 may include mass storage for data or instructions. By way of example and not limitation, memory 602 may include a hard disk drive (Hard Disk Drive, HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (Universal Serial Bus, USB) drive or two or more Combinations of multiple of the above. Storage 602 may include removable or non-removable (or fixed) media, where appropriate. Under appropriate circumstances, the storage 602 can be inside or outside the comprehensive gateway disaster recovery device. In a particular embodiment, memory 602 is a non-volatile solid-state memory.

Memory may include read only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical/tangible memory storage devices. Thus, in general, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software comprising computer-executable instructions, and when the software is executed (e.g., by one or multiple processors) operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

The processor 601 reads and executes the computer program instructions stored in the memory 602 to implement any method in the foregoing embodiments.

In one example, the electronic device may further include a communication interface 603 and a bus 610 . Wherein, the processor 601, the memory 602, and the communication interface 603 are connected through the bus 610 and complete mutual communication.

The communication interface 603 is mainly used to implement communication between modules, devices, units and/or devices in the embodiments of the present application.

The bus 610 includes hardware, software or both, and couples the components of the online data traffic charging device to each other. By way of example and not limitation, the bus may include Accelerated Graphics Port (AGP) or other graphics bus, Enhanced Industry Standard Architecture (EISA) bus, Front Side Bus (FSB), HyperTransport (HT) interconnect, Industry Standard Architecture (ISA) Bus, Infiniband Interconnect, Low Pin Count (LPC) Bus, Memory Bus, Micro Channel Architecture (MCA) Bus, Peripheral Component Interconnect (PCI) Bus, PCI-Express (PCI-X) Bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local (VLB) bus or other suitable bus or a combination of two or more of these. Bus 610 may comprise one or more buses, where appropriate. Although the embodiments of this application describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.

In addition, in combination with the methods in the foregoing embodiments, the embodiments of the present application may provide a computer storage medium for implementation. Computer program instructions are stored on the computer storage medium; when the computer program instructions are executed by a processor, any method in the above-mentioned embodiments is implemented.

It is to be understood that the application is not limited to the specific configurations and processes described above and shown in the figures. For conciseness, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art may make various changes, modifications and additions, or change the order of the steps after understanding the spirit of the present application.

The functional modules shown in the above structural block diagrams may be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an application specific integrated circuit (ASIC), suitable firmware, a plug-in, a function card, or the like. When implemented in software, the elements of the present application are the programs or code segments employed to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted over transmission media or communication links by data signals carried in carrier waves. "Machine-readable medium" may include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and the like. Code segments may be downloaded via a computer network such as the Internet, an Intranet, or the like.

It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps may be performed in the order mentioned in the embodiment, or may be different from the order in the embodiment, or several steps may be performed simultaneously.

Aspects of the present disclosure are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart and/or block diagrams, and combinations of blocks in the flowchart and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that execution of these instructions via the processor of the computer or other programmable data processing apparatus enables Implementation of the functions/actions specified in one or more blocks of the flowchart and/or block diagrams. Such processors may be, but are not limited to, general purpose processors, special purpose processors, application specific processors, or field programmable logic circuits. It can also be understood that each block in the block diagrams and/or flowcharts and combinations of blocks in the block diagrams and/or flowcharts can also be realized by dedicated hardware for performing specified functions or actions, or can be implemented by dedicated hardware and combination of computer instructions.

The above is only a specific implementation of the present application, and those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described systems, modules and units can refer to the foregoing method embodiments The corresponding process in , will not be repeated here. It should be understood that the protection scope of the present application is not limited thereto, and any person familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the application, and these modifications or replacements should cover all Within the protection scope of this application.

Claims (10)

1. A panoramic video distribution method, characterized in that, comprising: In the case of establishing a panoramic video transmission connection with the terminal, receiving the playback parameters of the terminal, wherein the playback parameters include position information, time information and field of view information; determining a primary panoramic image frame from the panoramic video according to the position information and the time information; clipping the primary panoramic image frame according to the field of view information to obtain a target image frame; Distributing the target image frame to the terminal. 2. The method according to claim 1, wherein the panoramic video includes N panoramic sub-videos, each of which is associated with a shooting point, and each of the panoramic sub-videos includes at least one Panoramic image frame, where N is an integer greater than or equal to 1, The determining the primary panoramic image frame from the panoramic video according to the position information and the time information includes: Determining a first panoramic sub-video matching the position information from the panoramic video according to the preset correspondence between the position information and the shooting point; According to the preset corresponding relationship between time information and panoramic image frames, a primary panoramic image frame matching the time information is determined from at least one panoramic image frame included in the first panoramic sub-video. 3. The method according to claim 2, wherein before the playback parameters of the receiving terminal, the method further comprises: Obtain N panoramic sub-videos in the panoramic video, and point information associated with each of the panoramic sub-videos, where the panoramic sub-videos are videos taken at each shooting point in the panoramic video; Acquiring a frame sequence of each panoramic sub-video, the frame sequence including at least one panoramic image frame, and a playback progress corresponding to each said panoramic image frame; The panoramic sub-video and the point information, as well as the panoramic image frame and the playback progress are stored in association. 4. The method according to claim 3, characterized in that, after the acquisition of the frame sequence of each panoramic sub-video, the method further comprises: Acquiring pixel coordinates of each pixel point of each panoramic image frame in a preset image coordinate system; The pixel point and the pixel coordinates are stored associatively. 5. The method according to claim 4, wherein the field of view information includes field of view coordinates and an initial field of view angle, and according to the field of view information, the primary panorama image frame is clipped to obtain the target Image frames, including: Determining a center point from the primary panoramic image frame according to the field of view coordinates, where the center point is a pixel point whose pixel coordinates match the field of view coordinates; Based on the center point and the initial field of view angle, the primary panoramic image frame is cropped to obtain a target image frame. 6. The method according to claim 5, wherein the playback parameters further comprise a scaling ratio, and the primary panorama image frame is cropped based on the center point and the initial field of view angle, Get the target image frame, including: Obtaining a target viewing angle according to the scaling ratio and the initial viewing angle; Based on the center point and the target field of view angle, the primary panoramic image frame is cropped to obtain a target image frame. 7. The method of claim 6, wherein, The initial field of view is obtained based on the terminal according to the field of view of the human eye, and the scaling ratio is obtained based on operation information collected by a sensor of the terminal, or based on a first input received by the terminal. 8. A panoramic video distribution device, characterized in that the device comprises: A receiving module, configured to receive playback parameters of the terminal when a panoramic video transmission connection is established with the terminal, wherein the playback parameters include position information, time information, and field of view information; A determining module, configured to determine a primary panoramic image frame from the panoramic video according to the position information and the time information; A clipping module, configured to clip the primary panoramic image frame according to the field of view information to obtain a target image frame; A distribution module, configured to distribute the target image frame to the terminal. 9. An electronic device, characterized in that the device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method according to any one of claims 1-7 is implemented. 10. A computer storage medium, wherein computer program instructions are stored on the computer storage medium, and when the computer program instructions are executed by a processor, the method according to any one of claims 1-7 is implemented.

Images