JP2025510260A - Patch mesh connection coding - Google Patents

Abstract

The connectivity and mapping information of the mesh surface patch can be encoded after projection into 2D. Regarding the connectivity information, the projection operation does not change the connections between vertices, so the same list of connected vertices can be carried in the atlas data. Similarly, the mapping information does not change after projection and can be carried in the atlas data. We disclose two methods for encoding the connectivity and mapping information. For the connectivity information, the video-based method uses adjacent color coding. For the mapping coordinates, the method uses the projected vertex positions. The connectivity and mapping can also be processed by an external mesh encoder. The newly proposed mapping information can be utilized to perform temporal compression.
[Selected Figure] Figure 1

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
This application claims priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application No. 63/269,905, entitled “PATCH MESH CONNECTIVITY CODING,” filed March 25, 2022, which is hereby incorporated by reference in its entirety for all purposes.

The present invention relates to three-dimensional graphics. More specifically, the present invention relates to coding of three-dimensional graphics.

Recently, a new method for compressing volumetric content such as point clouds is becoming standardized, based on a 3D to 2D projection. Also known as V3C (visual volumetric video-based compression), this method maps the 3D volume data into a number of 2D patches, which are then placed into an atlas image, which is then encoded by a video encoder. The atlas image corresponds to the geometry of the points, their respective textures, and an occupancy map that indicates which locations should be considered for point cloud reconstruction.

In 2017, MPEG conducted a Call for Proposals (CfP) for the compression of point clouds. After evaluation of several proposals, MPEG is currently considering two different techniques for point cloud compression: 3D native coding techniques (based on octrees and similar coding methods) or 3D to 2D projection followed by traditional video coding. For dynamic 3D scenes, MPEG is using Test Model Software (TMC2), based on patch surface modeling, projection of the patches from 3D to a 2D image, and coding of the 2D image by a video encoder such as HEVC. This method has proven to be more efficient than native 3D coding and can achieve competitive bitrates with acceptable quality.

Due to the success of coding 3D point clouds with projection-based methods (also known as video-based methods, or V-PCC), future versions of this standard are expected to include additional 3D data, such as 3D meshes. However, the current version of this standard is only suitable for transmitting a set of unconnected points, and therefore has no mechanism for transmitting point connectivity as required for 3D mesh compression.

Methods have also been proposed to extend the functionality of V-PCC to meshes. One possible method is to use V-PCC to code the vertices and then use a mesh compression method such as TFAN or Edgebreaker to code the connections. A limitation of this method is that the original mesh needs to be dense so that the point cloud generated from the vertices is not sparse and can be efficiently coded after projection. Furthermore, since the order of the vertices affects the coding of the connections, different methods have been proposed to reorganize the connections of the mesh. An alternative method for coding sparse meshes is to use the raw patch data to code the vertex positions in 3D. Since raw patches directly code (x,y,z), in this method all vertices are coded as raw data while the connections are coded by a similar mesh compression method as mentioned before. Note that in raw patches the vertices can be sent in any preferred order, so the order generated from the connection coding can be used. While this method can encode sparse point clouds, raw patches are not efficient at encoding 3D data, and further data such as triangle face attributes may be missing from this method.

The connectivity and mapping information of mesh surface patches can be encoded after projection into 2D. For connectivity information, the projection operation does not change the connections between vertices, so the same list of connected vertices can be carried in the atlas data. Similarly, the mapping information does not change after projection and can be carried in the atlas data. We disclose two methods for encoding the connectivity and mapping information. For connectivity information, a video-based method uses adjacent color coding. For mapping coordinates, the method uses the projected vertex positions. The connectivity and mapping can also be processed by an external mesh encoder. The newly proposed mapping information can be exploited to perform temporal compression.

In one aspect, a method for encoding connectivity and mapping information includes encoding vertex mapping information including delta information for geometry correction, and encoding patch connectivity information including implementing mesh simplification by fixing vertex positions in time. The method further includes transmitting a flag indicating whether the vertex mapping information is transmitted implicitly or explicitly. The steps of encoding the vertex mapping information and encoding the patch connectivity information are performed by an external encoder. The step of encoding the patch connectivity information includes using color coding in an occupancy map. The use of color coding in the occupancy map is limited to a maximum of four colors. The step of encoding the vertex mapping information further includes using rate distortion face transmission. Implementing mesh simplification includes transmitting only boundary vertices and not transmitting interior vertices. The interior vertices are determined based on a previous set of interior vertices from a previous frame.

In another aspect, an apparatus includes a non-transitory memory for storing an application, the application for encoding vertex mapping information including delta information for geometry correction, and encoding patch connection information including implementing mesh simplification by fixing vertex positions in time, and a processor coupled to the memory and configured to process the application. The application is further configured to transmit a flag indicating whether the vertex mapping information is transmitted implicitly or explicitly. The encoding of the vertex mapping information and the encoding of the patch connection information are performed by an external encoder. The encoding of the patch connection information includes using color coding in an occupancy map. The use of color coding in the occupancy map is limited to a maximum of four colors. The encoding of the vertex mapping information further includes using rate-distortion surface transmission. The implementing mesh simplification includes transmitting only boundary vertices and not transmitting interior vertices. The interior vertices are determined based on a previous set of interior vertices from a previous frame.

In another aspect, a system includes one or more cameras for acquiring three-dimensional content; an encoder for encoding the three-dimensional content, the encoding including encoding vertex mapping information including delta information for geometry correction, and encoding patch connectivity information including implementing mesh simplification by fixing vertex positions in time; and a decoder for decoding the encoded three-dimensional content, the decoding including adjusting a mesh using the delta information, and determining interior vertices of the patch connectivity information from previous interior vertices of a previous frame. The encoder is further configured to transmit a flag indicating whether the vertex mapping information is transmitted implicitly or explicitly. The encoding of the vertex mapping information and the encoding of the patch connectivity information are performed by an external encoder. The encoding of the patch connectivity information includes using color coding in an occupancy map. The use of color coding in the occupancy map is limited to a maximum of four colors. The encoding of the vertex mapping information further includes using rate-distortion surface transmission. Implementing mesh simplification includes transmitting only boundary vertices and not transmitting interior vertices.

FIG. 2 is a diagram of binary encoding according to some embodiments. FIG. 13 is an illustration of color coding using an occupancy map according to some embodiments. FIG. 13 is a diagram of RD plane transmission according to some embodiments. FIG. 13 is an illustration of temporal stability of mesh connections according to some embodiments. 1 is a flowchart of a method for patch mesh connectivity coding according to some embodiments. 1 is a block diagram of an example computing device configured to implement a patch mesh connectivity coding method according to some embodiments.

The connectivity and mapping information of mesh surface patches can be encoded after projection into 2D. For connectivity information, the projection operation does not change the connections between vertices, so the same list of connected vertices can be carried in the atlas data. Similarly, the mapping information does not change after projection and can be carried in the atlas data. We disclose two methods for encoding the connectivity and mapping information. For connectivity information, a video-based method uses adjacent color coding. For mapping coordinates, the method uses the projected vertex positions. The connectivity and mapping can also be processed by an external mesh encoder. The newly proposed mapping information can be exploited to perform temporal compression.

Connectivity information describes which pixels are connected. A triangle has a set of information. One of the sets of information is the triangle's position on the texture map. Each triangle on the texture map has two sets of information: 1) how the vertices are connected in 3D (e.g., a connection list), and 2) vertex mapping information.

There are three ways to encode vertex mapping information: implicit, explicit, and binary. In an implicit implementation, when projected onto a 2D surface, the projection is the same as the mapping. For example, the UV coordinates are where you hit when projecting onto the projection surface. In an explicit implementation, projection is performed but different coordinates are sent to the texture. In a binary implementation, the explicit information is encoded in an external encoder (e.g. Draco or AFX).

Here is the updated syntax for the explicit implementation:
AtlasPatch2dPosX = mpdu_vertex_pos_x[tileID][patchIdx][I]
AtlasPatch2dPosY = mpdu_vertex_pos_y[tileID][patchIdx][I]
if (asps_mesh_uv_coordinates_present_flag)
mappingU = (mpdu_vertex_u_coord[tileID][patchIdx][i]) / 2 ^{asps_mesh_coordinates_bit_depth_minus1+1} -1
mappingV = (mpdu_vertex_v_coord[tileID][patchIdx][i]) / 2 ^{asps_mesh_coordinates_bit_depth_minus1+1} -1
else
mappingU = AtlasPatch2dPosX
mappingV = AtlasPatch2dPosY
A flag can be sent to indicate whether the vertex mapping information is sent implicitly or explicitly. If the information is sent explicitly, the values are scaled by the bit depth.

If binary coding is implemented, an external mesh encoder can be used to encode the patch mesh information. U and V are added to the ply, and vertex mapping information is encoded with the ply. In some embodiments, delta information for the z coordinate is added. The delta information can be used for geometry correction.

There are many ways to encode patch connectivity information. In an explicit implementation, the syntax indicates which pixels are connected, so a list of pixel connections is sent with the patch. In a binary implementation, an external encoder can be utilized. In another implementation, mesh simplification can be performed by fixing vertex positions over time. In a color coding implementation, color coding using an occupancy map is implemented. Triangles can be mapped using only four colors. In another implementation, rate-distortion (RD) face transmission is utilized.

Figure 1 shows a diagram of binary encoding according to some embodiments. A geometry image is used to generate a map 100. There is still some information that can be corrected by video encoding. By explicitly sending delta information to an external encoder, a binary image can be generated while correcting errors from the video transmission. UV coordinates are also sent, which are useful for video compression. In some embodiments, an external mesh encoder is used to encode the patch mesh information.

Figure 2 shows a diagram of color coding using an occupancy map according to some embodiments. The mapping of triangle indices can use only four colors. Because the diagram is in grayscale, some colors, edges, and vertices may be difficult to distinguish, but only four colors are used and triangles of one color do not share edges with triangles of the same color. Connectivity information is added to the occupancy map using only the luminance channel (4 colors -> (0,0,0), (64,0,0), (128,0,0), (255,0,0)). Further details regarding color coding and mesh compression can be found in U.S. Patent Application No. 17/322,662, entitled "VIDEO BASED MESH COMPRESSION," filed May 17, 2021, U.S. Provisional Patent Application No. 63/088,705, entitled "VIDEO BASED MESH COMPRESSION," filed October 7, 2020, and U.S. Provisional Patent Application No. 63/087,958, entitled "VIDEO BASED MESH COMPRESSION," filed October 6, 2020, all of which are incorporated by reference in their entirety for all purposes. The connectivity of a 3D mesh or 2D patch mesh can be coded using temporal correlation using an occupancy map and using video-based mesh compression. Additionally, vertices can be detected using a color map 200, and triangle intersections can be detected based on color.

Figure 3 shows a diagram of RD-plane transmission according to some embodiments. A mesh is received/obtained (300). Mesh connection points are encoded by an encoder (302). At the decoder side, the mesh is reconstructed (304), but the positions of the points may differ slightly from the original mesh. Correction information (e.g., delta information) is sent to the decoder so that the decoder can adjust the mesh (306) to make it more accurate compared to the original mesh.

The vertices of the input mesh are coded with V-PCC and decoded locally. The encoder generates a mesh from the decoded point cloud. The encoder compares the generated face/connectivity information with the original information. The encoder signals inconsistent faces incorporating a rate/distortion tradeoff. The decoder decodes the mesh vertices using V-PCC and generates a mesh from the decoded vertices. The decoder modifies the mesh using the signaled inconsistent faces. In some embodiments, a similar approach can be applied where instead of 3D, 2D triangulation is used to code the UV coordinates and their connectivity.

Figure 4 shows an illustration of the temporal stability of mesh connections according to some embodiments. When a patch is transmitted, only the boundary vertices are transmitted. No interior vertices are transmitted. The decoder determines the interior vertices (e.g., based on previous or subsequent frames). For example, the interior vertices of patch 400 in frame 1 are used as the interior vertices of patch 402 in frame 2. In some embodiments, a first set of interior vertices is transmitted (e.g., for frame 0 or frame 1), and future frames use the interior vertices of the previous frame. The decoder can regenerate triangles from the boundary vertices and the interior vertices. The same interior points can be used even if the patch is rotated from frame to frame. The interior triangles may be slightly different due to the rotation, which is tolerable. By not transmitting the interior vertices, fewer bits are transmitted.

5 shows a flow chart of a method for patch mesh connectivity coding according to some embodiments. In step 500, vertex mapping information is encoded. For geometry correction, the vertex mapping information may include delta information. A flag may be transmitted indicating whether the vertex mapping information is transmitted implicitly or explicitly. In some embodiments, the encoding of the vertex mapping information uses RD face transmission. In step 502, patch connectivity information is encoded. The encoding of the patch connectivity information includes implementing mesh simplification by fixing the positions of the vertices in time. In some embodiments, the simplification of the mesh includes transmitting only boundary vertices and not interior vertices. The patch connectivity information may include color coding in the occupancy map. The color coding is limited to a maximum of four colors. In some embodiments, the vertex information and the patch connectivity information are performed by an external encoder. In some embodiments, the order of steps is changed. In some embodiments, fewer or additional steps are implemented. For example, the encoded information may use delta information to adjust the mesh. In another example, the interior vertices of the patch connections may be determined from previous interior vertices of a previous frame.

FIG. 6 illustrates a block diagram of an exemplary computing device configured to implement a patch mesh connectivity coding method according to some embodiments. The computing device 600 can be used to acquire, store, compute, process, communicate and/or display information, such as images and videos, including 3D content. The computing device 600 can implement any of the encoding/decoding aspects. In general, a hardware structure suitable for implementing the computing device 600 includes a network interface 602, memory 604, a processor 606, I/O device(s) 608, a bus 610 and a storage device 612. The selection of the processor is not critical as long as a suitable processor of sufficient speed is selected. The memory 604 can be any conventional computer memory known in the art. The storage device 612 can include a hard drive, CDROM, CDRW, DVD, DVDRW, high definition disk/drive, ultra high definition drive, flash memory card, or any other storage device. The computing device 600 can include one or more network interfaces 602. An example of a network interface includes a network card connected to an Ethernet or other type of LAN. The I/O device(s) 608 may include one or more of a keyboard, mouse, monitor, screen, printer, modem, touch screen, button interface, and other devices. The storage device 612 and memory 604 likely store a patch mesh connectivity coding application(s) 630 used to execute the implementation of patch mesh connectivity coding and process as an application would normally be processed. The computing device 600 may also include more or fewer components than those shown in FIG. 6. In some embodiments, a patch mesh connectivity coding hardware 620 is included. Although the computing device 600 of FIG. 6 includes an application 630 and hardware 620 for the implementation of patch mesh connectivity coding, the patch mesh connectivity coding method may also be implemented on the computing device in hardware, firmware, software, or any combination thereof. For example, in some embodiments, the patch mesh connectivity coding application 630 is programmed into memory and executed using a processor. As another example, in some embodiments, the patch mesh connectivity coding hardware 620 is programmed hardware logic that includes gates specifically designed to implement the patch mesh connectivity coding method.

In some embodiments, the patch mesh connectivity coding application(s) 630 include multiple applications and/or modules. In some embodiments, a module also includes one or more sub-modules. In some embodiments, fewer or additional modules may be included.

Examples of suitable computing devices include personal computers, laptop computers, computer workstations, servers, mainframe computers, handheld computers, personal digital assistants, cellular/mobile telephones, smart appliances, gaming consoles, digital cameras, digital camcorders, camera phones, smart phones, portable music players, tablet computers, mobile devices, video players, video disc writers/players (e.g., DVD writers/players, high definition disc writers/players, ultra high definition disc writers/players), televisions, home entertainment systems, augmented reality devices, virtual reality devices, smart jewelry (e.g., smart watches), vehicles (e.g., autonomous vehicles), or any other suitable computing device.

To utilize the patch mesh connectivity coding method, a device acquires or receives 3D content (e.g., point cloud content). The patch mesh connectivity coding method can be implemented with user assistance or automatically without user involvement.

In operation, the patch mesh connectivity coding method enables more efficient and accurate 3D content encoding compared to conventional implementations.

Some embodiments of patch mesh connectivity coding: 1. A method for encoding connectivity and mapping information, comprising:
encoding vertex mapping information including delta information for geometry correction;
encoding patch connectivity information, including implementing mesh simplification by fixing vertex positions in time;
The method includes:

2. The method of claim 1, further comprising the step of transmitting a flag indicating whether the vertex mapping information is transmitted implicitly or explicitly.

3. The method according to claim 1, wherein the steps of encoding the vertex mapping information and encoding the patch connection information are performed by an external encoder.

4. The method of claim 1, wherein the step of encoding the patch connectivity information includes using color coding in the occupancy map.

5. The method of claim 4, wherein the use of color coding in the occupancy map is limited to a maximum of four colors.

6. The method of claim 1, wherein the step of encoding the vertex mapping information further comprises using rate-distortion plane transmission.

7. The method of claim 1, wherein implementing mesh simplification includes transmitting only boundary vertices and not transmitting interior vertices.

8. The method of claim 7, wherein the interior vertices are determined based on a previous set of interior vertices from a previous frame.

9. An apparatus comprising:
A non-transitory memory for storing an application, the application comprising:
encoding vertex mapping information including delta information for geometry correction;
encoding patch connectivity information, including implementing mesh simplification by fixing vertex positions in time;
a non-transient memory for performing
a processor coupled to the memory and configured to process the application;
An apparatus comprising:

10. The device of claim 9, wherein the application is further configured to send a flag indicating whether the vertex mapping information is sent implicitly or explicitly.

11. The apparatus of claim 9, wherein the encoding of the vertex mapping information and the encoding of the patch connection information are performed by an external encoder.

12. The apparatus of claim 9, wherein encoding the patch connectivity information includes using color coding in an occupancy map.

13. The apparatus of claim 12, wherein the use of color coding in the occupancy map is limited to a maximum of four colors.

14. The apparatus of claim 9, wherein encoding the vertex mapping information further comprises using rate-distortion plane transmission.

15. The apparatus of claim 9, wherein implementing mesh simplification includes transmitting only boundary vertices and not transmitting interior vertices.

16. The apparatus of claim 15, wherein the interior vertices are determined based on a previous set of interior vertices from a previous frame.

17. A system comprising:
one or more cameras for acquiring three-dimensional content;
11. An encoder for encoding the three-dimensional content, the encoding comprising:
encoding vertex mapping information including delta information for geometry correction;
encoding patch connectivity information, including implementing mesh simplification by fixing vertex positions in time;
an encoder including:
A decoder for decoding the encoded three-dimensional content, the decoding comprising:
adjusting a mesh using the delta information; and
determining interior vertices of the patch connectivity information from previous interior vertices of a previous frame;
a decoder,
A system including:

18. The system of claim 17, wherein the encoder is further configured to transmit a flag indicating whether the vertex mapping information is transmitted implicitly or explicitly.

19. The system of claim 17, wherein encoding the vertex mapping information and encoding the patch connection information is performed by an external encoder.

20. The system of claim 17, wherein encoding the patch connectivity information includes using color coding in an occupancy map.

21. The system of claim 20, wherein the use of color coding in the occupancy map is limited to a maximum of four colors.

22. The system of claim 17, wherein encoding the vertex mapping information further comprises using rate-distortion plane transmission.

23. The system of claim 17, wherein implementing mesh simplification includes transmitting only boundary vertices and not transmitting interior vertices.

The present invention has been described with respect to specific embodiments including details so that the principles of construction and operation of the invention may be readily understood. Reference herein to such specific embodiments and details thereof is not intended to limit the scope of the claims appended hereto. It will be readily apparent to those skilled in the art that various other modifications may be made in the embodiments selected for illustration without departing from the spirit and scope of the invention as defined by the claims.

100 Map 200 Color Map 300 Receive/Get Mesh 302 Encode Mesh Connection Points 304 Reconstruct Mesh 306 Adjust Mesh 400 Patch 402 Patch 500 Encode Vertex Mapping Information 502 Encode Patch Connection Information 600 Computing Device 602 Network Interface 604 Memory 606 Processor 608 I/O Device 610 Bus 612 Storage Device 620 Patch Mesh Connection Coding Hardware 630 Patch Mesh Connection Coding Application

Claims (23)

1. A method for encoding connection information and mapping information, comprising the steps of:
encoding vertex mapping information including delta information for geometry correction;
encoding patch connectivity information, including implementing mesh simplification by fixing vertex positions in time;
The method according to claim 1, further comprising: The method of claim 1, further comprising the step of transmitting a flag indicating whether the vertex mapping information is transmitted implicitly or explicitly. The method of claim 1, wherein the steps of encoding the vertex mapping information and encoding the patch connection information are performed by an external encoder. The method of claim 1, wherein the step of encoding the patch connectivity information includes using color coding in an occupancy map. The method of claim 4, characterized in that the use of color coding in the occupancy map is limited to a maximum of four colors. The method of claim 1, wherein the step of encoding the vertex mapping information further comprises using rate distortion face transmission. The method of claim 1, wherein implementing mesh simplification includes transmitting only boundary vertices and not interior vertices. The method of claim 7, wherein the interior vertices are determined based on a previous set of interior vertices from a previous frame. An apparatus comprising:
A non-transitory memory for storing an application, the application comprising:
encoding vertex mapping information including delta information for geometry correction;
encoding patch connectivity information, including implementing mesh simplification by fixing vertex positions in time;
a non-transient memory for performing
a processor coupled to the memory and configured to process the application;
An apparatus comprising: The apparatus of claim 9, wherein the application is further configured to send a flag indicating whether the vertex mapping information is sent implicitly or explicitly. The apparatus of claim 9, wherein the encoding of the vertex mapping information and the encoding of the patch connection information are performed by an external encoder. The apparatus of claim 9, wherein encoding the patch connectivity information includes using color coding in an occupancy map. The device of claim 12, characterized in that the use of color coding in the occupancy map is limited to a maximum of four colors. The apparatus of claim 9, wherein encoding the vertex mapping information further comprises using rate-distortion plane transmission. The apparatus of claim 9, wherein implementing mesh simplification includes transmitting only boundary vertices and not transmitting interior vertices. The apparatus of claim 15, wherein the interior vertices are determined based on a set of previous interior vertices from a previous frame. 1. A system comprising:
one or more cameras for acquiring three-dimensional content;
11. An encoder for encoding the three-dimensional content, the encoding comprising:
encoding vertex mapping information including delta information for geometry correction;
encoding patch connectivity information, including implementing mesh simplification by fixing vertex positions in time;
an encoder including:
A decoder for decoding the encoded three-dimensional content, the decoding comprising:
adjusting a mesh using the delta information; and
determining interior vertices of the patch connectivity information from previous interior vertices of a previous frame;
a decoder,
A system comprising: The system of claim 17, wherein the encoder is further configured to transmit a flag indicating whether the vertex mapping information is transmitted implicitly or explicitly. The system of claim 17, wherein the encoding of the vertex mapping information and the encoding of the patch connection information are performed by an external encoder. The system of claim 17, wherein encoding the patch connectivity information includes using color coding in an occupancy map. The system of claim 20, wherein the use of color coding in the occupancy map is limited to a maximum of four colors. The system of claim 17, wherein encoding the vertex mapping information further comprises using rate-distortion surface transmission. The system of claim 17, wherein implementing mesh simplification includes transmitting only boundary vertices and not interior vertices.

Images