CN116805349A - Indoor scene reconstruction method, device, electronic equipment and media - Google Patents

Abstract

This application provides an indoor scene reconstruction method, device, electronic equipment and media. The method includes: obtaining training perspective parameters based on the training image and establishing a scene point cloud; selecting multiple location points as training sampling points on the ray starting from the origin of the training perspective and passing through the location point of each pixel in the training image; determining The point cloud points in the scene point cloud within a predetermined range near the training sampling point are used as a training point set. The current input information of the training point set is input into the current scene reconstruction model to obtain the color and light field intensity of the training point set; the training samples are obtained by interpolation After adjusting the color and light field intensity of the point, the rendered image under the training perspective parameters is obtained by rendering; based on the loss between the rendered image under the training perspective parameters and the training image, the current input information and scene reconstruction model are trained. The method of this application solves the problem of poor quality indoor scene rendering images when the number of training images is small.

Description

Indoor scene reconstruction method, device, electronic equipment and media

Technical field

The present application relates to the field of virtual reality technology, and in particular to an indoor scene reconstruction method, device, electronic equipment and media.

Background technique

Virtual Reality (VR) technology, based on computer technology, utilizes and integrates the latest development achievements of various high-tech technologies such as three-dimensional graphics technology, multimedia technology, simulation technology, display technology, and servo technology, and uses computers and other equipment to generate a A virtual world with realistic three-dimensional vision, touch, smell and other sensory experiences, thus giving people in the virtual world an immersive feeling. With the continuous development of social productivity and science and technology, the demand for VR technology in all walks of life is increasingly strong. VR technology has also made great progress and gradually become a new field of science and technology. New perspective synthesis of indoor scenes plays an important role in the interaction between virtual reality and people, so this task has great potential in many VR applications, such as indoor scene virtual tours or tourist navigation.

In the existing technology, new perspective synthesis of indoor scenes uses neural radiation fields extensively, uses continuous multi-layer perceptrons to encode the brightness and density of the three-dimensional scene, and reconstructs the light field through ray tracing.

In the above scheme, in order to reconstruct a room-sized scene, at least hundreds of RGB images densely collected around the scene are required, which requires large labor costs and time costs and has high requirements for shooting quality. If the number of input images is not enough, the generated rendered image will have many holes and erroneous renderings. Therefore, there is a problem of poor quality indoor scene rendering images when the number of training images is small.

Contents of the invention

This application provides an indoor scene reconstruction method, device, electronic equipment and media to solve the problem of poor quality of generated indoor scene rendering images when the number of training images is small.

On the one hand, this application provides an indoor scene reconstruction method, including:

Image collection is performed indoors to obtain training images; and, based on the motion recovery structure technology, the training perspective parameters corresponding to each training image are obtained according to the training images and a scene point cloud is established; wherein, the scene point cloud is Point clouds representing structural information of indoor scenes;

On the ray starting from the origin of the training perspective and passing through the position point corresponding to each pixel in the training image under the training perspective parameter in the scene point cloud, select a plurality of the position points as training sampling points; and, based on semantics The prediction technology determines the point cloud points in the scene point cloud located within a predetermined range near the multiple training sampling points as a training point set, and obtains the current input information of the training point set; wherein the input information includes Spatial location, neural characteristics, and ray direction;

Input the current input information of the training point set into the current scene reconstruction model to obtain the color and light field intensity of the training point set output by the scene reconstruction model; and, according to the color and light field intensity of the training point set Intensity, obtain the color and light field intensity of the training sampling point through interpolation; and, obtain the rendered image under the training perspective parameter by rendering according to the color and light field intensity of the training sampling point;

According to the loss between the rendered image under the training perspective parameters and the training image under the training perspective parameters, the current input information and the current scene reconstruction model are trained and corrected until the trained input information and the current scene reconstruction model are obtained. The scene reconstruction model.

Optionally, determining point cloud points in the scene point cloud within a predetermined range near the multiple training sampling points as a training point set based on semantic prediction technology includes:

Based on semantic prediction technology, the semantic information of each pixel point in the training image under the training perspective parameter and the semantic information of each point cloud point in the scene point cloud are obtained; the semantic information of the pixel point includes the object category, and the The semantic information of point cloud points includes object categories and corresponding probability values;

For each training sampling point, obtain all point cloud points in the scene point cloud located within a predetermined range around the training sampling point; and calculate the selection probability of each point cloud point within the predetermined range around the training sampling point, according to In order of selection probability from high to low, a preset value of point cloud points is selected as the training point set; wherein, calculating the selection probability includes: if the object category of the point cloud point corresponds to the training sampling point The object category of the pixel points is the same, then the selection probability of the point cloud point is one; if the object category of the point cloud point is different from the object category of the pixel point corresponding to the training sampling point, then the selection probability of the point cloud point is The probability is one minus the probability value corresponding to the object category of the pixel corresponding to the point cloud point.

Optionally, the neural features include first neural features and second neural features, and obtaining the current input information of the training point set includes:

Input all the training images and the scene point cloud into the semantic prediction network to obtain the first neural features of each point cloud point in the scene point cloud;

The training image corresponding to the origin of the training perspective closest to the point cloud point is obtained, and the training image is input into the convolutional neural network to obtain the second neural feature of the point cloud point.

Optionally, the method also includes:

Obtain the target perspective parameters of the scene to be reconstructed; and determine virtual pixels according to the target perspective parameters;

On the ray starting from the origin of the target perspective and passing through the position point corresponding to each of the virtual pixel points in the scene point cloud, select a plurality of the position points as the target sampling points; and, determine that the position points are located at the plurality of Point cloud points in the scene point cloud within a predetermined range near the target sampling point are used as a target point set, and the input information of the target point set is obtained;

Input the input information of the target point set into the scene reconstruction model to obtain the color and light field intensity of the target point set output by the scene reconstruction model; and, according to the color and light field intensity of the target point set Field intensity: obtain the color and light field intensity of the target sampling point through interpolation; and obtain the rendered image under the target viewing angle parameters through rendering according to the color and light field intensity of the target sampling point.

Optionally, determining the point cloud points in the scene point cloud located within a predetermined range near the multiple target sampling points as a target point set includes:

Obtain all point cloud points in the scene point cloud located within a predetermined range near the plurality of target sampling points, and select a predetermined number of the point cloud points closest to each of the sampling points as the target points. gather.

Optionally, the training image satisfies the following conditions: the training image contains a room surface; the overlap between any two adjacent training images under training perspective parameters is not less than 3% of the entire image size. ten.

On the other hand, this application provides an indoor scene reconstruction device, including:

An acquisition module is used to collect images indoors to obtain training images; and, based on the motion recovery structure technology, obtain the training perspective parameters corresponding to each of the training images according to the training images and establish a scene point cloud; wherein, The scene point cloud described above is a point cloud that represents the structural information of the indoor scene;

A sampling module, configured to select a plurality of the position points as training sampling points on the ray starting from the origin of the training perspective and passing through the position point corresponding to each pixel in the training image under the training perspective parameter in the scene point cloud. ; And, based on semantic prediction technology, determine the point cloud points in the scene point cloud located within a predetermined range near the multiple training sampling points as a training point set, and obtain the current input information of the training point set; wherein, The input information includes spatial position, neural characteristics and ray direction;

A rendering module, configured to input the current input information of the training point set into the current scene reconstruction model, and obtain the color and light field intensity of the training point set output by the scene reconstruction model; and, according to the training point The color and light field intensity of the set are interpolated to obtain the color and light field intensity of the training sampling point; and, based on the color and light field intensity of the training sampling point, the rendered image under the training perspective parameter is obtained through rendering;

A correction module, configured to train and correct the current input information and the current scene reconstruction model according to the loss between the rendered image under the training perspective parameters and the training image under the training perspective parameters until a trained The input information and the scene reconstruction model.

Optional, the sampling module is specifically used for:

Based on semantic prediction technology, the semantic information of each pixel point in the training image under the training perspective parameter and the semantic information of each point cloud point in the scene point cloud are obtained; the semantic information of the pixel point includes the object category, and the The semantic information of point cloud points includes object categories and corresponding probability values;

For each training sampling point, obtain all point cloud points in the scene point cloud located within a predetermined range around the training sampling point; and calculate the selection probability of each point cloud point within the predetermined range around the training sampling point, according to In order of selection probability from high to low, a preset value of point cloud points is selected as the training point set; wherein, calculating the selection probability includes: if the object category of the point cloud point corresponds to the training sampling point The object category of the pixel points is the same, then the selection probability of the point cloud point is one; if the object category of the point cloud point is different from the object category of the pixel point corresponding to the training sampling point, then the selection probability of the point cloud point is The probability is one minus the probability value corresponding to the object category of the pixel corresponding to the point cloud point.

Optionally, the neural features include first neural features and second neural features, and the sampling module is also specifically used for:

Input all the training images and the scene point cloud into the semantic prediction network to obtain the first neural features of each point cloud point in the scene point cloud;

The training image corresponding to the origin of the training perspective closest to the point distance is obtained, and the training image is input into the convolutional neural network to obtain the second neural feature of the point cloud point.

Optionally, the device also includes a reconstruction module, used for:

Obtain the target perspective parameters of the scene to be reconstructed; and determine virtual pixels according to the target perspective parameters;

On the ray starting from the origin of the target perspective and passing through the position point corresponding to each of the virtual pixel points in the scene point cloud, select a plurality of the position points as the target sampling points; and, determine that the position points are located at the plurality of Point cloud points in the scene point cloud within a predetermined range near the target sampling point are used as a target point set, and the input information of the target point set is obtained;

Input the input information of the target point set into the scene reconstruction model to obtain the color and light field intensity of the target point set output by the scene reconstruction model; and, according to the color and light field intensity of the target point set Field intensity: obtain the color and light field intensity of the target sampling point through interpolation; and obtain the rendered image under the target viewing angle parameters through rendering according to the color and light field intensity of the target sampling point.

Optional, the reconstruction module is specifically used for:

Obtain all point cloud points in the scene point cloud located within a predetermined range near the plurality of target sampling points, and select a predetermined number of the point cloud points closest to each of the sampling points as the target points. gather.

Optionally, the training images under known viewing angle parameters meet the following conditions: the training image contains the room surface; the overlap between any two adjacent training images under known viewing angle parameters is not less than the entire image size. Thirty percent.

In another aspect, the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, to implement the method as described previously.

In another aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and when executed by a processor, the computer-executable instructions are used to implement the method as described above.

In the indoor scene reconstruction method, device, electronic equipment and media provided by this application, image collection is performed indoors to obtain training images; and based on motion recovery structure technology, based on the training images, the corresponding data of each training image is obtained. Train the viewing angle parameters and establish a scene point cloud; wherein the scene point cloud is a point cloud that represents the structural information of the indoor scene; each training image starting from the origin of the training viewing angle and passing through the training viewing angle parameters in the scene point cloud On the ray of the position point corresponding to the pixel point, select a plurality of the position points as training sampling points; and, based on semantic prediction technology, determine points in the scene point cloud located within a predetermined range near the multiple training sampling points. Cloud points are used as a training point set, and the current input information of the training point set is obtained; wherein the input information includes spatial position, neural characteristics and ray direction; the current input information of the training point set is input into the current Scene reconstruction model, obtain the color and light field intensity of the training point set output by the scene reconstruction model; and, according to the color and light field intensity of the training point set, obtain the color and light field intensity of the training sampling point through interpolation ; And, according to the color and light field intensity of the training sampling point, obtain the rendering image under the training perspective parameter through rendering; according to the relationship between the rendering image under the training perspective parameter and the training image under the training perspective parameter loss, perform training and correction on the current input information and the current scene reconstruction model until the trained input information and the scene reconstruction model are obtained. By selecting a set of training points around the training sampling points based on semantic prediction technology, this solution improves the possibility that the object category of the point cloud point is the same as the object category of the corresponding training sampling point, so that the point cloud point color and light field intensity interpolation can be obtained The color and light field intensity of the training sampling points are more accurate, thereby improving the quality of the generated rendered images, and the input information of the point cloud points and the scene reconstruction model can be better trained and corrected based on the training images and corresponding rendered images. , reducing the requirement on the number of training images, thus solving the problem of poor quality of generated indoor scene rendering images when the number of training images is small.

Description of the drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.

Figure 1 exemplarily shows a flow chart of the indoor scene reconstruction method provided by Embodiment 1 of the present application;

Figure 2 schematically shows an example diagram of generating a scene point cloud provided by Embodiment 1 of the present application;

Figure 3 schematically shows an example diagram of a ray tracing scene provided by Embodiment 1 of the present application;

Figure 4 schematically shows an example diagram of a scene for semantic prediction provided by Embodiment 1 of the present application;

Figure 5 exemplarily shows a schematic flow chart for obtaining neural features of point cloud points provided by Embodiment 1 of the present application;

Figure 6 schematically shows an example scene diagram of real-time rendering with dynamic resolution provided by Embodiment 1 of the present application;

Figure 7 exemplarily shows a schematic structural diagram of an indoor scene reconstruction device provided in Embodiment 2 of the present application;

Figure 8 exemplarily shows a schematic structural diagram of an indoor scene reconstruction electronic device provided in Embodiment 3 of the present application.

Through the above-mentioned drawings, clear embodiments of the present application have been shown, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present application's concepts in any way, but are intended to illustrate the application's concepts for those skilled in the art with reference to specific embodiments.

Detailed ways

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this application. Rather, they are merely examples of apparatus and methods consistent with aspects of the application as detailed in the appended claims.

Virtual Reality (VR) technology, based on computer technology, utilizes and integrates the latest development achievements of various high-tech technologies such as three-dimensional graphics technology, multimedia technology, simulation technology, display technology, and servo technology, and uses computers and other equipment to generate a A virtual world with realistic three-dimensional vision, touch, smell and other sensory experiences, thus giving people in the virtual world an immersive feeling. With the continuous development of social productivity and science and technology, the demand for VR technology in all walks of life is increasingly strong. VR technology has also made great progress and gradually become a new field of science and technology. New perspective synthesis of indoor scenes plays an important role in the interaction between virtual reality and people, so this task has great potential in many VR applications, such as indoor scene virtual tours or tourist navigation.

Existing indoor scene new perspective synthesis technology requires at least hundreds of RGB images densely collected around the scene in order to reconstruct a room-sized scene, which requires large labor costs and time costs and has high requirements for shooting quality. If the number of input images is not enough, the generated rendered image will have many holes and erroneous renderings. Therefore, there is a problem of poor quality indoor scene rendering images when the number of training images is small.

The technical solution of the present application is illustrated below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

Embodiment 1

Figure 1 is a schematic flowchart of an indoor scene reconstruction method provided by an embodiment of the present application. As shown in Figure 1, the indoor scene reconstruction method provided by this embodiment may include:

S101, perform image collection indoors to obtain training images; and, based on the motion recovery structure technology, obtain the training perspective parameters corresponding to each training image according to the training images and establish a scene point cloud; wherein, the scene points The cloud is a point cloud that represents the structural information of the indoor scene;

S102. On the ray starting from the origin of the training perspective and passing through the location point corresponding to each pixel in the training image under the training perspective parameter in the scene point cloud, select multiple location points as training sampling points; and, Based on semantic prediction technology, the point cloud points in the scene point cloud located within a predetermined range near the multiple training sampling points are determined as a training point set, and the current input information of the training point set is obtained; wherein, the input Information includes spatial location, neural characteristics, and ray direction;

S103, input the current input information of the training point set into the current scene reconstruction model, and obtain the color and light field intensity of the training point set output by the scene reconstruction model; and, according to the color of the training point set and light field intensity, and obtain the color and light field intensity of the training sampling point through interpolation; and, according to the color and light field intensity of the training sampling point, obtain the rendered image under the training perspective parameter through rendering;

S104. Based on the loss between the rendering image under the training perspective parameters and the training image under the training perspective parameters, perform training correction on the current input information and the current scene reconstruction model until the trained input is obtained information to reconstruct the model with the scene.

In practical applications, the execution subject of this embodiment can be an indoor scene reconstruction device, which can be implemented through a computer program, such as application software, etc.; or it can also be implemented as a medium storing relevant computer programs, such as a U disk, Cloud disk, etc.; or, it can also be realized through physical devices integrated or installed with relevant computer programs, such as chips, servers, etc.

Specifically, a camera is used to shoot in the room to obtain RGB training images; based on structure from motion (SfM) technology such as Colmap, the training perspective parameters corresponding to each training image are obtained based on the training images, and a representation of the indoor scene is established. Structured 3D scene point cloud.

In practical applications, the viewing angle parameters can be camera parameters, which include camera intrinsic parameters and camera extrinsic parameters; among them, camera intrinsic parameters include intrinsic parameter matrices, and camera extrinsic parameters represent the camera pose, including rotation matrix and translation vector. Based on a certain viewing angle parameter, the corresponding spatial position of each pixel in the world coordinate system in the image obtained by the corresponding camera imaging under the viewing angle parameter can be uniquely determined.

Starting from the origin of the training perspective, a ray is emitted as a ray to the position corresponding to each pixel in the training image under the training perspective parameters, and multiple training sampling points are selected on each ray; among them, the origin of the training perspective, the pixel The corresponding position points and scene point clouds are located in the same coordinate system, and each ray passes through the scene point cloud; in practical applications, the origin of the training perspective can be the optical center of the camera, and its spatial position can be determined based on the training perspective parameters. And, based on semantic prediction technology, select multiple training point cloud points within a predetermined range near each training sampling point as a training point set, and obtain the current input information of the training point set; where the input information includes spatial location, neural features and ray direction.

Input the current input information of the training point set into the current scene reconstruction model to obtain the color and light field intensity of the training point set output by the scene reconstruction model; in practical applications, the scene reconstruction model can include a light field intensity decoder and a color decoder. , specific parameters can be selected according to actual production needs; according to the color and light field intensity of the training point set, the color and light field intensity of the training sampling point are obtained through interpolation; according to the color and light field intensity of the training sampling point, the training perspective parameters are obtained through rendering The color of each pixel in the rendered image below is used to generate a rendered image.

According to the training images and rendering images under the training perspective parameters, the input information of the point cloud points and the scene reconstruction model are corrected by calculating the loss until the input information of the trained point cloud points and the scene reconstruction model are obtained.

For example, in a room of about 10 square meters, 40 training images with different training perspective parameters are collected for indoor scene reconstruction.

Based on SfM technology, the training perspective parameters corresponding to the training images are obtained based on the training images, and the scene point cloud is generated. Figure 2 is an example diagram of generating a scene point cloud provided by an embodiment of the present application. As shown in Figure 2, according to training images containing indoor scenes, a 3D scene point cloud representing the indoor scene structure can be obtained based on SfM technology.

Figure 3 is an example diagram of a ray tracing scene provided by an embodiment of the present application. As shown in Figure 3, the origin of the training perspective and the corresponding position point of each pixel in the training image in the coordinate system where the scene point cloud is located are determined according to the training perspective parameters. Starting from the origin of the training perspective, a ray is drawn through the position of each pixel point, and N sampling points q ₁ , q ₂ ,..., q _N are evenly sampled in the set area on each ray, and based on semantic prediction technology, Within the sphere of radius r around each sampling point, a predetermined number of point cloud points are selected as a set of training points. In one possible implementation, 128 training sampling points are selected on each ray, and 8 point cloud points are selected around each training sampling point as a training point set.

Input the current spatial position, neural characteristics and ray direction of the training point set into the decoder, and obtain the color and light field intensity of each point cloud point in the training point set output by the decoder. The decoder is divided into two groups: light field intensity σ decoder and color c decoder, which generate light field intensity and color respectively. The decoder consists of six fully connected hidden layers and input and output layers. The number of neurons of the light field intensity decoder is 128-256-256-96-32-16, and the number of neurons of the color decoder is 128-256- 256-128-64-8.

According to the color and light field intensity of the training point set, the color and light field intensity of the training sampling point are obtained through inverse distance interpolation. Perform volume rendering integration on the color and square intensity of the training sampling points to obtain the color of the corresponding pixel.

For a pixel, the volume rendering integral calculation formula of its color c is as follows:.

c＝∑ _N τ _j (1-exp(-σ _j δ _j ))r _j

Among them, N is the number of sampling points selected from the ray passing through the camera origin and the pixel point, σ _j is the light field intensity of the j-th sampling point, r _j is the color of the j-th sampling point, and δ _j is the phase The distance between adjacent sampling points.

After obtaining the color of each pixel in the rendered image, the rendered image can be generated. Calculate the loss function between the rendered image and the real RGB values of the training view

in, To render an image,/> is the corresponding real training image. According to/> Modify the input information of point cloud points and the scene reconstruction model.

In practical applications, the collection of training images needs to meet certain requirements. In one example, the training images under the training perspective parameters meet the following conditions: the training image contains the room surface; any two adjacent training perspective parameters The overlap between training images is no less than thirty percent of the entire image size.

Specifically, the room surface features are included in at least one training image, and two consecutively shot training images under adjacent training perspective parameters overlap, and the size of the overlapping area is not less than 30% of the entire image size, thereby ensuring that the collected training images Images can achieve better training results.

For example, for a 10-square-meter room, use a monocular camera to capture 35-40 RGB images, so that the surface of the room is captured by at least one RGB image, and the overlap of the images from two consecutive camera angles is no less than 30% of image size.

There are many ways to determine the training point set. In one example, the semantic prediction technology is used to determine the point cloud points in the scene point cloud located within a predetermined range near the multiple training sampling points as training points. Point collections can include:

Based on semantic prediction technology, the semantic information of each pixel point in the training image under the training perspective parameter and the semantic information of each point cloud point in the scene point cloud are obtained; the semantic information of the pixel point includes the object category, and the The semantic information of point cloud points includes object categories and corresponding probability values;

For each training sampling point, obtain all point cloud points in the scene point cloud located within a predetermined range around the training sampling point; and calculate the selection probability of each point cloud point within the predetermined range around the training sampling point, according to In order of selection probability from high to low, a preset value of point cloud points is selected as the training point set; wherein, calculating the selection probability includes: if the object category of the point cloud point corresponds to the training sampling point The object category of the pixel points is the same, then the selection probability of the point cloud point is one; if the object category of the point cloud point is different from the object category of the pixel point corresponding to the training sampling point, then the selection probability of the point cloud point is The probability is one minus the probability value corresponding to the object category of the pixel corresponding to the point cloud point.

Specifically, the training image and scene point cloud are input into the semantic prediction network to obtain the semantic information of each pixel in the training image, including the predicted object category of the pixel, and the semantic information of each point cloud in the scene point cloud, including the point cloud. The predicted object category and corresponding probability value of the point.

For each training sampling point, obtain all point cloud points within a predetermined range around it, calculate the selection probability of each point cloud point, and select a predetermined value of point cloud points in order from high to low according to the selection probability as a training point set.

Among them, for a point cloud point, the calculation formula for its selection probability p _chosen is as follows:

in For the predicted object category of the point cloud points output by the semantic prediction network, let the predicted object category of the ray where the current training sampling point is located/> The predicted object category of the pixel corresponding to the position point through which the ray passes Same, let the predicted object category of the ray /> Corresponding probability value/> The predicted object category of the pixel corresponding to the position point where the ray passes/> Corresponding probability value/> same.

For example, Figure 4 is an example diagram of a scene for semantic prediction provided by an embodiment of the present application. As shown in Figure 4, the training image and scene point cloud are input into the semantic prediction module to obtain the semantics of each pixel in the output training image. information and the semantic information of each point cloud point in the scene point cloud. The pixel points and each point cloud point represent different predicted object categories through different grayscales. The probability values corresponding to the predicted object categories of the point cloud points are not reflected in the figure. , this is only for illustration.

As in the above example, obtain all the point cloud points in a ball with a radius r around each training sampling point, calculate the selection probability of the point cloud point, and select the 8 point cloud points with the highest selection probability as the training point set.

By making the probability that the predicted object category of the point cloud point is the same as the predicted object category of the ray is positively correlated with the selection probability of the point cloud point, and selecting the point cloud point with a high probability as the training point set, the training point cloud point and the training are improved The probability that the object categories corresponding to the sampling points are the same makes the color and light field intensity of the training sampling points obtained based on the training point cloud point parameters more accurate.

In one example, the neural features include first neural features and second neural features, and obtaining the current input information of the training point set may include:

Input all the training images and the scene point cloud into the semantic prediction network to obtain the first neural features of each point cloud point in the scene point cloud;

The training image corresponding to the origin of the training perspective closest to the point cloud point is obtained, and the training image is input into the convolutional neural network to obtain the second neural feature of the point cloud point.

Figure 5 is a schematic flowchart of obtaining neural features of point cloud points provided by an embodiment of the present application. As shown in Figure 5, the neural features of point cloud points may include first neural features and second neural features. Obtaining the first neural features of the point cloud points includes: inputting all training images and scene point clouds into the semantic prediction network to obtain the first neural features of each point cloud point in the scene point cloud. Obtaining the second neural feature of the point cloud point includes: acquiring a training image corresponding to the origin of the training perspective closest to the point cloud point, inputting the training image into the convolutional neural network, and obtaining the second neural feature of the point cloud point. The semantic prediction network is the same as the above-mentioned semantic prediction network. In practical applications, the training images and scene point clouds are input into the semantic prediction network, and the second neural features and predictions of each point cloud point in the scene point cloud output by the semantic prediction network can be obtained. Object categories and corresponding probability values.

For example, assuming that the resolution of the training image is 640×480, input the training image and scene point cloud into the semantic prediction network to obtain the first neural feature s of each point cloud point in the scene point cloud; the semantic prediction network is pre-trained BP-Net; input the training image corresponding to the origin of the training perspective closest to the point cloud point into the convolutional neural network to obtain the 32-dimensional second neural feature n; where the convolutional neural network can extract convolutions for the pre-trained image features The network consists of six convolution modules, with the number of channels being 64-128-128-256-128-32. The convolution module of each layer is composed of convolution layer-linear rectification-instance normalization unit.

By convolving the training image corresponding to the origin of the training perspective closest to the point cloud point and semantically predicting the semantic information of the point cloud point as the neural feature of the point cloud point, the structural information of the point cloud point other than the spatial position is obtained. Semantic information makes the input information of the point cloud point input scene reconstruction model richer, thereby obtaining more accurate point cloud point color and light field intensity.

Based on the corrected input information and scene reconstruction model, indoor new perspective scenes can be reconstructed. In one example, the method may further include:

Obtain the target perspective parameters of the scene to be reconstructed; and determine virtual pixels according to the target perspective parameters;

On the ray starting from the origin of the target perspective and passing through the position point corresponding to each of the virtual pixel points in the scene point cloud, select a plurality of the position points as the target sampling points; and, determine that the position points are located at the plurality of Point cloud points in the scene point cloud within a predetermined range near the target sampling point are used as a target point set, and the input information of the target point set is obtained;

Input the input information of the target point set into the scene reconstruction model to obtain the color and light field intensity of the target point set output by the scene reconstruction model; and, according to the color and light field intensity of the target point set Field intensity: obtain the color and light field intensity of the target sampling point through interpolation; and obtain the rendered image under the target viewing angle parameters through rendering according to the color and light field intensity of the target sampling point.

Specifically, after determining the target perspective of the scene to be reconstructed, the target perspective parameters are obtained. According to the target perspective parameters, determine the target perspective origin and the corresponding position point of the virtual pixel point in the coordinate system of the scene point cloud. Starting from the target perspective origin, pass through the scene point cloud and pass through the location point corresponding to each virtual pixel point to make a ray. In the ray Select target sampling points within a predetermined area, and select point cloud points in the scene point cloud within a predetermined range around each target sampling point to obtain a target point set.

Input the trained input information of the target point set into the trained scene reconstruction model to obtain the color and light field intensity of the target point set; interpolate the color and light field intensity of the target point set to obtain the color and light field intensity of the corresponding target sampling point. ; Render according to the color and light field intensity of the target sampling point to obtain a rendered image from a new perspective.

Based on the trained input information and the trained scene reconstruction model, the indoor scene new perspective image rendering can generate a better rendering image, thereby improving the indoor scene new perspective rendering image generated when the number of training images is small. the quality of.

For example, assume that you currently want to generate a new perspective image, obtain a training image of an indoor scene under the training perspective parameters, and obtain the input information of the trained point cloud points and the trained scene reconstruction model based on the training image.

After determining the target perspective parameters of the new perspective, determine the spatial position of the target perspective origin and the position point corresponding to the virtual pixel point in the coordinate system of the scene point cloud according to the target perspective parameter, and do a process starting from the target perspective origin and passing through each virtual pixel point. For the ray of the position point, 128 target sampling points are uniformly selected within the predetermined area of the ray, and the point cloud points in the scene point cloud are selected within a sphere with a radius r of each target sampling point to obtain a target point set.

Input the trained input information of the target point set into the trained scene reconstruction model to obtain the color and light field intensity of the target point set; perform inverse distance interpolation on the color and light field intensity of the target point set to obtain the color and light of the corresponding target sampling point Field intensity; perform volume rendering integration based on the color and light field intensity of the target sampling point to obtain a rendered image from a new perspective.

Furthermore, this solution can provide an indoor scene efficient collection and reconstruction roaming system. The user inputs sparsely collected RGB images of indoor scenes. After training, real-time roaming of the scene and high-quality new perspective rendering can be achieved. The system can render new perspectives, including moving the position of the line of sight, moving the direction of the line of sight, zooming in and out of the visible range of the line of sight, etc.

The system framework is implemented based on Python language, and the screen corresponding to the current perspective is visualized through software equipped with an interactive control panel. The neural network model is implemented based on PyTorch and trained on NVIDIA Quadro P6000 GPU and 24G memory single card. By migrating the code to the GPU for acceleration through Pycuda, real-time rendering can be achieved.

Figure 6 is an example diagram of a scene rendered with dynamic resolution in real time provided by an embodiment of the present application. As shown in Figure 6, the system allows the user to browse the scene in real time. When the user turns on the real-time rendering button, the system can dynamically adjust the rendering according to the rendering time. The resolution of the image. For example, when the rendering time exceeds the preset value, select a low resolution; and based on the resolution of the rendered image and the target perspective parameters, determine the position of each pixel in the rendered image, and generate a rendered image to ensure that the user Browse indoor scenes in real time.

There are many ways to select a set of target points. In one example, determining a set of target points contained within a predetermined range near the plurality of target sampling points may include:

All point cloud points within a predetermined range near the plurality of target sampling points are obtained, and a predetermined value of point cloud points closest to each of the sampling points are selected as the target point set.

Specifically, for each target sampling point, all point cloud points within a predetermined range around it are obtained, and a predetermined number of point cloud points are selected in order of distance from near to far as a set of target points.

For example, obtain all point cloud points in a ball with a radius r around the target sampling point, and select the 8 point cloud points closest to the target sampling point as the target point set.

In the indoor scene reconstruction method provided by this application, image collection is performed indoors to obtain training images; and, based on the motion recovery structure technology, based on the training images, the training perspective parameters corresponding to each training image are obtained and scene points are established Cloud; wherein, the scene point cloud is a point cloud that represents indoor scene structure information; starting from the origin of the training perspective and passing through the position point corresponding to each pixel point in the training image under the training perspective parameter in the scene point cloud On the ray, select a plurality of the position points as training sampling points; and, based on semantic prediction technology, determine the point cloud points in the scene point cloud located within a predetermined range near the multiple training sampling points as a training point set, And obtain the current input information of the training point set; wherein the input information includes spatial position, neural characteristics and ray direction; input the current input information of the training point set into the current scene reconstruction model to obtain the The color and light field intensity of the training point set output by the scene reconstruction model; and, according to the color and light field intensity of the training point set, obtain the color and light field intensity of the training sampling point through interpolation; and, according to the training The color and light field intensity of the sampling point are rendered to obtain the rendered image under the training perspective parameters; according to the loss between the rendering image under the training perspective parameters and the training image under the training perspective parameters, the current input The information and the current scene reconstruction model are trained and corrected until the trained input information and the scene reconstruction model are obtained. By selecting a set of training points around the training sampling points based on semantic prediction technology, this solution improves the possibility that the object category of the point cloud point is the same as the object category of the corresponding training sampling point, so that the point cloud point color and light field intensity interpolation can be obtained The color and light field intensity of the training sampling points are more accurate, thereby improving the quality of the generated rendered images, and the input information of the point cloud points and the scene reconstruction model can be better trained and corrected based on the training images and corresponding rendered images. , reducing the requirement on the number of training images, thus solving the problem of poor quality of generated indoor scene rendering images when the number of training images is small.

Embodiment 2

Figure 7 is a schematic structural diagram of an indoor scene reconstruction device provided by an embodiment of the present application. As shown in Figure 7, the indoor scene reconstruction device 70 provided in this embodiment may include:

The collection module 71 is used to collect images indoors to obtain training images; and, based on motion recovery structure technology, obtain the training perspective parameters corresponding to each training image according to the training images and establish a scene point cloud; wherein, The scene point cloud is a point cloud that represents the structural information of the indoor scene;

The sampling module 72 is used to select a plurality of the position points as training samples on the ray starting from the origin of the training perspective and passing through the position point corresponding to each pixel in the training image under the training perspective parameter in the scene point cloud. points; and, based on semantic prediction technology, determine the point cloud points in the scene point cloud located within a predetermined range near the multiple training sampling points as a training point set, and obtain the current input information of the training point set; wherein , the input information includes spatial position, neural characteristics and ray direction;

The rendering module 73 is used to input the current input information of the training point set into the current scene reconstruction model to obtain the color and light field intensity of the training point set output by the scene reconstruction model; and, according to the training The color and light field intensity of the point set are obtained by interpolation to obtain the color and light field intensity of the training sampling point; and, based on the color and light field intensity of the training sampling point, the rendered image under the training perspective parameter is obtained by rendering;

The correction module 74 is used to perform training correction on the current input information and the current scene reconstruction model according to the loss between the rendered image under the training perspective parameter and the training image under the training perspective parameter until the trained image is obtained. The input information and the scene reconstruction model.

In practical applications, the indoor scene reconstruction device can be implemented through a computer program, such as application software, etc.; or it can also be implemented as a medium storing relevant computer programs, such as a U disk, cloud disk, etc.; or it can also be integrated Or implemented by physical devices installed with relevant computer programs, such as chips, servers, etc.

Specifically, a camera is used to shoot in the room to obtain RGB training images; based on structure from motion (SfM) technology such as Colmap, the training perspective parameters corresponding to each training image are obtained based on the training images, and a representation of the indoor scene is established. Structured 3D scene point cloud.

In practical applications, the viewing angle parameters can be camera parameters, which include camera intrinsic parameters and camera extrinsic parameters; among them, camera intrinsic parameters include intrinsic parameter matrices, and camera extrinsic parameters represent the camera pose, including rotation matrix and translation vector. Based on a certain viewing angle parameter, the corresponding spatial position of each pixel in the world coordinate system in the image obtained by the corresponding camera imaging under the viewing angle parameter can be uniquely determined.

Starting from the origin of the training perspective, a ray is emitted as a ray to the position corresponding to each pixel in the training image under the training perspective parameters, and multiple training sampling points are selected on each ray; among them, the origin of the training perspective, the pixel The corresponding position points and scene point clouds are located in the same coordinate system, and each ray passes through the scene point cloud; in practical applications, the origin of the training perspective can be the optical center of the camera, and its spatial position can be determined based on the training perspective parameters. And, based on semantic prediction technology, select multiple training point cloud points within a predetermined range near each training sampling point as a training point set, and obtain the current input information of the training point set; where the input information includes spatial location, neural features and ray direction.

Input the current input information of the training point set into the current scene reconstruction model to obtain the color and light field intensity of the training point set output by the scene reconstruction model; in practical applications, the scene reconstruction model can include a light field intensity decoder and a color decoder. , specific parameters can be selected according to actual production needs; according to the color and light field intensity of the training point set, the color and light field intensity of the training sampling point are obtained through interpolation; according to the color and light field intensity of the training sampling point, the training perspective parameters are obtained through rendering The color of each pixel in the rendered image below is used to generate a rendered image.

According to the training images and rendering images under the training perspective parameters, the input information of the point cloud points and the scene reconstruction model are corrected by calculating the loss until the input information of the trained point cloud points and the scene reconstruction model are obtained.

In practical applications, the collection of training images needs to meet certain requirements. In one example, the training images under the training perspective parameters meet the following conditions: the training image contains the room surface; any two adjacent training perspective parameters The overlap between training images is no less than thirty percent of the entire image size.

Specifically, the room surface features are included in at least one training image, and two consecutively shot training images under adjacent training perspective parameters overlap, and the size of the overlapping area is not less than 30% of the entire image size, thereby ensuring that the collected training images Images can achieve better training results.

There are many ways to determine the training point set. In one example, the sampling module 72 can be used to:

Based on semantic prediction technology, the semantic information of each pixel point in the training image under the training perspective parameter and the semantic information of each point cloud point in the scene point cloud are obtained; the semantic information of the pixel point includes the object category, and the The semantic information of point cloud points includes object categories and corresponding probability values;

For each training sampling point, obtain all point cloud points in the scene point cloud located within a predetermined range around the training sampling point; and calculate the selection probability of each point cloud point within the predetermined range around the training sampling point, according to In order of selection probability from high to low, a preset value of point cloud points is selected as the training point set; wherein, calculating the selection probability includes: if the object category of the point cloud point corresponds to the training sampling point The object category of the pixel points is the same, then the selection probability of the point cloud point is one; if the object category of the point cloud point is different from the object category of the pixel point corresponding to the training sampling point, then the selection probability of the point cloud point is The probability is one minus the probability value corresponding to the object category of the pixel corresponding to the point cloud point.

Specifically, the training image and scene point cloud are input into the semantic prediction network to obtain the semantic information of each pixel in the training image, including the predicted object category of the pixel, and the semantic information of each point cloud in the scene point cloud, including the point cloud. The predicted object category and corresponding probability value of the point.

For each training sampling point, obtain all point cloud points within a predetermined range around it, calculate the selection probability of each point cloud point, and select a predetermined value of point cloud points in order from high to low according to the selection probability as a training point set.

Among them, for a point cloud point, the calculation formula for its selection probability p _chosen is as follows:

in For the predicted object category of the point cloud points output by the semantic prediction network, let the predicted object category of the ray where the current training sampling point is located/> The predicted object category of the pixel corresponding to the position point through which the ray passes Same, let the predicted object category of the ray /> Corresponding probability value/> The predicted object category of the pixel corresponding to the position point where the ray passes/> Corresponding probability value/> same.

By making the probability that the predicted object category of the point cloud point is the same as the predicted object category of the ray is positively correlated with the selection probability of the point cloud point, and selecting the point cloud point with a high probability as the training point set, the training point cloud point and the training are improved The probability that the object categories corresponding to the sampling points are the same makes the color and light field intensity of the training sampling points obtained based on the training point cloud point parameters more accurate.

In one example, the neural features include first neural features and second neural features, and the sampling module 72 can also be used to:

Input all the training images and the scene point cloud into the semantic prediction network to obtain the first neural features of each point cloud point in the scene point cloud;

The training image corresponding to the origin of the training perspective closest to the point cloud point is obtained, and the training image is input into the convolutional neural network to obtain the second neural feature of the point cloud point.

Specifically, the neural features of the point cloud points may include first neural features and second neural features. Obtaining the first neural features of the point cloud points includes: inputting all training images and scene point clouds into the semantic prediction network to obtain the first neural features of each point cloud point in the scene point cloud. Obtaining the second neural feature of the point cloud point includes: acquiring a training image corresponding to the origin of the training perspective closest to the point cloud point, inputting the training image into the convolutional neural network, and obtaining the second neural feature of the point cloud point. The semantic prediction network is the same as the above-mentioned semantic prediction network. In practical applications, the training images and scene point clouds are input into the semantic prediction network, and the second neural features and predictions of each point cloud point in the scene point cloud output by the semantic prediction network can be obtained. Object categories and corresponding probability values.

By convolving the training image corresponding to the origin of the training perspective closest to the point cloud point and semantically predicting the semantic information of the point cloud point as the neural feature of the point cloud point, the structural information of the point cloud point other than the spatial position is obtained. Semantic information makes the input information of the point cloud point input scene reconstruction model richer, thereby obtaining more accurate point cloud point color and light field intensity.

Based on the corrected input information and scene reconstruction model, indoor new perspective scenes can be reconstructed. In one example, the device further includes a reconstruction module, which can be used to:

Obtain the target perspective parameters of the scene to be reconstructed; and determine virtual pixels according to the target perspective parameters;

On the ray starting from the origin of the target perspective and passing through the position point corresponding to each of the virtual pixel points in the scene point cloud, select a plurality of the position points as the target sampling points; and, determine that the position points are located at the plurality of Point cloud points in the scene point cloud within a predetermined range near the target sampling point are used as a target point set, and the input information of the target point set is obtained;

Input the input information of the target point set into the scene reconstruction model to obtain the color and light field intensity of the target point set output by the scene reconstruction model; and, according to the color and light field intensity of the target point set Field intensity: obtain the color and light field intensity of the target sampling point through interpolation; and obtain the rendered image under the target viewing angle parameters through rendering according to the color and light field intensity of the target sampling point.

Specifically, after determining the target perspective of the scene to be reconstructed, the target perspective parameters are obtained. According to the target perspective parameters, determine the target perspective origin and the corresponding position point of the virtual pixel point in the coordinate system of the scene point cloud. Starting from the target perspective origin, pass through the scene point cloud and pass through the location point corresponding to each virtual pixel point to make a ray. In the ray Select target sampling points within a predetermined area, and select point cloud points in the scene point cloud within a predetermined range around each target sampling point to obtain a target point set.

Input the trained input information of the target point set into the trained scene reconstruction model to obtain the color and light field intensity of the target point set; interpolate the color and light field intensity of the target point set to obtain the color and light field intensity of the corresponding target sampling point. ; Render according to the color and light field intensity of the target sampling point to obtain a rendered image from a new perspective.

Based on the trained input information and the trained scene reconstruction model, the new perspective image of the indoor scene can be rendered to generate a rendering image with better effect.

There are many ways to select the target point set. In one example, the reconstruction module is specifically used to:

All point cloud points within a predetermined range near the plurality of target sampling points are obtained, and a predetermined value of point cloud points closest to each of the sampling points are selected as the target point set.

Specifically, for each target sampling point, all point cloud points within a predetermined range around it are obtained, and a predetermined number of point cloud points are selected in order of distance from near to far as a set of target points.

In the indoor scene reconstruction device provided by this application, image collection is performed indoors to obtain training images; and, based on the motion recovery structure technology, the training perspective parameters corresponding to each training image are obtained and scene points are established based on the training images. Cloud; wherein, the scene point cloud is a point cloud that represents indoor scene structure information; starting from the origin of the training perspective and passing through the position point corresponding to each pixel point in the training image under the training perspective parameter in the scene point cloud On the ray, select a plurality of the position points as training sampling points; and, based on semantic prediction technology, determine the point cloud points in the scene point cloud located within a predetermined range near the multiple training sampling points as a training point set, And obtain the current input information of the training point set; wherein the input information includes spatial position, neural characteristics and ray direction; input the current input information of the training point set into the current scene reconstruction model to obtain the The color and light field intensity of the training point set output by the scene reconstruction model; and, according to the color and light field intensity of the training point set, obtain the color and light field intensity of the training sampling point through interpolation; and, according to the training The color and light field intensity of the sampling point are rendered to obtain the rendered image under the training perspective parameters; according to the loss between the rendering image under the training perspective parameters and the training image under the training perspective parameters, the current input The information and the current scene reconstruction model are trained and corrected until the trained input information and the scene reconstruction model are obtained. By selecting a set of training points around the training sampling points based on semantic prediction technology, this solution improves the possibility that the object category of the point cloud point is the same as the object category of the corresponding training sampling point, so that the point cloud point color and light field intensity interpolation can be obtained The color and light field intensity of the training sampling points are more accurate, thereby improving the quality of the generated rendered images, and the input information of the point cloud points and the scene reconstruction model can be better trained and corrected based on the training images and corresponding rendered images. , reducing the requirement on the number of training images, thus solving the problem of poor quality of generated indoor scene rendering images when the number of training images is small.

Embodiment 3

Figure 8 is a schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. As shown in Figure 8, the electronic device includes:

The electronic device also includes a processor (processor) 291 and a memory (memory) 292; it may also include a communication interface (Communication Interface) 293 and a bus 294. Among them, the processor 291, the memory 292, and the communication interface 293 can communicate with each other through the bus 294. Communication interface 293 may be used for information transmission. The processor 291 can call logical instructions in the memory 292 to execute the methods of the above embodiments.

In addition, the above-mentioned logical instructions in the memory 292 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

As a computer-readable storage medium, the memory 292 can be used to store software programs, computer-executable programs, such as program instructions/modules corresponding to the methods in the embodiments of the present disclosure. The processor 291 executes software programs, instructions and modules stored in the memory 292 to execute functional applications and data processing, that is, to implement the methods in the above method embodiments.

The memory 292 may include a stored program area and a stored data area, where the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created according to the use of the terminal device, etc. In addition, the memory 292 may include high-speed random access memory and may also include non-volatile memory.

Embodiments of the present disclosure provide a non-transitory computer-readable storage medium. Computer-executable instructions are stored in the computer-readable storage medium. When the computer-executable instructions are executed by a processor, they are used to implement the methods described in the previous embodiments. method.

Embodiment 4

An embodiment of the disclosure provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method provided by any of the above embodiments of the disclosure is implemented.

Other embodiments of the present application will be readily apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary technical means in the technical field that are not disclosed in this application. . It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

It is to be understood that the present application is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims (10)

1. An indoor scene reconstruction method, characterized by including: Image collection is performed indoors to obtain training images; and, based on the motion recovery structure technology, the training perspective parameters corresponding to each training image are obtained according to the training images and a scene point cloud is established; wherein, the scene point cloud is Point clouds representing structural information of indoor scenes; On the ray starting from the origin of the training perspective and passing through the position point corresponding to each pixel in the training image under the training perspective parameter in the scene point cloud, select a plurality of the position points as training sampling points; and, based on semantics The prediction technology determines the point cloud points in the scene point cloud located within a predetermined range near the multiple training sampling points as a training point set, and obtains the current input information of the training point set; wherein the input information includes Spatial location, neural characteristics, and ray direction; Input the current input information of the training point set into the current scene reconstruction model to obtain the color and light field intensity of the training point set output by the scene reconstruction model; and, according to the color and light field intensity of the training point set Intensity, obtain the color and light field intensity of the training sampling point through interpolation; and, obtain the rendered image under the training perspective parameter by rendering according to the color and light field intensity of the training sampling point; According to the loss between the rendered image under the training perspective parameters and the training image under the training perspective parameters, the current input information and the current scene reconstruction model are trained and corrected until the trained input information and the current scene reconstruction model are obtained. The scene reconstruction model. 2. The method according to claim 1, characterized in that the point cloud points in the scene point cloud located within a predetermined range near the plurality of training sampling points are determined as a training point set based on semantic prediction technology, include: Based on semantic prediction technology, the semantic information of each pixel point in the training image under the training perspective parameter and the semantic information of each point cloud point in the scene point cloud are obtained; the semantic information of the pixel point includes the object category, and the The semantic information of point cloud points includes object categories and corresponding probability values; For each training sampling point, obtain all point cloud points in the scene point cloud located within a predetermined range around the training sampling point; and calculate the selection probability of each point cloud point within the predetermined range around the training sampling point, according to In order of selection probability from high to low, a preset value of point cloud points is selected as the training point set; wherein, calculating the selection probability includes: if the object category of the point cloud point corresponds to the training sampling point The object category of the pixel points is the same, then the selection probability of the point cloud point is one; if the object category of the point cloud point is different from the object category of the pixel point corresponding to the training sampling point, then the selection probability of the point cloud point is The probability is one minus the probability value corresponding to the object category of the pixel corresponding to the point cloud point. 3. The method according to claim 1, wherein the neural features include first neural features and second neural features, and obtaining the current input information of the training point set includes: Input all the training images and the scene point cloud into the semantic prediction network to obtain the first neural features of each point cloud point in the scene point cloud; The training image corresponding to the origin of the training perspective closest to the point cloud point is obtained, and the training image is input into the convolutional neural network to obtain the second neural feature of the point cloud point. 4. The method according to claim 1, characterized in that, the method further comprises: Obtain the target perspective parameters of the scene to be reconstructed; and determine virtual pixels according to the target perspective parameters; On the ray starting from the origin of the target perspective and passing through the position point corresponding to each of the virtual pixel points in the scene point cloud, select a plurality of the position points as the target sampling points; and, determine that the position points are located at the plurality of Point cloud points in the scene point cloud within a predetermined range near the target sampling point are used as a target point set, and the input information of the target point set is obtained; Input the input information of the target point set into the scene reconstruction model to obtain the color and light field intensity of the target point set output by the scene reconstruction model; and, according to the color and light field intensity of the target point set Field intensity: obtain the color and light field intensity of the target sampling point through interpolation; and obtain the rendered image under the target viewing angle parameters through rendering according to the color and light field intensity of the target sampling point. 5. The method according to claim 4, wherein determining the point cloud points in the scene point cloud within a predetermined range near the plurality of target sampling points as a target point set includes: Obtain all point cloud points in the scene point cloud located within a predetermined range near the plurality of target sampling points, and select a predetermined number of the point cloud points closest to each of the sampling points as the target points. gather. 6. The method according to any one of claims 1 to 5, characterized in that the training image satisfies the following conditions: the training image contains a room surface; training images under any two adjacent training perspective parameters The overlap between them is no less than thirty percent of the entire image size. 7. An indoor scene reconstruction device, characterized by comprising: An acquisition module is used to collect images indoors to obtain training images; and, based on the motion recovery structure technology, obtain the training perspective parameters corresponding to each of the training images according to the training images and establish a scene point cloud; wherein, The scene point cloud described above is a point cloud that represents the structural information of the indoor scene; A sampling module, configured to select a plurality of the position points as training sampling points on the ray starting from the origin of the training perspective and passing through the position point corresponding to each pixel in the training image under the training perspective parameter in the scene point cloud. ; And, based on semantic prediction technology, determine the point cloud points in the scene point cloud located within a predetermined range near the multiple training sampling points as a training point set, and obtain the current input information of the training point set; wherein, The input information includes spatial position, neural characteristics and ray direction; A rendering module, configured to input the current input information of the training point set into the current scene reconstruction model, and obtain the color and light field intensity of the training point set output by the scene reconstruction model; and, according to the training point The color and light field intensity of the set are interpolated to obtain the color and light field intensity of the training sampling point; and, based on the color and light field intensity of the training sampling point, the rendered image under the training perspective parameter is obtained through rendering; A correction module, configured to train and correct the current input information and the current scene reconstruction model according to the loss between the rendered image under the training perspective parameters and the training image under the training perspective parameters until a trained The input information and the scene reconstruction model. 8. The device according to claim 7, characterized in that the sampling module is specifically used for: Based on semantic prediction technology, the semantic information of each pixel point in the training image under the training perspective parameter and the semantic information of each point cloud point in the scene point cloud are obtained; the semantic information of the pixel point includes the object category, and the The semantic information of point cloud points includes object categories and corresponding probability values; For each training sampling point, obtain all point cloud points in the scene point cloud located within a predetermined range around the training sampling point; and calculate the selection probability of each point cloud point within the predetermined range around the training sampling point, according to In order of selection probability from high to low, a preset value of point cloud points is selected as the training point set; wherein, calculating the selection probability includes: if the object category of the point cloud point corresponds to the training sampling point The object category of the pixel points is the same, then the selection probability of the point cloud point is one; if the object category of the point cloud point is different from the object category of the pixel point corresponding to the training sampling point, then the selection probability of the point cloud point is The probability is one minus the probability value corresponding to the object category of the pixel corresponding to the point cloud point. 9. An electronic device, characterized by comprising: a processor, and a memory communicatively connected to the processor; The memory stores computer execution instructions; The processor executes computer-executable instructions stored in the memory to implement the method according to any one of claims 1-6. 10. A computer-readable storage medium, characterized in that computer-executable instructions are stored in the computer-readable storage medium, and when executed by a processor, the computer-executable instructions are used to implement any one of claims 1-6. method described in the item.