CN114913397A - Modal deficiency-oriented graph representation learning method, system, device and storage medium - Google Patents

Abstract

The invention discloses a modal deficiency-oriented graph representation learning method, a modal deficiency-oriented graph representation learning system, a modal deficiency-oriented graph representation learning device and a storage medium, and relates to the technical field of computers. According to the method, the characteristic interaction is carried out on the representation matrixes of various modes in the resource hyper-vertex of the modal missing hyper-graph so as to perfect modal data, and then the characteristic fusion matrix of the resource hyper-vertex is obtained, then the characteristic fusion matrix interaction between the resource hyper-vertices is carried out so as to perfect the missing modal data in the resource hyper-vertex, and then the modal aggregation operation is carried out on the resource hyper-vertex after the mode is completed, so that the characteristic extraction vector of the resource hyper-vertex is obtained, the multi-modal hyper-graph is updated, and then the multi-modal hyper-graph is input into the representation model so as to obtain a relatively accurate representation result.

Description

Method, system, device and storage medium for graph representation learning oriented to missing modality

technical field

The present invention relates to the field of computer technology, and in particular, to a method, system, device and storage medium for graph representation learning for modal deletion.

Background technique

With the advancement of data collection technology, multi-modal data has increased dramatically, and the integration of multi-modal data can improve the performance of machine learning in various application scenarios. However, how to integrate modal-missing data is still a challenging problem. At present, most graph neural networks assume that all the features of the vertices are available, but in fact, only part of the features of the vertices may be available. For example, in social networks, some users are unwilling to fill in information such as age and gender. resulting in the absence of such modal data.

Faced with the lack of modal data, strategies such as deleting incomplete data samples or imputing missing modalities are often used to deal with this problem. However, data deletion will significantly reduce the amount of training data. Large-scale samples can lead to overfitting of deep learning models, imputation-based methods introduce additional noise in the original data, which negatively affects the performance of the learned model, and is sometimes associated with complex auxiliary models , which increases the computational difficulty. To sum up, the accuracy of the results obtained by the current data integration methods for large-scale missing modalities is low.

SUMMARY OF THE INVENTION

The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a method, system, device and storage medium for modal-missing graph representation learning, which can effectively integrate modal-missing graph data and provide more accurate graph representation results.

On the one hand, an embodiment of the present invention provides a modal deletion-oriented graph representation learning method, including the following steps:

obtaining a modal missing hypergraph, wherein the modal missing hypergraph includes a plurality of resource hypervertices, and the resource hypervertices include representation matrices of multiple modalities;

performing feature interaction on the representation matrices of a plurality of the modalities in each of the resource super-vertices to obtain a feature fusion matrix of each of the resource super-vertices;

For each resource super vertex, interactively fuse the feature fusion matrix of the neighbor resource super vertex with the feature fusion matrix of the current resource super vertex, and perform modal aggregation on the feature fusion matrix of the current resource super vertex to obtain the features of the current resource super vertex extract vector;

Update the missing modal hypergraph according to the feature extraction vector of each resource hypervertex to obtain a multimodal hypergraph;

The multimodal hypergraph is input into a graph representation model to obtain a graph representation result.

According to some embodiments of the present invention, the graph representation model comprises a two-layer self-attention residual connected graph neural network, the graph neural network comprising a filter bank constructed based on a compact frame, the filter bank comprising low-pass filtering filter, a first high-pass filter, and a second high-pass filter.

According to some embodiments of the present invention, the feature interaction fusion of the representation matrices of the multiple modalities in each of the resource super-vertices, to obtain the feature fusion matrix of each of the resource super-vertices, includes the following steps:

Perform feature fusion in the modal according to the representation matrix of each modal in the resource super-vertex to obtain a first fusion vector;

The second fusion vector is obtained by interacting the features between the multiple modes in the resource super vertex according to the representation matrix of the multiple modes in the resource super vertex;

The feature fusion matrix is obtained according to a plurality of the first fusion vectors and a plurality of the second fusion vectors, wherein the feature fusion matrix includes a plurality of modal vectors.

According to some embodiments of the present invention, performing feature fusion in the modality according to the representation matrix of each modality in the resource super-vertex to obtain a first fusion vector includes the following steps:

determining the eigenvalues of the modality according to the representation matrix of the modality;

Perform a pairwise inner product on all eigenvectors in the representation matrix of the modal to obtain a Gram matrix;

The first fusion vector is obtained according to the eigenvalue and the Gram matrix.

According to some embodiments of the present invention, obtaining the feature fusion matrix according to a plurality of the first fusion vectors and a plurality of the second fusion vectors includes the following steps:

Inputting the first fusion vector and the second fusion vector into a tensor-type random configuration neural network to perform vector dimension conversion and splicing to obtain the feature fusion matrix.

According to some embodiments of the present invention, the interactive fusion of the feature fusion matrix of the neighbor resource super-vertex and the feature fusion matrix of the current resource super-vertex includes the following steps:

Determine the missing modality of the current resource supervertex;

The feature of the current resource super vertex is determined according to the first influence weight of the neighbor resource super vertex on the missing modality and the modal vector corresponding to the missing modality in the feature fusion matrix of the neighbor resource super vertex the missing modal vectors in the fusion matrix;

Update the missing modality in the feature fusion matrix of the current resource super-vertex according to the second influence weight of the modal vector different from the missing modality in the feature fusion matrix of the neighbor resource super-vertex on the missing modality vector.

According to some embodiments of the present invention, the modal aggregation of the feature fusion matrix of the current resource super vertex to obtain the feature extraction vector of the current resource super vertex includes the following steps:

determining the weight coefficients of multiple modal vectors in the feature fusion matrix;

multiplying each of the modal vectors by the weighting coefficient to obtain a plurality of weighted modal vectors;

A feature extraction vector of the current resource super-vertex is obtained by adding up a plurality of the weighted modal vectors.

On the other hand, an embodiment of the present invention also provides a modal deletion-oriented graph representation learning system, including:

a first module, configured to obtain a modality-missing hypergraph, wherein the modality-missing hypergraph includes multiple resource hypervertices, and the resource hypervertices include representation matrices of multiple modes;

The second module is used to perform feature interaction on the representation matrices of multiple said modalities in each of the resource super-vertices to obtain a feature fusion matrix of each of the resource super-vertices;

The third module is used to interactively fuse the feature fusion matrix of the neighbor resource super vertex with the feature fusion matrix of the current resource super vertex for each resource super vertex, and modally aggregate the feature fusion matrix of the current resource super vertex to obtain The feature extraction vector of the current resource super vertex;

The fourth module is used to update the modal missing hypergraph according to the feature extraction vector of each resource hypervertex to obtain a multimodal hypergraph;

The fifth module is used for inputting the multimodal hypergraph into a graph representation model to obtain a graph representation result.

On the other hand, an embodiment of the present invention also provides a modal absence-oriented graph representation learning device, including:

at least one processor;

at least one memory for storing at least one program;

When the at least one program is executed by the at least one processor, the at least one processor implements the modal absence-oriented graph representation learning method as described above.

On the other hand, an embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause a computer to execute the model-oriented method described above. Graphs with missing states represent learning methods.

The above technical solution of the present invention has at least one of the following advantages or beneficial effects: by performing feature interaction between the representation matrices of multiple modalities in the resource hypervertex of the modal-missing hypergraph to improve the modal data, and then obtain the resource hypervertex feature fusion matrix, and then perform the feature fusion matrix interaction between resource super-vertices to improve the missing modal data in the resource super-vertices, and then perform modal aggregation operations on the resource super-vertices after modal improvement, so as to obtain resources The feature extraction vector of the super vertices and update the multi-modal hypergraph, and then input the multi-modal hypergraph into the graph representation model to obtain a more accurate graph representation result.

Description of drawings

FIG. 1 is a flowchart of a modal deletion-oriented graph representation learning method provided by an embodiment of the present invention;

2 is a schematic diagram of a modal deletion-oriented graph representation learning system provided by an embodiment of the present invention;

3 is a schematic diagram of a modal deletion-oriented graph representation learning device provided by an embodiment of the present invention;

4 is a schematic diagram of a process of constructing a modal deletion hypergraph provided by an embodiment of the present invention;

5 is a schematic diagram of an interaction process between resource super-vertex feature representation and internal features provided by an embodiment of the present invention;

6 is a schematic diagram of a model processing process of a modal-missing hypergraph in a three-layer graph attention network mechanism provided by an embodiment of the present invention;

7 is a schematic diagram of the construction process of a graph neural network based on a tight frame provided by an embodiment of the present invention;

FIG. 8 is a schematic diagram of a processing process of a graph representation model of a multi-modal hypergraph in a graph neural network based on two-layer self-attention residual connections provided by an embodiment of the present invention.

Detailed ways

The following describes in detail the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, only used to explain the present invention, and should not be construed as a limitation of the present invention.

In the description of the present invention, it should be understood that the azimuth description, for example, the azimuth or position relationship indicated by up, down, left, right, etc., is based on the azimuth or position relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention. The invention and simplified description do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operate in a particular orientation, and therefore should not be construed as limiting the invention.

In the description of the present invention, if the first, second, etc. are described only for the purpose of distinguishing technical features, it should not be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or implicitly indicating The order of the indicated technical features.

An embodiment of the present invention provides a modal absence-oriented graph representation learning method. Referring to FIG. 1 , the modal absence oriented graph representation learning method according to an embodiment of the present invention includes but is not limited to step S110 , step S120 , step S130 , and step S130 . S140 and step S150.

Step S110, acquiring a modality-missing hypergraph, wherein the modality-missing hypergraph includes multiple resource hypervertices, and the resource hypervertex includes representation matrices of multiple modes;

包括用户u₁包括模态i₁的数据，资源超顶点

包括用户u₂和模态i₁、i₂的数据，资源超顶点

包括用户u₃和模态i₁、i₂、i₃的数据，资源超顶点

用户u₄和模态i₂、i₃的数据。为了有效地表示超顶点之间的高阶连接，使用超边E共享资源超顶点的相似信息并显示模态缺失超图中的资源超顶点之间的关系，根据资源超顶点和超边构建包含多个资源超顶点的模态缺失超图G。In some embodiments, in a real social network, the interaction between users and resources generates data, but not all users will generate resources with complete modalities, and there may be cases where user modalities are missing. Based on this, construct The graph structure is a modal-missing hypergraph. Exemplarily, referring to FIG. 4 , three modal data are defined as i ₁ , i ₂ , and i ₃ respectively. User u ₁ generates resources of modal i ₁ , and user u ₂ generates resources of modal i ₁ , i ₂ , User u ₃ generates resources of modalities i ₁ , i ₂ , i ₃ , and user u ₄ generates resources of modal i ₂ , i ₃ . Therefore, the resource super-vertex corresponding to the user is constructed, and the modalities are obtained according to the relationship between users Missing graph, resource hypervertex

include data for user u ₁ including modality i ₁ , resource supervertex

Contains data for user u ₂ and modalities i ₁ , i ₂ , resource hypervertex

Contains data for user u ₃ and modalities i ₁ , i ₂ , i ₃ , resource hypervertex

Data for user u ₄ and modalities i ₂ , i ₃ . In order to efficiently represent higher-order connections between hypervertices, hyperedges E are used to share similar information of resource hypervertices and show the relationship between resource hypervertices in the modal-missing hypergraph. Modal missing hypergraph G for multiple resource hypervertices.

Further, referring to FIG. 5 , the single modal data in the modal absence hypermap is represented, and a simple MLP (Multilayer Perceptron) is used to construct an encoder, wherein the encoder parameters corresponding to each modal are not shared, so that Representation matrix for each mode. Define f _m (·; θ _m ) as the MLP embedding network of modality m, θ _m is a parameter that can be trained and learned, and the representation matrix of modality m for each resource super-vertex v _i is shown in formula (1) :

F_m为模态m的嵌入维度。in,

F _m is the embedding dimension of modality m.

Step S120, performing feature interaction on the representation matrices of multiple modalities in each resource super-vertex to obtain a feature fusion matrix of each resource super-vertex;

In some embodiments, after obtaining the representation matrix of the modalities, in order to narrow the "semantic gap" between the abstract representations of various modalities within the resource super-vertex, it is necessary to implement a single-modal feature interactive encoding and various modalities within the resource super-vertex. Deep interactions for new encodings of features between modalities.

Gram矩阵即格拉姆矩阵，n维欧式空间中任意k个向量y₁，y₂，...y_k之间两两的内积所组成的矩阵，即为这k个向量的Gram矩阵，Gram矩阵是一个对称矩阵，Gram矩阵用于度量矩阵中各个维度本身的特性以及各个维度之间的关系，矩阵中的特征向量内积之后得到的多尺度矩阵中，对角线元素提供了不同特征向量本身的信息，其余元素提供了不同特征向量之间的相关信息。采用Gram矩阵变换的第一融合向量既能体现出有特征的个数，又能体现出不同特征间的紧密程度，其主要的表示形式如公式(2)所示：Specifically, referring to FIG. 5, first, for the representation matrix of a single modality, use the Gram matrix to calculate the relationship between the features, and obtain the first fusion vector corresponding to the single modality respectively

Gram matrix is Gram matrix, the matrix formed by the inner product of any k vectors y ₁ , y ₂ , ... y _k in n-dimensional Euclidean space, is the Gram matrix of these k vectors, Gram The matrix is a symmetric matrix. The Gram matrix is used to measure the characteristics of each dimension in the matrix and the relationship between each dimension. In the multi-scale matrix obtained after the inner product of the eigenvectors in the matrix, the diagonal elements provide different eigenvectors. information of itself, and the remaining elements provide relevant information between different feature vectors. The first fusion vector transformed by Gram matrix can not only reflect the number of features, but also reflect the degree of closeness between different features. Its main representation is shown in formula (2):

再通过计算三个不同模态的表示矩阵之间的外积得到表示三者信息交互的三阶张量

以此类推即得到有限个不同阶次的张量，通过各个模态的表示矩阵之间的外积进行深度交互，得到多个第二融合向量

Secondly, for each pair of modalities or between more than two modalities in the resource super-vertex, there may be higher-order modal interactions. For the representation matrices of different modalities, the outer product of the representation matrices between the modalities can be used. Get a second-order tensor representing the information interaction between the two

Then, by calculating the outer product between the representation matrices of the three different modes, a third-order tensor representing the information interaction of the three is obtained.

By analogy, a limited number of tensors of different orders are obtained, and through the deep interaction between the representation matrices of each mode, a plurality of second fusion vectors are obtained.

可以从每个因子S上学到一种多模态互动及与其相关的一系列的信息。资源超顶点内的特征互补信息由许多因子组成，每个因子用一个F’维向量表示，在进行资源超顶点内的各模态特征交互的计算过程如下：Further, assume that H( ) represents a power set operation and for each subset

A multimodal interaction and a series of information related to it can be learned from each factor S. The feature complementary information in the resource super-vertex is composed of many factors, and each factor is represented by an F'-dimensional vector. The calculation process of each modal feature interaction in the resource super-vertex is as follows:

的计算过程如公式组(3)所示：If there is only one element in S, it means that the first fusion vector of a certain mode m∈S is being calculated, the first fusion vector

The calculation process is shown in formula group (3):

b_m∈R^F′，

b_m，m∈R^F′，均为神经网络g_m(·)和g_m，m(·)的可训练参数，

为表示矩阵

对应的Gram矩阵，

表示模态m的特征值，该特征值可以为模态m的特征均值。in,

b _m ∈ R ^F ′,

b _{m, m} ∈ R ^F ′, both are trainable parameters of neural networks g _m ( ) and g _{m, m} ( ),

to represent the matrix

The corresponding Gram matrix,

Represents the eigenvalue of mode m, which can be the eigenvalue of mode m.

之间的跨模态交互信息的第二融合向量。第二融合向量

的计算过程如公式组(4)所示：If there is more than one element in S, it means that different modes are being calculated

A second fusion vector of cross-modal interaction information between. second fusion vector

The calculation process is shown in formula group (4):

求得S中的单一模态m的表示矩阵

的交叉验证，进而得到S中相关模态表示矩阵的|S|层交叉验证

最后再将

输入图神经网络g_S(·)中进行训练，b_S和U_s均为图神经网络g_S(·)的可训练参数。Among them, the first

Find the representation matrix for a single mode m in S

, and then obtain the |S| layer cross-validation of the relevant modal representation matrix in S

Finally, the

The input graph neural network g _S ( ) is used for training, and b _S and U _s are both trainable parameters of the graph neural network g _S ( ).

在本发明实施例中的图5，可以得到7种融合向量，分别为

根据多种融合向量得到资源超顶点的进行模态特征交互后的特征融合矩阵，其中，特征融合矩阵包括多个模态向量。After obtaining the first fusion vector and the second fusion vector, a random configuration neural network supporting tensor input is used to realize the conversion of tensors of different orders to vector representations of the same dimension. Through the above process, a plurality of first fusion vectors and a plurality of second fusion vectors with different dimensions and different orders are obtained. Therefore, the plurality of first fusion vectors and the plurality of second fusion vectors are input into the support tensor input Randomly configure the neural network to obtain a fusion vector represented by a uniform dimension

In FIG. 5 in the embodiment of the present invention, 7 kinds of fusion vectors can be obtained, which are

A feature fusion matrix of the resource super-vertex after modal feature interaction is obtained according to various fusion vectors, wherein the feature fusion matrix includes a plurality of modal vectors.

Step S130, for each resource super vertex, interactively fuse the feature fusion matrix of the neighbor resource super vertex and the feature fusion matrix of the current resource super vertex, and perform modal aggregation on the feature fusion matrix of the current resource super vertex to obtain the current resource super vertex. Feature extraction vector of vertices;

In some embodiments, since the lack of modalities in some resource supervertices will cause the modal information imbalance between resource supervertices, model completion and complete resource supervertices based on the three-layer graph attention network mechanism can be designed. Missing modal data. For the construction of the model based on the three-layer graph attention network mechanism, the first influence weight of the missing modality of the current resource supervertex on the neighbor supervertices with the same modality as the missing modality is trained in the first layer graph attention network , trained in the second-layer graph attention network for neighbor resource supervertices of different modalities from the missing modality on the missing modality of the current resource supervertex, the second influence weights, trained in the third-layer graph attention network Weight coefficients for different modalities in the resource supervertex. Through the above-mentioned model based on the three-layer graph attention network mechanism, the missing modalities of the current resource super-vertex can be completed based on the modalities of the same neighbor resource super-vertices as the missing modalities, and then based on the same neighbors as the missing modalities. The modality of the resource supervertex completes the missing modality of the current resource supervertex, and then aggregates multiple modes in the resource supervertex to obtain the feature extraction vector of the resource supervertex.

Exemplarily, the solution process of the attention network is described by taking the resource super-vertices as an example.

The data obtained by the modal-missing hypergraph through a tensor-type random configuration network is h={h ₁ , h ₂ , h ₃ ,...h _{N }} , hi ∈ R ^F , N represents the number of resource super vertices, F Represents the number of features in the resource super-vertex, the size of the matrix h is N×F, the matrix h represents the features of all resource super-vertices, R represents the feature of a resource super-vertex, and the size of R is F×1. The output data through the attention network can be expressed as {h ₁ ', h ₂ ', h ₃ ', ... h _N ' _} , hi '∈RF , ^F ' represents the output feature vector dimension of the resource super-vertex, The weight matrix W is set to represent the relationship between the input and output features of the attention network.

The self-attention mechanism is implemented for each resource super vertex, then the importance of the resource super vertex v _j to the resource super vertex v _i is shown in formula (5), where the relation function is defined as a:

e _ij =a(Wh _i , Wh _j ); (5)

In order to make the attention coefficient easier to calculate and compare, a softmax function is introduced to regularize all resource super-vertices v _j adjacent to resource super-vertices v _i to obtain the attention coefficients. The attention coefficient α _ij is calculated as formula (6 ) as shown:

After the above attention coefficient passes through the LeakyReLU function of the output layer of the attention network, the complete attention coefficient obtained is shown in the following formula (7), where a is the weight matrix between the connection layers in the attention network.

Further, according to the above-mentioned attention coefficient between resource super-vertices, the output feature of each resource super-vertex can be predicted as shown in formula (8):

In this embodiment, the above-mentioned method for solving the attention network is applied to the model of the three-layer graph attention network mechanism in this embodiment, so as to realize the completion and improvement of the missing modalities of the resource super-vertex and the Modal aggregation to obtain feature extraction vectors of resource supervertices.

Referring to Fig. 6, first, the modal-missing hypergraph is input into the first-layer self-attention network, and the expression of the missing modality m of the resource super-vertex v _i for all the neighbor vertices of the resource super-vertex v _i is obtained as formula (9) shown:

为资源超顶点v_i的所有邻居资源超顶点v_j对于资源超顶点v_i的缺失模态m的注意力系数，

表示邻居资源超顶点v_j的模态向量m，W_mn表示模态向量m到映射低维的模态向量n的可训练权重，

表示资源超顶点v_i的邻居资源超顶点，

的计算如公式(10)所示：where σ is the activation function,

is the attention coefficient of resource super-vertex v _j for all neighbors of resource super-vertex v _i for the missing modality m of resource super-vertex v _i ,

represents the modal vector m of the neighbor resource super-vertex v _j , W _mn represents the trainable weight of the modal vector m to the modal vector n of the mapping low-dimensional,

represents the neighbor resource supervertex of resource supervertex v _i ,

The calculation of is shown in formula (10):

Among them, α _m represents the first influence weight of the resource super-vertex v _j to the resource super-vertex v _i .

k是资源超顶点v_i所具有的邻居的数量。Through the above process, it can be obtained that the resource super vertex v _i has added information

_k is the number of neighbors that the resource supervertex vi has.

Second, input the modal-missing hypergraph into the second-layer self-attention network, get all the neighbor vertices of the resource super-vertex v _i to complete the missing modal m of the resource-super-vertex v _i , and get the modal m of the resource super-vertex v _i . The expression is shown in formula (11):

表示资源超顶点v_i的低维度模态向量n，

表示模态向量n到模态向量m空间的转换，β_m，n的计算如公式(12)所示：where σ is the activation function, β _{m, n} is the attention coefficient between modality m and modality n,

represents the low-dimensional modal vector n of the resource supervertex v _i ,

Represents the transformation of the modal vector n to the modal vector m space, and the calculation of β _{m, n} is shown in formula (12):

where βm _,n represents the second influence weight of other modalities of neighboring supervertices on the missing modality.

Through the above process, all the modal information of the resource super vertex v _i is completed.

Then, the modal-missing hypergraph is input into the third-layer self-attention network to perform modal fusion on each resource super-vertex to obtain the feature extraction matrix h _i ′ of the resource super-vertex v _i , and h _i ′ is calculated as formula (13) shown:

表示资源超顶点v_i内模态向量j，N表示资源超顶点v_i内的模态向量数量。Among them, w _j represents the weight coefficient of the modal vector j and other modal vectors in the resource super-vertex v _i ,

represents the modal vector j in the resource super-vertex v _i , and N represents the number of modal vectors in the resource super-vertex v _i .

Step S140, updating the modal missing hypergraph according to the feature extraction vector of each resource hypervertex to obtain a multimodal hypergraph;

Step S150, inputting the multimodal hypergraph into the graph representation model to obtain a graph representation result.

In some embodiments, the graph representation model includes a two-layer self-attention residual connected graph neural network, the graph neural network includes a filter bank constructed based on a compact frame, the filter bank includes a low-pass filter, a first high-pass filter and a second high pass filter.

Among them, referring to Figure 7, the graph neural network is constructed in the following manner:

以此得到树形结构。For the obtained large-scale modal-missing hypergraph Γ, the features of the graph are extracted through multi-layer clustering. Refinement of large-scale modal-missing hypergraphs into subgraphs at different levels

This results in a tree structure.

According to the tree structure, an orthogonal basis corresponding to the layer is constructed at each layer of the tree. The advantage of the orthonormal basis is that it can easily represent a point in the space, which is expressed as the product of two vectors u and v in mathematical language. When u·v=0, these two vectors are perpendicular or orthogonal to each other, and u and v is also the orthonormal basis of the layer. At the same time, the sharing and expansion of the orthonormal basis can be quickly realized between the layers, so as to obtain the multi-scale compact frame transform on the graph, which is equivalent to the fast Fourier transform. (FFT) algorithmic complexity.

表示小波的基。Based on the convolution theorem, a graph convolution network based on a multi-scale compact frame is designed. The network uses a tensor-based wavelet transform to directly replace the basis of the Fourier transform F(ω), that is, the infinitely long trigonometric function basis is replaced by The finite wavelet base with attenuation can not only obtain the frequency of the signal, but also locate the time, which solves the shortcomings that the Fourier transform cannot describe the local characteristics of the signal in the time domain and has no time-frequency analysis. The wavelet transform function WT ( The expression of α, τ) is shown in formula (14), wherein, the wavelet transform has two variables: scale α and translation τ. The scale α controls the expansion and contraction of the wavelet function, the translation amount τ controls the translation of the wavelet function, f(t) represents the input graph signal,

represents the basis of the wavelet.

是r次切比雪夫多项式，

是图拉普拉斯矩阵，将其与输入特征矩阵连续相乘以产生小波系数，可表示为公式(15)：To design the compact frame transformation on the graph, the specific frame operation flow is: by giving a graph signal f with a structure (adjacency matrix) and a feature information graph, construct a low-pass and two high-pass wavelet transform matrices W _{r, j} |r=0,1,...,n; j=1,...,j}, meanwhile, define

is a Chebyshev polynomial of degree r,

is the graph Laplacian matrix, which is continuously multiplied by the input feature matrix to generate wavelet coefficients, which can be expressed as Equation (15):

The graph signal can be decomposed according to the established frame, and then reconstructed with the dual frame, while adjusting some new vectors and spatial positions in the orthogonal basis set, and setting the upper and lower bounds equal to become a compact frame. The input graph signal is inner product with the basis or frame to transform the function space to the coefficient space, and the transformed energy (the measure of the square sum of the inner product) still has an upper and lower bound greater than 0. Due to the redundancy of the frame, The expression of the low-pass and two high-pass wavelet transform function coefficients within the framework is not unique. In addition, the coefficients are calculated by the trainable network filter, and the image is compressed by the compression rate, and the correct embedded graph of the graph convolution is obtained. The frame convolution representation of the contraction activation is shown in formula (16):

λShrinkage(diag(θ)(WX'))), X'=XW; (16)

的特征矩阵，重构算子λ是分解算子W的帧变换矩阵的重新排列。通过再次使用转置对齐的变换矩阵，重构激活系数并将其发送回空间域作为卷积输出。where θ is the network filter and X is the graph

The feature matrix of , the reconstruction operator λ is the rearrangement of the frame transformation matrix of the decomposition operator W. By using the transposed aligned transformation matrix again, the activation coefficients are reconstructed and sent back to the spatial domain as the convolution output.

In this embodiment, referring to FIG. 8 , the processing procedure of the multimodal hypergraph in the graph representation model of the graph neural network based on the two-layer self-attention residual connection is as follows:

For the feature extraction vector of the multi-modal hypergraph, input it into the graph neural network of the above multi-scale framework to perform normal filtering operations, and the obtained output vector is x ⁽¹⁾ . The vector x ⁽²⁾ , the vector obtained after decomposing the matrix is denoted as x ⁽³⁾ , and the vector x ⁽¹⁾ , x ⁽²⁾ and x ⁽³⁾ are weighted and averaged to obtain the output vector x', the output vector x ' is calculated by formula (17):

Among them, i takes the value of 1, 2, 3, and w _i represents the weight of x ⁽ⁱ⁾ .

After repeating the above steps many times, the final graph representation result x is obtained through the reconstruction factor λ, so as to realize the graph representation learning of large-scale modal-missing hypergraphs.

In this embodiment, the method of wavelet analysis is proposed to be combined to design a tight frame transformation on the graph, thereby defining a new spectral graph convolution, and a fast convolution calculation algorithm is designed to perform the convolution operation of the feature extraction vector. Efficient graph representation learning is achieved by building a deep recurrent graph neural network based on residual connections. The graph convolutional neural network based on the above-mentioned compact framework supports the collaborative processing of high-frequency and low-frequency graph information, that is, the resources on the graph are layered through high-pass and low-pass filters respectively. The aggregation of super-vertex information, compared with the existing graph neural network that only supports low-pass filtering, the graph neural network of this embodiment can avoid information loss caused by low-pass filtering.

According to some specific embodiments of the present invention, step S120 includes but is not limited to the following steps:

Step S210, perform feature fusion in the modality according to the representation matrix of each modality in the resource super-vertex to obtain a first fusion vector;

Step S220, according to the representation matrix of the various modalities in the resource super-vertex, interact with the features between the various modalities in the resource super-vertex to obtain a second fusion vector;

Step S230, obtaining a feature fusion matrix according to the plurality of first fusion vectors and the plurality of second fusion vectors, wherein the feature fusion matrix includes a plurality of modal vectors.

According to some specific embodiments of the present invention, step S210 includes but is not limited to the following steps:

Step S310, determining the eigenvalues of the modality according to the representation matrix of the modality;

Step S320, performing a pairwise inner product on all eigenvectors in the modal representation matrix to obtain a Gram matrix;

Step S330, obtaining a first fusion vector according to the eigenvalue and the Gram matrix.

According to some specific embodiments of the present invention, step S230 includes but is not limited to the following steps:

Step S410, inputting the first fusion vector and the second fusion vector into a tensor-type random configuration neural network to perform vector dimension transformation and splicing to obtain a feature fusion matrix.

According to some specific embodiments of the present invention, the interactive fusion of the feature fusion matrix of the neighbor resource super-vertex and the feature fusion matrix of the current resource super-vertex in step S130 includes but is not limited to the following steps:

Step S510, determining the missing mode of the current resource super vertex;

Step S520, determining the missing mode in the feature fusion matrix of the current resource super-vertex according to the first influence weight of the neighbor resource super-vertex on the missing mode and the modal vector corresponding to the missing mode in the feature fusion matrix of the neighbor resource super-vertex vector;

Step S530: Update the missing modal vector in the feature fusion matrix of the current resource super-vertex according to the second influence weight of the modal vector different from the missing modal in the feature fusion matrix of the neighboring resource super-vertex on the missing modal.

According to some specific embodiments of the present invention, performing modal aggregation on the feature fusion matrix of the current resource super-vertex to obtain the feature extraction vector of the current resource super-vertex in step S130 includes but is not limited to the following steps:

Step S610, determining the weight coefficients of multiple modal vectors in the feature fusion matrix;

Step S620, multiply each modal vector by the weight coefficient to obtain a plurality of weighted modal vectors;

Step S630, adding a plurality of weighted modal vectors to obtain a feature extraction vector of the current resource super vertex.

On the other hand, an embodiment of the present invention also provides a modal absence-oriented graph representation learning system. Referring to FIG. 2 , the modal absence oriented graph representation learning system includes:

a first module, configured to obtain a modality-missing hypergraph, wherein the modality-missing hypergraph includes multiple resource hypervertices, and the resource hypervertices include representation matrices of multiple modes;

The second module is used to perform feature interaction on the representation matrices of multiple said modalities in each of the resource super-vertices to obtain a feature fusion matrix of each of the resource super-vertices;

The third module is used to interactively fuse the feature fusion matrix of the neighbor resource super vertex with the feature fusion matrix of the current resource super vertex for each resource super vertex, and modally aggregate the feature fusion matrix of the current resource super vertex to obtain The feature extraction vector of the current resource super vertex;

The fourth module is used to update the modal missing hypergraph according to the feature extraction vector of each resource hypervertex to obtain a multimodal hypergraph;

The fifth module is used for inputting the multimodal hypergraph into a graph representation model to obtain a graph representation result.

It can be understood that, the contents in the above-mentioned embodiments of the learning method for modal absence-oriented graph representation are all applicable to the embodiments of this system, and the functions specifically implemented by the embodiments of this system are the same as the implementation of the above-mentioned modal absence-oriented graph representation learning method. Examples are the same, and the beneficial effects achieved are also the same as the beneficial effects achieved by the above-mentioned embodiment of the modal deletion-oriented graph representation learning method.

Referring to FIG. 3 , FIG. 3 is a schematic diagram of a modal deletion-oriented graph representation learning apparatus provided by an embodiment of the present invention. The modal deletion-oriented graph representation learning device according to the embodiment of the present invention includes one or more control processors and a memory. In FIG. 3 , one control processor and one memory are used as an example.

The control processor and the memory may be connected by a bus or in other ways, and the connection by a bus is taken as an example in FIG. 3 .

As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. Additionally, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory may optionally include memory located remotely from the control processor, and these remote memories may be connected to the modality-absence-oriented graphical representation learning device via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

Those skilled in the art can understand that the device structure shown in FIG. 3 does not constitute a limitation on the modal absence-oriented graphical representation learning device, and may include more or less components than those shown in the figure, or combine some components, Or a different component arrangement.

The non-transitory software programs and instructions required to realize the modal absence oriented graph representation learning method applied to the modal absence oriented graph representation learning device in the above embodiment are stored in the memory, and when executed by the control processor, execute In the above-mentioned embodiments, the modal absence-oriented graph representation learning method applied to the modal absence-oriented graph representation learning apparatus.

In addition, an embodiment of the present invention also provides a computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more control processors, so that the above-mentioned One or more control processors execute the modality-absent-oriented graph representation learning method in the above method embodiments.

Those of ordinary skill in the art can understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit . Such software may be distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those of ordinary skill in the art, the term computer storage media includes both volatile and nonvolatile implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules or other data flexible, removable and non-removable media. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices, or may Any other medium used to store desired information and which can be accessed by a computer. In addition, communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and can include any information delivery media, as is well known to those of ordinary skill in the art .

The embodiments of the present invention have been described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned embodiments, and within the scope of knowledge possessed by those of ordinary skill in the art, various Variety.

Claims (10)

1. A method for learning representations of modal absences, comprising the following steps: obtaining a modal missing hypergraph, wherein the modal missing hypergraph includes a plurality of resource hypervertices, and the resource hypervertices include representation matrices of multiple modalities; performing feature interaction on the representation matrices of a plurality of the modalities in each of the resource super-vertices to obtain a feature fusion matrix of each of the resource super-vertices; For each resource super vertex, interactively fuse the feature fusion matrix of the neighbor resource super vertex with the feature fusion matrix of the current resource super vertex, and perform modal aggregation on the feature fusion matrix of the current resource super vertex to obtain the features of the current resource super vertex extract vector; Update the missing modal hypergraph according to the feature extraction vector of each resource hypervertex to obtain a multimodal hypergraph; The multimodal hypergraph is input into a graph representation model to obtain a graph representation result. 2 . The modal deletion-oriented graph representation learning method according to claim 1 , wherein the graph representation model comprises a graph neural network with two-layer self-attention residual connections, and the graph neural network comprises a graph neural network based on tight A filter bank constructed by the framework, the filter bank includes a low-pass filter, a first high-pass filter, and a second high-pass filter. 3. The modal deletion-oriented graph representation learning method according to claim 1, wherein the feature interaction fusion is performed on the representation matrices of a plurality of the modalities in each of the resource super-vertices to obtain The feature fusion matrix of each resource super-vertex includes the following steps: Perform feature fusion in the modal according to the representation matrix of each modal in the resource super-vertex to obtain a first fusion vector; The second fusion vector is obtained by interacting the features between the multiple modes in the resource super vertex according to the representation matrix of the multiple modes in the resource super vertex; The feature fusion matrix is obtained according to a plurality of the first fusion vectors and a plurality of the second fusion vectors, wherein the feature fusion matrix includes a plurality of modal vectors. 4. The modal absence-oriented graph representation learning method according to claim 3, wherein the feature fusion in the modality is performed according to the representation matrix of each modality in the resource super-vertex, Obtaining the first fusion vector includes the following steps: determining the eigenvalues of the modality according to the representation matrix of the modality; Perform a pairwise inner product on all eigenvectors in the representation matrix of the modal to obtain a Gram matrix; The first fusion vector is obtained according to the eigenvalue and the Gram matrix. 5 . The modal deletion-oriented graph representation learning method according to claim 3 , wherein the obtaining the feature fusion matrix according to a plurality of the first fusion vectors and a plurality of the second fusion vectors comprises the following steps: 6 . The following steps: Inputting the first fusion vector and the second fusion vector into a tensor-type random configuration neural network to perform vector dimension conversion and splicing to obtain the feature fusion matrix. 6. The modal deletion-oriented graph representation learning method according to claim 5, characterized in that, the described feature fusion matrix of neighbor resource super-vertex and the feature fusion matrix of current resource super-vertex are interactively fused and comprise the following steps: Determine the missing modality of the current resource supervertex; The feature of the current resource super vertex is determined according to the first influence weight of the neighbor resource super vertex on the missing modality and the modal vector corresponding to the missing modality in the feature fusion matrix of the neighbor resource super vertex the missing modal vectors in the fusion matrix; Update the missing modality in the feature fusion matrix of the current resource super-vertex according to the second influence weight of the modal vector different from the missing modality in the feature fusion matrix of the neighbor resource super-vertex on the missing modality vector. 7. The modal absence-oriented graph representation learning method according to claim 6, wherein the feature extraction vector that the feature fusion matrix of the current resource super-vertex is obtained by modal aggregation to obtain the current resource super-vertex comprises the following steps : determining the weight coefficients of multiple modal vectors in the feature fusion matrix; multiplying each of the modal vectors by the weighting coefficient to obtain a plurality of weighted modal vectors; A feature extraction vector of the current resource super-vertex is obtained by adding up a plurality of the weighted modal vectors. 8. A modal deletion-oriented graph representation learning system, characterized in that it comprises: a first module, configured to obtain a modality-missing hypergraph, wherein the modality-missing hypergraph includes multiple resource hypervertices, and the resource hypervertices include representation matrices of multiple modes; The second module is used to perform feature interaction on the representation matrices of multiple said modalities in each of the resource super-vertices to obtain a feature fusion matrix of each of the resource super-vertices; The third module is used to interactively fuse the feature fusion matrix of the neighbor resource super vertex with the feature fusion matrix of the current resource super vertex for each resource super vertex, and modally aggregate the feature fusion matrix of the current resource super vertex to obtain The feature extraction vector of the current resource super vertex; The fourth module is used to update the modal missing hypergraph according to the feature extraction vector of each resource hypervertex to obtain a multimodal hypergraph; The fifth module is used for inputting the multimodal hypergraph into a graph representation model to obtain a graph representation result. 9. A modal absence-oriented graph representation learning device, characterized in that it comprises: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the modal absence-oriented graph representation learning method according to any one of claims 1 to 7. 10. A computer-readable storage medium in which a processor-executable program is stored, wherein the processor-executable program is used to implement any of claims 1 to 7 when executed by the processor. A described modality-missing-oriented graph representation learning method.

Images