CN111061851B - Question generation method and system based on given facts - Google Patents

Abstract

The invention relates to a question generation method and a question generation system based on given facts, wherein the question generation method comprises the following steps: acquiring historical reference data, wherein the historical reference data comprises historical input information of a plurality of different users; expanding each history input information to obtain a corresponding context representation; establishing a question generation model according to each piece of input information and the corresponding context representation; and determining a question sequence corresponding to the current input information according to the current input information of the current user based on the question generation model. According to the invention, a question generation model is established through historical reference data; the question generation model can be based on the question generation model, and the question sequence corresponding to the current input information can be accurately determined according to a small amount of current input information given by the current user.

Description

Method and system for generating question sentences based on given facts

technical field

The invention relates to the technical field of natural language processing, in particular to a method and system for generating question sentences based on given facts.

Background technique

With the vigorous development of the Internet and the increasing popularity of network communication terminals, people are exposed to massive amounts of information in various fields every day. Knowledge base question answering can help people quickly acquire knowledge from massive information, thereby reducing the cost of human learning. However, knowledge base question answering relies heavily on human-labeled data, and the labeled data of question-answer pairs has become a bottleneck resource that restricts question technology and question-answer system development. Question generation can effectively solve this problem.

The task of question generation is mainly to automatically generate questions from a given answer and its auxiliary information. The given answer and auxiliary information may be in the form of plain text or a structured knowledge base. Question generation has the following uses: 1. Automatically construct question-and-answer data resources, or reduce the workload of manually labeling question-and-answer pairs; 2. Used for data enhancement to improve the performance of question-answering systems; 3. As a typical text generation task, you can Contribute to the development and advancement of text generation technologies.

However, traditional question generation methods are prone to generate questions with mismatched predicates, such as given input <Statue of Liberty, location, New York City> in Table 1, may generate Q1 (Who created the Statue of Liberty?) so cannot Questions that express a given predicate. In addition, questions generated by traditional methods tend to correspond to multiple ambiguous answers. For example, Q2 (Where is the Statue of Liberty?) has multiple correct answers (such as the United States, New York State, New York City, etc. etc.), which makes the questions generated by traditional methods difficult to be practical.

Table 1

Contents of the invention

In order to solve the above-mentioned problems in the prior art, that is, to solve the problem of accurately determining based on a small number of given facts, the present invention provides a method and system for generating question sentences based on given facts.

In order to solve the problems of the technologies described above, the present invention provides the following solutions:

A method for generating questions based on given facts, the method for generating questions includes:

Acquiring historical reference data, the historical reference data including multiple pieces of historical input information of different users;

Extend each historical input information to obtain the corresponding context representation;

Establishing a question generation model according to each of the input information and the corresponding context representation;

Based on the question generation model, according to the current input information of the current user, a question sequence corresponding to the current input information is determined.

Optionally, the historical reference data also includes multiple pieces of supervisory information, each of which includes manually marked questions and reference answers corresponding to historical input information;

The question generation method also includes:

According to the supervisory information, the question generation model is corrected to obtain the corrected question generation model.

Optionally, according to the supervision information, the question generation model is corrected to obtain a revised question generation model, which specifically includes:

Based on the question generation model, according to each historical input information, determine the corresponding historical question sequence;

According to each of the historical question sequence and the corresponding manually marked question, calculate and generate the question loss

Calculate the auxiliary answer loss based on the historical question sequence and the corresponding reference answer

Wherein, each reference answer includes answer type words corresponding to historical input information, and the historical question sequence includes generated words corresponding to historical input information, is the set of answer type words, |A| represents the number of answer type words in the set of answer type words, /> is the loss of the generated word y _t and the corresponding answer type word a _n in a question sequence;

Generate question sentences according to the loss and auxiliary answer loss /> Determining Supervisory Information Loss

Among them, λ represents the reference coefficient;

According to the supervisory information loss The question generation model is corrected to obtain the corrected question generation model.

Optionally, the format of the historical input information of the different users is head entity-relationship-tail entity;

The step of expanding each historical input information to obtain a corresponding context representation specifically includes:

For the head entity and/or the tail entity, use the type information in the knowledge base as the context representation of the head entity and/or tail entity;

For the relationship, at least one of the sentences corresponding to the field, value range, topic and distance supervision mark in the knowledge base is used as the context representation of the relationship.

Optionally, when there are multiple types of information in the knowledge base, the most frequently used and most differentiated type is selected as the context representation of the head entity and/or tail entity.

Optionally, the establishment of a question generation model according to each of the input information and the corresponding context representation specifically includes:

For each pair of input information and the corresponding context representation,

performing training on the input information to obtain training information;

Obtaining a representation sequence according to the context representation based on the first sequence model;

fusing the training information and the representation sequence to obtain fusion information;

Encoding the fusion information to obtain a hidden layer state sequence;

Each hidden layer state sequence is decoded to obtain a corresponding decoding sequence function, and the decoding sequence function is a question generation model.

Optionally, the decoding of each of the hidden layer state sequences, and calculation to obtain a corresponding decoding sequence function specifically includes:

Decoding each of the hidden layer state sequences based on the second sequence model to obtain decoding information;

According to the decoding information, respectively calculate the knowledge base copy mode probability of copying the corresponding name of the historical input information from the knowledge base, the context copy mode probability of copying the context representation and the vocabulary generation mode probability of generating words from the vocabulary;

According to the knowledge base copy mode probability p _cpkb , the context copy mode probability p _cptx and the vocabulary generation mode probability p _genv , calculate the predicted probability P(y _t |s _t , y _t-1 , F, C) of the target word:

Among them, genv, cpkb and cpctx represent vocabulary generation mode, knowledge base replication mode and context replication mode respectively, p. represents the probability of three different modes, P(*|*) represents the probability of generating target words in various modes, F and C represent the input information and context respectively, s _t represents the current decoding state, and y _t represents the words generated at the current moment;

According to the predicted probability P(y _t |s _t , y _t-1 , F, C) of the target word, it is decoded word by word to obtain the decoded question sequence function.

In order to solve the problems of the technologies described above, the present invention also provides the following solutions:

A question generation system based on given facts, said question generation system comprising:

an acquisition unit, configured to acquire historical reference data, the historical reference data including multiple pieces of historical input information of different users;

An expansion unit, configured to expand each historical input information to obtain a corresponding context representation;

A modeling unit, configured to establish a question generation model according to each of the input information and the corresponding context representation;

The determining unit is configured to determine a question sequence corresponding to the current input information based on the question generation model and according to the current input information of the current user.

In order to solve the problems of the technologies described above, the present invention also provides the following solutions:

A question generation system based on given facts, including:

processor; and

a memory arranged to store computer-executable instructions that, when executed, cause the processor to:

Acquiring historical reference data, the historical reference data including multiple pieces of historical input information of different users;

Extend each historical input information to obtain the corresponding context representation;

Establishing a question generation model according to each of the input information and the corresponding context representation;

Based on the question generation model, according to the current input information of the current user, a question sequence corresponding to the current input information is determined.

In order to solve the problems of the technologies described above, the present invention also provides the following solutions:

A computer-readable storage medium that stores one or more programs that, when executed by an electronic device including a plurality of application programs, cause the electronic device to perform the following operations :

Acquiring historical reference data, the historical reference data including multiple pieces of historical input information of different users;

Extend each historical input information to obtain the corresponding context representation;

Establishing a question generation model according to each of the input information and the corresponding context representation;

Based on the question generation model, according to the current input information of the current user, a question sequence corresponding to the current input information is determined.

According to the embodiments of the present invention, the present invention discloses the following technical effects:

The present invention establishes a question generation model through historical reference data; based on the question generation model, the question sequence corresponding to the current input information can be accurately determined according to a small amount of current input information given by the current user.

Description of drawings

Fig. 1 is the flow chart of the present invention based on the question generation method of given fact;

Figure 2 is a schematic diagram of answer-assisted supervision

Fig. 3 is a schematic diagram of the module structure of the system for generating questions based on given facts in the present invention.

Symbol Description:

Acquisition unit-1, expansion unit-2, modeling unit-3, determination unit-4.

Detailed ways

Preferred embodiments of the present invention are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present invention, and are not intended to limit the protection scope of the present invention.

The object of the present invention is to provide a question generation method based on given facts, and establish a question generation model through historical reference data; based on the question generation model, it can be based on a small amount of current input information given by the current user, namely The question sequence corresponding to the current input information can be accurately determined.

In order to make the above objects, features and advantages of the present invention more comprehensible, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

As shown in Figure 1, the question generation method of the present invention based on given fact comprises:

Step 100: Acquiring historical reference data, the historical reference data includes multiple pieces of historical input information of different users;

Step 200: Extend each historical input information to obtain a corresponding context representation;

Step 300: Establish a question generation model according to the input information and the corresponding context representation;

Step 400: Based on the question generation model, according to the current user's current input information, determine the question sequence corresponding to the current input information.

Further, the historical reference data also includes multiple pieces of supervisory information, each of which includes manually marked questions and reference answers corresponding to historical input information.

In order to improve the determination accuracy, the question generation method based on given facts of the present invention also includes:

According to the supervisory information, the question generation model is corrected to obtain the corrected question generation model.

The traditional method only has the supervision information at the question end, that is, compares the generated questions with the artificially labeled questions, uses the difference between them as a loss, and then uses the optimizer for training, so it may be easy to get ambiguous questions, that is, a question Corresponds to multiple correct answers. To this end, in addition to manually labeling questions as supervision information, answer information is also used as auxiliary supervision. The goal is to generate questions that contain any type of answer.

Further, according to the supervision information, the question generation model is corrected to obtain the corrected question generation model, which specifically includes:

Step S1: Based on the question generation model, according to each historical input information, determine the corresponding historical question sequence.

Step S2: According to each of the historical question sequences and the corresponding manually marked questions, calculate and generate the question loss

Step S3: Calculate the auxiliary answer loss according to the historical question sequence and the corresponding reference answer (as shown in picture 2):

Wherein, each reference answer includes answer type words corresponding to historical input information, and the historical question sequence includes generated words corresponding to historical input information, is the collection of answer type words (as the type words of New York City include city, administrative region, etc.), |A| represents the quantity of answer type words in the collection of said answer type words, /> is the loss of the generated word y _t and the corresponding answer type word a _n in a question sequence. where the auxiliary answer loss /> is the minimum of the two calculations.

Step S4: Generate a question loss according to the and auxiliary answer loss /> Determining Supervisory Information Loss />

Among them, λ represents the reference coefficient.

Step S5: According to the supervisory information loss The question generation model is corrected to obtain the corrected question generation model.

In this embodiment, the format of the historical input information of different users is head entity (subject)-relationship (predicate)-tail entity (object).

Further, in step 200, the expansion of each historical input information to obtain the corresponding context representation specifically includes:

For the head entity and/or the tail entity, use the type information in the knowledge base as the context representation of the head entity and/or tail entity;

For the relationship, at least one of the sentences corresponding to the field, value range, topic and distance supervision mark in the knowledge base is used as the context representation of the relationship.

Wherein, when there are multiple types of information in the knowledge base, the most frequently used and most differentiated type is selected as the context representation of the head entity and/or tail entity.

In step 300, the question generation model is established according to the input information and the corresponding context representation, which specifically includes:

For each pair of input information and the corresponding context representation,

performing training on the input information to obtain training information;

Obtaining a representation sequence according to the context representation based on the first sequence model;

fusing the training information and the representation sequence to obtain fusion information;

Encoding the fusion information to obtain a hidden layer state sequence;

Each hidden layer state sequence is decoded to obtain a corresponding decoding sequence function, and the decoding sequence function is a question generation model.

Among them, for the fact that the input is symbolic, knowledge base representation learning methods such as TransE can be used to pre-train on large-scale corpus, or random initialization can be used to train with the first sequence model. For the context information, the first sequence model can be used for modeling, and the first sequence model can be a recurrent neural network (RNN, Recurrent NeuralNetworks), a gated recurrent unit (GRU, Gated Recurrent Unit), a long short-term memory network (LSTM, LongShort Term Memory) and Transformer models, etc. In this way, each element in the reference information has both a symbolic representation and a contextual representation, which can be fused through the Gate, and finally, the fused information can be encoded into a hidden layer state sequence (such as H _f =[h ^s ; h ^p ; h ^o ]).

Further, the decoding of each of the hidden layer state sequences is calculated to obtain a corresponding decoding sequence function, which specifically includes:

Decoding each of the hidden layer state sequences based on the second sequence model to obtain decoding information;

According to the decoding information, respectively calculate from the knowledge base duplication mode probability of the name corresponding to the history input information, the context duplication mode probability of duplication context representation and the vocabulary generation mode probability of generating words from the vocabulary from the knowledge base;

According to the knowledge base replication mode probability p _cpkb , the context replication mode probability p _cpctx and the vocabulary generation mode probability p _genv , calculate the predicted probability P(y _t |s _t , y _t-1 , F, C) of the target word:

Among them, genv, cpkb and cpctx represent vocabulary generation mode, knowledge base replication mode and context replication mode respectively, p. represents the probability of three different modes, P(*|*) represents the probability of generating words in various modes, F and C represents input information and context respectively, s _t represents the current decoding state, and y _t represents the words generated at the current moment;

According to the predicted probability P(y _t |s _t , y _t-1 , F, C) of the target word, decode word by word to obtain the question sequence function.

Similarly, the decoder can use the second sequence model to decode, and the second sequence model can be a recurrent neural network (RNN, Recurrent Neural Networks), a gated recurrent unit (GRU, Gated Recurrent Unit), a long short-term memory network (LSTM , Long Short Term Memory) and Transformer models, etc. In the decoding process, in order to better capture the input information, the following copy (copy) forms can be selectively used: 1. Considering that the head entity often appears in the generated question, so copy the name corresponding to the head entity symbol in the knowledge base; 2. The context of .copy extension, it should be noted that the input context may have many repeated words, so the maxout pointer mechanism is used, that is, when multiple identical tokens (words) appear, the token with the highest copy score is used as the token in the Scores in copy mode, not sums. Finally, there are two different copy forms and three modes of generating tokens from the vocabulary. The weighted sum of these three modes is used as the final selection token probability, and then according to the predicted probability of the target word, word-by-word decoding is obtained to obtain the question generation model , further modifying the question generation model to obtain the revised question generation model, and according to the revised question generation model, the question sequence information can be accurately determined based on a small number of given facts.

Verify effectiveness of the present invention below by following experiment:

test corpus

SimpleQuestions: Currently the largest knowledge base question answering dataset.

Comparison method:

Template: template generation questions

Serban et al. (2016): Sequence-to-Sequence Model Generation Questions

Elsahar et al. (2018): Introducing a Single Context to Generate Questions

Shown experimental result (as table 2)

Table 2

Overall performance comparison: its effectiveness is illustrated by comparing the effects of the existing methods and the present invention. The performance of the present invention is significantly better than the benchmark method, and with the help of answer supervision (the last row), the performance continues to improve.

table 3

Predicate coverage comparison (as shown in table 3): artificially evaluated whether the predicate of given input is correctly expressed in the generated question sentence, by calculating the ratio (predicate coverage degree Predicate Identification) of correct expression of given predicate, find the performance of the present invention most.

Table 4

Comparison of answer coverage: the present invention further defines an evaluation index Ans _cov of the proportion of answer-type words contained in generated questions, which is used to evaluate the degree of certainty of the answers corresponding to generated questions. Adjust the weight λ of the answer auxiliary supervision information. It is found that when the answer is assisted by supervision, the BLEU score is higher, and the Ans _cov improvement is more obvious.

In addition, the present invention also provides a question generation system based on given facts, which can accurately determine questions based on a small number of given facts.

As shown in FIG. 3 , the system for generating question sentences based on given facts in the present invention includes: an acquisition unit 1 , an expansion unit 2 , a modeling unit 3 and a determination unit 4 .

Specifically, the acquiring unit 1 is configured to acquire historical reference data, the historical reference data including multiple pieces of historical input information of different users;

The expansion unit 2 is used to expand each historical input information to obtain a corresponding context representation;

The modeling unit 3 is used to establish a question generation model according to each of the input information and the corresponding context representation;

The determining unit 4 is configured to determine a sequence of questions corresponding to the current input information based on the question generation model and according to the current input information of the current user.

Further, the present invention also provides a question generation system based on given facts, including:

processor; and

a memory arranged to store computer-executable instructions that, when executed, cause the processor to:

Acquiring historical reference data, the historical reference data including multiple pieces of historical input information of different users;

Extend each historical input information to obtain the corresponding context representation;

Establishing a question generation model according to each of the input information and the corresponding context representation;

Based on the question generation model, according to the current input information of the current user, a question sequence corresponding to the current input information is determined.

The present invention also provides a computer-readable storage medium that stores one or more programs that, when executed by an electronic device including a plurality of application programs, cause the electronic The device does the following:

Acquiring historical reference data, the historical reference data including multiple pieces of historical input information of different users;

Extend each historical input information to obtain the corresponding context representation;

Establishing a question generation model according to each of the input information and the corresponding context representation;

Based on the question generation model, according to the current input information of the current user, a question sequence corresponding to the current input information is determined.

Compared with the prior art, the system for generating question sentences based on given facts and the computer-readable storage medium of the present invention have the same beneficial effects as the method for generating question sentences based on given facts, and will not be repeated here.

So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings, but those skilled in the art will easily understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to relevant technical features, and the technical solutions after these changes or substitutions will all fall within the protection scope of the present invention.

Claims (8)

1. A question generating method based on given facts, characterized in that, the question generating method comprises: Obtaining historical reference data, the historical reference data includes a plurality of historical input information of different users; the historical reference data also includes a plurality of supervision information, and each of the supervision information includes manually marked questions and reference answers corresponding to the historical input information ; Extend each historical input information to obtain the corresponding context representation; Establishing a question generation model according to each of the input information and the corresponding context representation; Its modification method of described question generation model also includes: According to the supervisory information, the question generation model is corrected to obtain the revised question generation model: Based on the question generation model, according to each historical input information, determine the corresponding historical question sequence; According to each of the historical question sequence and the corresponding manually marked question, calculate and generate the question loss Calculate the auxiliary answer loss based on the historical question sequence and the corresponding reference answer Wherein, each reference answer includes answer type words corresponding to historical input information, and the historical question sequence includes generated words corresponding to historical input information, is the set of answer type words, |A| represents the number of answer type words in the set of answer type words, /> is the loss of the generated word y _t and the corresponding answer type word a _n in a question sequence; Generate question sentences according to the loss and auxiliary answer loss /> Determining Supervisory Information Loss /> Among them, λ represents the reference coefficient; According to the supervisory information loss modifying the question generation model to obtain the revised question generation model; Based on the question generation model, according to the current input information of the current user, a question sequence corresponding to the current input information is determined. 2. the question generation method based on given fact according to claim 1, is characterized in that, the format of the history input information of described different users is head entity-relationship-tail entity; The step of expanding each historical input information to obtain a corresponding context representation specifically includes: For the head entity and/or the tail entity, use the type information in the knowledge base as the context representation of the head entity and/or tail entity; For the relationship, at least one of the sentences corresponding to the field, value range, topic and distance supervision mark in the knowledge base is used as the context representation of the relationship. 3. The method for generating question sentences based on given facts according to claim 2, wherein when there are multiple types of information in the knowledge base, the most frequently used and most differentiated type is selected as the head entity and/or the contextual representation of the tail entity. 4. the question generation method based on given fact according to claim 1, is characterized in that, described according to each described input information and corresponding context representation, sets up question generation model, specifically comprises: For each pair of input information and the corresponding context representation, performing training on the input information to obtain training information; Obtaining a representation sequence according to the context representation based on the first sequence model; fusing the training information and the representation sequence to obtain fusion information; Encoding the fusion information to obtain a hidden layer state sequence; Each of the hidden layer state sequences is decoded to obtain a corresponding decoding sequence function, and the decoding sequence function is a question generation model. 5. the question generation method based on given fact according to claim 4, is characterized in that, described each described hidden layer state sequence is decoded, calculates and obtains corresponding decoding sequence function, specifically comprises: Decoding each of the hidden layer state sequences based on the second sequence model to obtain decoding information; According to the decoding information, respectively calculate the knowledge base copy mode probability of copying the corresponding name of the historical input information from the knowledge base, the context copy mode probability of copying the context representation and the vocabulary generation mode probability of generating words from the vocabulary; According to the knowledge base replication mode probability p _cpkb , the context replication mode probability p _cpctx and the vocabulary generation mode probability p _genυ , calculate the predicted probability P(y _t |s _t , y _t-1 , F, C) of the target word: P(y _t |s _t , y _t-1 , F, C)＝p _genυ P _genυ (y _t |s _t , V)+p _cpkb P _cpkb (y _t |s _t , F)+p _cpctx P _cpctx (y _t |s _t , C); Among them, genv, cpkb and cpctx represent vocabulary generation mode, knowledge base replication mode and context replication mode respectively, p. represents the probability of three different modes, P(*|*) represents the probability of generating words in various modes, F and C represents input information and context respectively, s _t represents the current decoding state, and y _t represents the words generated at the current moment; According to the predicted probability P(y _t |s _t , y _t-1 , F, C) of the target word, it is decoded word by word to obtain the question sequence function. 6. A question generation system based on given facts, characterized in that, the question generation system includes: An acquisition unit, configured to acquire historical reference data, the historical reference data including multiple pieces of historical input information of different users; the historical reference data also includes multiple pieces of supervisory information, each of which includes manual annotations corresponding to historical input information Questions and reference answers; An expansion unit, configured to expand each historical input information to obtain a corresponding context representation; A modeling unit, configured to establish a question generation model according to each of the input information and the corresponding context representation; Its modification method of described question generation model also includes: According to the supervisory information, the question generation model is corrected to obtain the revised question generation model: Based on the question generation model, according to each historical input information, determine the corresponding historical question sequence; According to each of the historical question sequence and the corresponding manually marked question, calculate and generate the question loss Calculate the auxiliary answer loss based on the historical question sequence and the corresponding reference answer Wherein, each reference answer includes answer type words corresponding to historical input information, and the historical question sequence includes generated words corresponding to historical input information, is the set of answer type words, |A| represents the number of answer type words in the set of answer type words, /> is the loss of the generated word y _t and the corresponding answer type word a _n in a question sequence; Generate question sentences according to the loss and auxiliary answer loss /> Determining Supervisory Information Loss /> Among them, λ represents the reference coefficient; According to the supervisory information loss modifying the question generation model to obtain the revised question generation model; The determining unit is configured to determine a question sequence corresponding to the current input information based on the question generation model and according to the current input information of the current user. 7. A question generation system based on given facts, comprising: processor; and a memory arranged to store computer-executable instructions that, when executed, cause the processor to: Obtaining historical reference data, the historical reference data includes a plurality of historical input information of different users; the historical reference data also includes a plurality of supervision information, and each of the supervision information includes manually marked questions and reference answers corresponding to the historical input information ; Extend each historical input information to obtain the corresponding context representation; Establishing a question generation model according to each of the input information and the corresponding context representation; Its modification method of described question generation model also includes: According to the supervisory information, the question generation model is corrected to obtain the revised question generation model: Based on the question generation model, according to each historical input information, determine the corresponding historical question sequence; According to each of the historical question sequence and the corresponding manually marked question, calculate and generate the question loss Calculate the auxiliary answer loss based on the historical question sequence and the corresponding reference answer Wherein, each reference answer includes answer type words corresponding to historical input information, and the historical question sequence includes generated words corresponding to historical input information, is the set of answer type words, |A| represents the number of answer type words in the set of answer type words, /> is the loss of the generated word y _t and the corresponding answer type word a _n in a question sequence; Generate question sentences according to the loss and auxiliary answer loss /> Determining Supervisory Information Loss /> Among them, λ represents the reference coefficient; According to the supervisory information loss modifying the question generation model to obtain the revised question generation model; Based on the question generation model, according to the current input information of the current user, a question sequence corresponding to the current input information is determined. 8. A computer-readable storage medium storing one or more programs that, when executed by an electronic device including a plurality of application programs, cause the electronic device to perform Do the following: Obtaining historical reference data, the historical reference data includes a plurality of historical input information of different users; the historical reference data also includes a plurality of supervision information, and each of the supervision information includes manually marked questions and reference answers corresponding to the historical input information ; Extend each historical input information to obtain the corresponding context representation; Establishing a question generation model according to each of the input information and the corresponding context representation; Its modification method of described question generation model also includes: According to the supervisory information, the question generation model is corrected to obtain the revised question generation model: Based on the question generation model, according to each historical input information, determine the corresponding historical question sequence; According to each of the historical question sequence and the corresponding manually marked question, calculate and generate the question loss Calculate the auxiliary answer loss based on the historical question sequence and the corresponding reference answer Wherein, each reference answer includes answer type words corresponding to historical input information, and the historical question sequence includes generated words corresponding to historical input information, is the set of answer type words, |A| represents the number of answer type words in the set of answer type words, /> is the loss of the generated word y _t and the corresponding answer type word a _n in a question sequence; Generate question sentences according to the loss and auxiliary answer loss /> Determining Supervisory Information Loss /> Among them, λ represents the reference coefficient; According to the supervisory information loss modifying the question generation model to obtain the revised question generation model; Based on the question generation model, according to the current input information of the current user, a question sequence corresponding to the current input information is determined.