CN118283182A - AI-based intelligent voice call prediction method, program product, device and medium - Google Patents

Abstract

The application provides an AI-based intelligent voice call prediction method, a program product, equipment and a medium, wherein the method comprises the following steps: receiving user call data in the current call; detecting the call data of the user to obtain a call detection result; the detection processing comprises at least one of emotion detection, audio detection, speech speed detection, keyword detection and dialogue flow detection; based on the call detection result, the ending time of the current call is predicted, and based on the ending time of the current call, whether the next call is called is determined. And obtaining at least one call detection result corresponding to the detection processing, predicting the ending time of the current call, and confirming whether to call the next call according to the ending time. Customer service can answer the call of the next call faster, so that waiting time of the customer service is reduced, and working efficiency is improved.

Description

AI-based intelligent voice call prediction method, program product, device and medium

Technical Field

The present application relates to the field of artificial intelligence technology, and more specifically, to an AI-based intelligent voice call prediction method, program product, device, and medium.

Background technique

At present, customer service work mainly relies on manual agent telephone operations. During the call process, the manual agent needs to make calls one by one. After the call is connected, the manual agent and the customer will enter the conversation process, and then continue to make the next call after the call is completed. In this way, the manual agent needs to spend a lot of time waiting for the user to answer the call, resulting in low work efficiency.

Summary of the invention

The purpose of the embodiments of the present application is to provide an AI-based intelligent voice call prediction method, program product, device and medium, which are used to solve the problem of spending a lot of time waiting for users to answer the call, resulting in low work efficiency.

In the first aspect, an embodiment of the present application provides an AI-based intelligent voice call prediction method, including: receiving user call data in a current call; detecting and processing the user call data to obtain a call detection result; the detection processing includes at least one of emotion detection, audio detection, speech speed detection, keyword detection, and dialogue flow detection; based on the call detection result, predicting the end duration of the current call, and based on the end duration of the current call, determining whether to call the next call.

In the above implementation process, by performing real-time detection on the user's call data in the current call, the detection process includes at least one of emotion detection, audio detection, speech speed detection, keyword detection and dialogue flow detection, obtaining at least one call detection result corresponding to the detection process, predicting the end time of the current call, and confirming whether to call the next call based on the end time. The customer service can answer the next call faster, reduce the waiting time of the customer service, and improve work efficiency.

Optionally, based on any embodiment, the user call data includes user call audio data and/or user call text data; the call detection result includes an emotion detection result; the user call data is detected and processed to obtain the call detection result, including: inputting the user call data into a preset emotion detection model to obtain the emotion detection result; the emotion detection model is obtained by training the preset emotion data and the emotion labels corresponding to the emotion data.

In the above implementation process, the emotion detection model is used to perform emotion detection on the user call audio data and/or the user call text data in the user call data to obtain the emotion detection result. The emotional state can reflect the user's willingness to answer or hang up the call to a certain extent. The emotion detection result is used as one of the factors for predicting the end time of the current call, thereby improving the accuracy of the end time prediction.

Optionally, based on any embodiment, the call detection result includes an audio detection result, and the user call data is detected and processed to obtain the call detection result, including: performing audio detection on the user call audio data in the user call data to obtain the audio detection result; the audio detection includes tone detection and/or decibel detection.

In the above implementation process, audio detection is performed on the user call audio data in the user call data to obtain an audio detection result, which includes a tone detection result and/or a decibel detection result. The audio detection result is used as one of the factors for predicting the end time of the current call, thereby improving the accuracy of the end time prediction.

Optionally, based on any embodiment, the call detection result includes a call step detection result, and the user call data is detected and processed to obtain the call detection result, including: obtaining the customer service call data in the current call; inputting the customer service call data and the user call data into a preset call process model to obtain the call step detection result; the call process model obtains step features by extracting features from the question and answer data corresponding to each step in the call process; adding corresponding step labels to the step features; and training the step features and the labels corresponding to the step features.

In the above implementation process, the call process model is used to obtain the call step detection results based on the customer service call data and the user call data. The call step detection results represent the step of the current call, and largely reflect the time required for the current call to end. The end time of the current call is predicted based on the call step detection results, thereby improving the accuracy of the end time prediction.

Optionally, based on any embodiment, predicting the end duration of the current call based on the call detection result includes: using a preset fitting prediction function to obtain the end duration of the current call according to the call detection result.

In the above implementation process, the preset fitting prediction function is used to obtain the end time of the current call according to the call detection result, and the next call is called when the end time of the current call is less than the time threshold. The customer service can answer the next call faster, reduce the waiting time of the customer service, and improve work efficiency.

Optionally, based on any embodiment, the call detection result includes an emotion detection result, an audio detection result, a speech speed detection result, a keyword detection result, and a call step detection result, and the fitting prediction function includes:

Among them, y is the end time of the current call, k is the first parameter, emotion.pt is the emotion detection result, a is the emotion detection parameter, Tone.pt is the audio detection result, b is the audio detection parameter, Speed.pt is the speech speed detection result, c is the speech speed detection parameter, Key.pt is the keyword detection result, d is the keyword detection parameter, Process.pt is the dialogue process detection result, and e is the dialogue process detection parameter.

Optionally, based on any embodiment, after confirming the call for the next call, the method also includes: calling the next call, and using the intelligent outbound call robot to talk to the second user corresponding to the next call; performing real-time voice recognition on the call voice data of the second user to obtain the user's real-time call text, and the time of the user's real-time call text; obtaining the intelligent outbound call robot's real-time call text, and the time of the intelligent outbound call robot's real-time call text; the intelligent outbound call robot's real-time call text is generated according to the preset text and/or the call voice data of the second user; sorting the user's real-time call text and the intelligent outbound call robot's real-time call text in chronological order to obtain the conversation content between the intelligent outbound call robot and the second user; displaying the conversation content so that the customer service can confirm whether to take over the intelligent outbound call robot and continue to talk to the second user based on the conversation content between the intelligent outbound call robot and the second user.

In the above implementation process, the conversation content between the intelligent outbound call robot and the second user is displayed in real time, so that the customer service can decide whether to take over the next call and serve the second user according to the conversation content. This reduces the waiting time of the customer service, improves work efficiency, and improves the user experience and service quality.

Optionally, based on any embodiment, the method also includes: obtaining conversation data of the current call; the conversation data includes conversation data between the intelligent outbound call robot and the user and/or conversation data between the customer service and the user; inputting the conversation data into a preset text summary model to obtain a call summary; the text summary model is obtained by performing LORA fine-tuning on the pre-trained model; performing LORA fine-tuning on the pre-trained model, including: loading the pre-trained model and keeping the preset weight parameters of the pre-trained model unchanged; adding a trainable layer to the pre-trained model; training the trainable layer using a preset conversation sample data set through a low-rank adaptive method; and updating the pre-trained model using the trained trainable layer to obtain a text summary model.

In the above implementation process, the conversation data is input into the preset text summary model to obtain the call summary, which reduces the time of manually recording the call summary and improves the summary efficiency. In addition, the text summary model is obtained by fine-tuning the pre-trained model with LORA to improve the problem of the weak text content summary ability of the pre-trained model. After fine-tuning, its summary ability in professional fields is strengthened and the accuracy of the summary is improved.

Optionally, based on any embodiment, the method also includes: identifying user call audio data in the user call data to obtain user call text data; predicting punctuation marks on the user call text data, and dividing the user call text data according to the punctuation mark prediction results to obtain target user sentences; using a preset speech recommendation model to obtain corresponding reply speech according to the target user sentences; the speech recommendation model obtains knowledge base data and a pre-built large language model; integrates the large language model and the language chain tool; and uses the language chain tool to vectorize the knowledge base data.

In the above implementation process, the preset speech recommendation model is used to obtain the corresponding reply speech according to the target user's sentence, thereby improving the accuracy and efficiency of the speech recommendation.

On the second aspect, the embodiment of the present application also provides an AI-based intelligent voice call prediction device, including: a user data receiving module, used to receive user call data in a current call; a detection module, used to detect and process the user call data to obtain a call detection result; the detection process includes at least one of emotion detection, audio detection, speech speed detection, keyword detection and dialogue flow detection; a call prediction module, used to predict the end duration of the current call based on the call detection result, and determine whether to call the next call based on the end duration of the current call.

In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the method described above is performed.

In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described above is executed.

This application provides an AI-based intelligent voice call prediction method, program product, device and medium. By performing real-time detection on the user call data in the current call, the detection process includes at least one of emotion detection, audio detection, speech speed detection, keyword detection and dialogue flow detection, obtaining at least one call detection result corresponding to the detection process, predicting the end time of the current call, and confirming whether to call the next call based on the end time. Customer service can answer the next call faster, reduce the waiting time of customer service, and improve work efficiency.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

FIG1 is a flow chart of an AI-based intelligent voice call prediction method provided in an embodiment of the present application;

FIG2 is a schematic diagram of a detection process provided in an embodiment of the present application;

FIG3 is a schematic diagram of obtaining a call summary based on a text summary model provided in an embodiment of the present application;

FIG4 is a schematic diagram of the process of speech recommendation shown in the present application;

FIG5 is a schematic diagram of a smart call process according to the present application;

FIG6 is a schematic diagram of the structure of an AI-based intelligent voice call prediction device 200 provided in an embodiment of the present application;

FIG. 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

Detailed ways

The following embodiments of the technical solution of the present application will be described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application.

In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

Currently, there are telephone customer service positions in many fields. Customer service mainly provides professional consulting services to customers through telephone. The specific workflow is that the human agent makes a call and waits for the user to answer. After the call is ended, the next call is made. Of course, after the current call is ended, the current call can also be recorded to obtain a call summary. In other words, the human agent needs to wait for the user to answer each call made by the telephone operator. A large part of the working time of the human agent is spent waiting, resulting in low work efficiency.

The present application provides an AI-based intelligent voice call prediction method, which detects the user call data in the current call in real time, including emotion detection, audio detection, speech speed detection, keyword detection, and dialogue flow detection. The detection process obtains at least one call detection result corresponding to the detection process, predicts the end time of the current call, and confirms whether to call the next call based on the end time. Customer service can answer the next call faster, reduce the waiting time of customer service, and improve work efficiency.

Please refer to FIG. 1 for a flow chart of an AI-based intelligent voice call prediction method provided in an embodiment of the present application. The AI-based intelligent voice call prediction method provided in an embodiment of the present application can be applied to an electronic device equipped with an intelligent call system. The electronic device may include a terminal and a server; wherein the terminal may specifically be a smart phone, a tablet computer, a computer, a personal digital assistant (PDA), etc.; the server may specifically be an application server or a web server. The AI-based intelligent voice call prediction method may include:

Step S110: receiving call data of the user in the current call.

Step S120: Detect and process the user's call data to obtain a call detection result; the detection process includes at least one of emotion detection, audio detection, speech speed detection, keyword detection, and dialogue flow detection.

Step S130: Based on the call detection result, predict the end time of the current call, and determine whether to make the next call based on the end time of the current call.

In step S110, a call is made to the user's phone through the intelligent outbound calling system. The intelligent outbound calling system can provide a standardized automatic outbound calling service and record the conversation content during the call with the user's consent.

After the current call is connected, a human customer service or intelligent outbound call robot can have a conversation with the user. Before the business conversation, it is necessary to confirm whether the user agrees to record the call content. After the user agrees, the conversation will be conducted with the user and the user's call data will be recorded. The user's call data is the content expressed by the user during the call. The user's call data includes the user's audio data in the current call and the text data obtained by recognizing the audio data.

It is understandable that the user call data can be obtained directly, or the complete conversation content between the human customer service or intelligent outbound call robot and the user can be obtained, and the user call data can be extracted from the complete conversation content.

In step S120, emotion detection refers to the process of evaluating the emotional state expressed in the user call audio data and/or the user call text data in the user call data. Emotion detection can be implemented by a rule-based, machine learning-based, or deep learning-based method. The emotion detection result can be, for example, positive, negative, neutral, etc., and can also be an emotion such as question.

Audio detection refers to detecting the intonation and volume of user call audio data in the user call data. The audio detection result may be, for example, the classification of the intonation or the decibel of the volume.

Speech rate detection refers to detecting the speech rhythm and speech speed of the user call audio data in the user call data.

Keyword detection refers to the matching detection of preset keywords on the user call text data in the user call data. The keyword detection result can indicate whether the user call text data matches the preset keyword; and if the keyword is matched, the keyword detection result can also indicate the matched keyword. For example, matching keywords such as goodbye may indicate that the call is about to be hung up.

Dialogue process detection refers to real-time detection of user call data and customer service call data in the current call, and determines the steps of the current call in the entire process. The dialogue process detection result is the step detection result of the current call data. For example, in some outbound call services, there are relatively fixed process steps. After the call is connected, a normal call may include the following steps: the first step is to obtain user authorization; the second step is to introduce the purpose of this call to the user; the third step is to introduce specific business projects and user feedback intentions; the fourth step can be to ask whether the service is satisfactory, and end the conversation, etc. Each step can have a specific pattern or conversation, and the steps of the current call can be determined through dialogue process detection.

It is understandable that the detection and processing of user call data can determine one or more detection processes from emotion detection, audio detection, speech rate detection, keyword detection and dialogue flow detection; when multiple detections are required, the detection order can be determined according to needs, and of course concurrent detection is also possible.

In another embodiment, not only the user call data but also the complete conversation content may be detected to obtain the call detection result.

In step S130, based on the call detection result, the end time of the current call is predicted, and the call detection result includes at least one of the emotion detection result, the audio detection result, the speech speed detection result, the keyword detection result, and the call step detection result. A preset function or deep learning model can be used to predict the end time of the current call based on the call detection result, and the end time of the current call refers to the time required for the current call to end.

After obtaining the end time of the current call, it can be determined whether to call the next call based on the end time of the current call and the preset time threshold. For example, if the end time of the current call is less than the time threshold, the next call is called through the intelligent outbound calling system. The time threshold can be set according to actual needs, such as 15 seconds, 20 seconds or 30 seconds, etc., and this application does not limit this.

In the above implementation process, by performing real-time detection on the user's call data in the current call, the detection process includes at least one of emotion detection, audio detection, speech speed detection, keyword detection and dialogue flow detection, obtaining at least one call detection result corresponding to the detection process, predicting the end time of the current call, and confirming whether to call the next call based on the end time. The customer service can answer the next call faster, reduce the waiting time of the customer service, and improve work efficiency.

Please refer to FIG. 2 which is a schematic diagram of the detection process provided in an embodiment of the present application.

Optionally, based on any embodiment, the user call data includes user call audio data and/or user call text data; the call detection result includes an emotion detection result; the user call data is detected and processed to obtain the call detection result, including: inputting the user call data into a preset emotion detection model to obtain the emotion detection result; the emotion detection model is obtained by training the preset emotion data and the emotion labels corresponding to the emotion data.

The user's emotions during a call can better reflect the user's call intention. For example, when the emotion is positive or neutral, the intention to answer the call is higher; when the emotion is negative, the intention to hang up the call is higher, that is, the call will end shorter. Therefore, the user's call data during the call can be emotionally detected to obtain the user's emotion detection results, and the emotion detection results can be used as one of the factors to predict the end time of the current call.

The process of generating an emotion detection model is described below. First, emotion data is obtained. The emotion data includes emotion text data and/or emotion audio data. Depending on the training data, the emotion detection model may include a text emotion detection model, an audio emotion detection model, and a multimodal emotion detection model. Among them, the text emotion detection model can be obtained by training emotion text data, the audio emotion detection model can be obtained by training emotion audio data, and the multimodal emotion detection model can be obtained by training information of both emotion text data and emotion audio data.

For the text emotion detection model, the emotion text data can be obtained by real-time speech ASR recognition of the user's voice. Emotion labels are added to the emotion text data. The emotion labels can be positive, negative, neutral, normal or abnormal, etc. The emotion labels depend on the required emotion classification granularity. The emotion labels can also be marked, for example, abnormal is 1 and normal is 0. The emotion text data is preprocessed and the text is converted into a fixed-length vector representation using word embedding (such as Word2Vec, GloVe, etc.). Then, a suitable machine learning algorithm or deep learning model is selected, and the preprocessed emotion text data and the corresponding emotion labels are input into the model for training. The text emotion detection model is obtained by adjusting the parameters of the model.

For the audio emotion detection model, you can add emotion tags to the emotion audio data. For emotion tags, see the above. You can also perform noise reduction, sampling conversion rate or frame division on the emotion audio data. Then extract the audio features and select the appropriate audio features as the training data for the audio emotion detection model according to the needs. Select the appropriate machine learning algorithm or deep learning model, such as convolutional neural network (CNN) and recurrent neural network (RNN), input the preprocessed audio features and corresponding emotion tags into the model for training, and obtain the audio emotion detection model by adjusting the parameters of the model.

For the multimodal emotion detection model, text features and audio features can be fused to obtain multimodal features. The fusion methods include concatenating or weighted averaging text features and audio features. Select a model that can handle multimodal input, such as a multimodal fusion model based on deep learning, input multimodal features into the model, and train it with a dataset with emotion labels. After adjusting the parameters of the model, a multimodal emotion detection model is obtained.

When performing emotion detection, the user call data is input into a preset emotion detection model to obtain an emotion detection result. For example, the user call audio data is input into an audio emotion detection model to obtain an emotion detection result; or the user call text data is input into a text emotion detection model to obtain an emotion detection result; or the user call audio data and the user call text data are input into a multimodal emotion detection model to obtain an emotion detection result.

In the above implementation process, the emotion detection model is used to perform emotion detection on the user call audio data and/or the user call text data in the user call data to obtain the emotion detection result. The emotional state can reflect the willingness of the phone user to answer or hang up the call to a certain extent. The emotion detection result is used as one of the factors for predicting the end time of the current call, thereby improving the accuracy of the end time prediction.

Optionally, based on any embodiment, the call detection result includes an audio detection result, and the user call data is detected and processed to obtain the call detection result, including: performing audio detection on the user call audio data in the user call data to obtain the audio detection result; the audio detection includes tone detection and/or decibel detection.

Tone detection or decibel detection can reflect the user's call intention to a certain extent. For example, if the tone is mostly that of a question sentence, it may indicate that the user is more interested in the content of the call and wants to inquire for more information.

Based on this consideration, tone detection and/or decibel detection can be performed on the user's call audio data, and the corresponding audio detection result includes a tone detection result and/or a decibel detection result. Tone detection, for example, extracts the frequency characteristics of the user's call audio data, identifies the tone based on the frequency characteristics, and obtains the tone detection result. Decibel detection, for example, measures the sound waves of the user's call audio data to obtain the decibel detection result.

In the above implementation process, audio detection is performed on the user call audio data in the user call data to obtain an audio detection result, which includes a tone detection result and/or a decibel detection result. The audio detection result is used as one of the factors for predicting the end time of the current call, thereby improving the accuracy of the end time prediction.

Optionally, based on any embodiment, the call detection result includes a call step detection result, and the user call data is detected and processed to obtain the call detection result, including: obtaining the customer service call data in the current call; inputting the customer service call data and the user call data into a preset call process model to obtain the call step detection result. For example, the call process model extracts features from the customer service call data and the user call data to obtain features to be matched; the call process model is used to classify and match the features to be matched with the step features in the call process model to obtain the target matching features, and the steps corresponding to the target matching features are used as the call step detection results.

The following describes the process of generating a call flow model. Feature extraction is performed on the question and answer data corresponding to each step in the call flow to obtain step features. The step features can reflect the characteristics of the process step to which the text belongs. Corresponding step labels are added to the step features, and the step labels are used to characterize the process step to which the question and answer data belongs. According to the characteristics of the problem and the scale of the data, a suitable machine learning or deep learning model is selected, and the model is trained using labeled data. The step features and the step labels corresponding to the step features are input into the model for training. By adjusting the parameters and structure of the model, the performance of the model is optimized to obtain the final call flow model.

In the above implementation process, the call process model is used to obtain the call step detection results based on the customer service call data and the user call data. The call step detection results represent the step of the current call, and to a large extent reflect the time required for the current call to end. For example, when the call is at a later step, the end time may be shorter. The end time of the current call is predicted based on the call step detection results to improve the accuracy of the end time prediction.

Optionally, based on any embodiment, the end duration of the current call is predicted based on the call detection result, including: using a preset fitting prediction function to obtain the end duration of the current call according to the call detection result.

The function expression of the preset fitting prediction function is an exponential function.

The call detection results include emotion detection results, audio detection results, speech speed detection results, keyword detection results, and call step detection results, and the fitting prediction function includes:

Among them, y is the end time of the current call, k is the first parameter, emotion.pt is the emotion detection result, a is the emotion detection parameter, Tone.pt is the audio detection result, b is the audio detection parameter, Speed.pt is the speech speed detection result, c is the speech speed detection parameter, Key.pt is the keyword detection result, d is the keyword detection parameter, Process.pt is the dialogue process detection result, and e is the dialogue process detection parameter.

The first parameter in the fitting prediction function, the emotion detection parameter, the audio detection parameter, the speech rate detection parameter, the keyword detection result, and the dialogue flow detection parameter can be determined based on historical data.

In the above implementation process, the preset fitting prediction function is used to obtain the end time of the current call according to the call detection result, and the next call is called when the end time of the current call is less than the time threshold. The customer service can answer the next call faster, reduce the waiting time of the customer service, and improve work efficiency.

Optionally, based on any embodiment, there may be a situation where the customer service's previous call has not been hung up, the intelligent outbound call system calls the next call, and the next call has been connected. In this case, the intelligent outbound call robot can be used to talk to the user of the next call, and the content of the conversation between the intelligent outbound call robot and the user of the next call can be displayed in real time, so that the manual customer service or agent can decide whether to take over the next call and serve the user of the next call based on the content of the call.

After confirming the next call, the method further includes: automatically calling the next call through the intelligent outbound calling system, and using the intelligent outbound calling robot to talk to the second user corresponding to the next call. The intelligent outbound calling robot is an artificial intelligence-based customer service function provided by the intelligent outbound calling system, and the intelligent outbound calling robot can communicate with the second user using voice.

Perform real-time speech recognition on the second user's call voice data to obtain the user's real-time call text and the time of the user's real-time call text. For example, during the second user's speech, perform real-time voice activity detection (VAD) detection on the call voice data. VAD detection can accurately locate the start and end points of the voice in the voice. When the second user is speaking, start the real-time stream ASR recognition service for the call voice data. The function of the ASR recognition service is to convert human voice into computer-readable text or commands. Record the user's real-time call text in the call voice data and the time of each sentence in the user's real-time call text. The time here refers to the time of the call process, such as the 10th second during the call, or the 1st minute during the call, etc.

Exemplarily, the real-time call text of the second user and the time of the real-time call text of the user can be expressed as: {"00.09": "What do you want from me?", "00.27": "You called the wrong number. I don't have a loan", "..."}, where "00.09" is the time of the voice of the real-time call text of the user "What do you want from me?", which can indicate the 9th second from the start of the call.

Obtain the real-time call text of the intelligent outbound call robot and the time of the real-time call text of the intelligent outbound call robot; the real-time call text of the intelligent outbound call robot is generated according to the preset text and/or the call voice data of the second user. In one case, the voice issued by the intelligent outbound call robot to the second user is synthesized according to the text, so there is no need to perform voice recognition on the intelligent outbound call robot, but the real-time call text of the intelligent outbound call robot and the time of the conversation between the intelligent outbound call robot and the second user can be directly obtained. Exemplary real-time call text of the intelligent outbound call robot and the time of the real-time call text of the intelligent outbound call robot can be expressed as: {"00.00": "Excuse me, are you Mr. XX?", "00.10": "Your loan in our bank is overdue", "..."}.

The real-time call text of the user and the real-time call text of the intelligent outbound call robot are sorted in chronological order to obtain the conversation content between the intelligent outbound call robot and the second user. Continuing with the above embodiment, the conversation content can be: {“00.00” intelligent outbound call robot: “Excuse me, are you Mr. XX?”, “00.09” second user: “What do you want to talk to me about?”, “00.10” intelligent outbound call robot: “Your loan in our bank has been overdue”, “00.27” second user: “You called the wrong number. I don’t have a loan”, “…”}.

The content of the conversation can be displayed on the display interface of the manual customer service, so that the manual customer service can confirm whether to take over the intelligent outbound call robot and continue the conversation with the second user based on the content of the conversation between the intelligent outbound call robot and the second user. For example, if the intelligent outbound call robot cannot determine the answer, the manual customer service can take over the intelligent customer service and answer the second user's questions to improve the user experience.

In the above implementation process, the conversation content between the intelligent outbound call robot and the second user is displayed in real time, so that the customer service can decide whether to take over the next call and serve the second user according to the conversation content. This reduces the waiting time of the customer service, improves work efficiency, and improves the user experience and service quality.

The speech emotion model in Figure 2 is used for emotion detection. Tone and decibel detection correspond to audio detection. It also includes speech rate detection, keyword detection and dialogue flow detection. These detections are used to predict the AI robot index, that is, to predict the end time of the current call and determine whether to make the next call based on the end time of the current call.

Please refer to FIG. 3 , which is a schematic diagram of obtaining a call summary based on a text summary model provided in an embodiment of the present application.

Optionally, based on any embodiment, the method further includes: obtaining conversation data of the current call; the conversation data includes conversation data between the intelligent outbound call robot and the user and/or conversation data between the customer service and the user; inputting the conversation data into a preset text summary model to obtain a call summary. As an implementation method, an interface corresponding to the text summary model can be called to generate a call summary.

For example, the conversation data may be: {“Agent A”: “Hello, sir”, “Customer B”: “What can I do for you?”, “Agent A”: “We are from an institution. Our institution has recently launched a loan product. Are you interested?”, “Customer B”: “Oh, your interest rate is a bit high”, “Agent A”: “Our interest rate is very low in the entire industry”, “….”}

The call summary can be: {“Business”: “Loan product”; “Is it resolved”: “Yes”; “Is there a follow-up visit”: “Yes”; “Summary of call content”: “The customer currently has no intention and also feels that the interest rate is high. If the interest rate is lowered, he hopes to go to the business hall to find out”}.

In an optional embodiment, after the call summary is generated by LORA fine-tuning, the target user's sentence can be generated by calling the interface API and performing voice conversion. At the same time, the call summary can also be generated by calling the interface API after the target user's sentence is obtained by voice conversion.

The following describes the process of generating a text summary model. LORA refers to the Low-Rank Adaptation method. The text summary model is obtained by fine-tuning the pre-trained model using LORA fine-tuning. It means fine-tuning the pre-trained model using the Low-Rank Adaptation method to obtain the text summary model. The pre-trained model includes building a large language model based on Qwen-72b or ChatGLM. LORA optimizes the pre-trained model by adding a small number of trainable parameters without modifying all the parameters of the original pre-trained model.

Fine-tune the pre-trained model using LoRa, including: loading the pre-trained model and keeping the preset weight parameters of the pre-trained model unchanged. For example, load a large pre-trained model to the GPU graphics card A100, ensure that the model can run correctly on the A100, and freeze the preset weight parameters of the pre-trained model.

Then add trainable layers to the pre-trained model. For example, inject LORA layers into each Transformer block in the pre-trained model according to the model architecture and fine-tuning requirements. These trainable layers will be used to replace some of the original parameters and set the appropriate rank size, which determines the scale of the trainable layer.

Prepare a conversation sample dataset for fine-tuning and load it onto the GPU graphics card A100. The conversation sample dataset should be preprocessed in an appropriate format so that it can be input into the pre-trained model. The format of the conversation sample dataset refers to the conversation data and call summary of "Agent A" and "Customer B" mentioned above. If the conversation sample dataset is the user's real data, the user's consent must be obtained before recording.

Through the low-rank adaptive method, the trainable layer is trained using the preset dialogue sample dataset; the pre-trained model is updated with the trained trainable layer to obtain the text summarization model. Configure the training parameters, such as learning rate, batch size, number of training rounds, etc. These parameters will affect the effect and speed of fine-tuning. Start the training process on the GPU graphics card A100. Only the parameters of the LORA layer will be updated, while the preset weight parameters of the original pre-trained model remain unchanged.

In the above implementation process, the conversation data is input into the preset text summary model to obtain the call summary, which reduces the time of manually recording the call summary and improves the summary efficiency. In addition, the text summary model is obtained by fine-tuning the pre-trained model with LORA to improve the problem of the weak text content summary ability of the pre-trained model. After fine-tuning, its summary ability in professional fields is strengthened and the accuracy of the summary is improved.

In an optional embodiment, the content of the conversation can also be quality checked. At the end of the conversation between the customer service and the client, it is necessary to perform voice quality checks on the client side and the customer service voice to identify whether the customer service client has a tendency to complain, whether the customer service is operating in accordance with the process requirements, whether the customer service is operating in compliance, etc., to quality check whether there are any problems with the content of the conversation that need to be dealt with immediately or an alarm needs to be issued.

Quality inspection includes voice quality inspection on the client side and voice quality inspection on the customer service side. Voice quality inspection on the client side: For the client side, it is to identify complaints, loss of connection, etc. After obtaining the user's consent, keyword matching can be used, such as "complaint" and other words to identify whether the user has complained; if the keyword is not hit, Fasttext is used to establish a complaint semantic prediction model to identify whether the user has complained based on the text conversation content to achieve semantic recognition. Voice quality inspection on the customer service side: For the customer service side, it is necessary to identify whether the agent is operating according to the process, whether the customer service is emotionally unstable, yelling loudly or interrupting, etc., identify abnormal behavior on the agent side based on keyword matching and regular matching, segment the voice with call VAD detection, and locate whether there is interruption based on the time of the segmented voice.

Please refer to FIG. 4 for a flow chart of the speech recommendation process shown in the present application.

Optionally, based on any embodiment, the method further includes: identifying the user call audio data in the user call data to obtain the user call text data; predicting punctuation marks on the user call text data, and dividing the user call text data according to the punctuation mark prediction results to obtain the target user sentence. The target user sentence can be any sentence in the sentence set divided from the call text data. In order to improve effectiveness, the target user sentence can also be the latest sentence in the sentence set divided from the call text data, that is, the latest question raised by the user.

Input the target user's sentence into the preset speech recommendation model, and use the preset speech recommendation model to obtain the corresponding reply speech according to the target user's sentence. The reply speech can be combined with the preset prompt template and displayed to the customer service, who will answer the user.

The following describes the process of generating a speech recommendation model. Obtain the data of the knowledge base and the pre-built large language model, as shown in Figure 4. The knowledge base data can be a local question-and-answer knowledge base, and the large language model can be used for large-model local question-and-answer. Large-model local question-and-answer refers to the process of using a large language model to perform local question-and-answer queries. The knowledge base data can be question-and-answer data in a business field. The question-and-answer data can be cleaned, marked, and preprocessed. The preprocessing includes texting the data of the knowledge base, segmenting the text, and generating text chunks. The large language model can be LLM. Select a suitable LLM model architecture: for example, the Transformer architecture.

Integrate the large language model with the language chain tool: The language chain tool can be lang-chain, which is a framework for building complex text processing pipelines. Lang-chain can interact with specific types of API interfaces. Therefore, a wrapper can be created for LLM to conform to the expected format of lang-chain. The expected format of lang-chain includes input format and output format, etc., where the input format can be a string or an instruction, etc.; the output format can also be a string, or JSON, XML or other structured data formats, etc. This wrapper should be able to receive the request sent by lang-chain, convert it into a format that LLM can understand, and then integrate the large language model with the language chain tool.

Use the language chain tool to vectorize the knowledge base data. Use the tools or APIs provided by lang-chain to import the processed documents. In this process, lang-chain will use some vector representation method (such as word embedding, TF-IDF, etc.) to convert the document content into vector form, which can be recorded as vector Store. These vectors can capture the semantic information in the document, making the subsequent vector similarity calculation and question-answer matching more accurate.

With the vectorized knowledge base, we can obtain the speech recommendation model. We can vectorize the target user's sentence, that is, the query vectorization in Figure 4, to obtain the user vector, which is the query vector. We can perform a vector similarity query between the user vector and the vectorized knowledge base (vector store) in the speech recommendation model to obtain the reply speech corresponding to the user vector and mark it as related chunks. We can obtain the prompt template and the prompt word from the user's query, input the prompt word into the prompt template, generate the question in the question-answer query, and then use the large model (large language model) to generate speech.

In the above implementation process, the preset speech recommendation model is used to obtain the corresponding reply speech according to the target user's sentence, thereby improving the accuracy and efficiency of the speech recommendation.

Please refer to FIG. 5 which is a schematic diagram of the process of the intelligent call shown in the present application.

In an optional embodiment, when using the intelligent call system for the first time, that is, when AI calls for the first time, the user can choose an intelligent outbound call robot related to his or her business to make a call. After the intelligent outbound call robot obtains the mobile phone number and customer information, it uses the active outbound call function to call the SIP of the call platform to make a call. When the call is connected, the transfer to manual function of the intelligent call system is used to insert the call into the manual agent queue. After the call enters the manual queue, the agent will immediately pick up the call and talk to the customer. The user experience is improved without the user's perception of the automatic call process.

After that, the agent will have a real-time conversation and recommend words of conversation. For example, after the agent picks up the phone and communicates with the customer in real time, he can use the preset word recommendation model to obtain the corresponding reply words according to the target user's statement. And during the call, the conversation can be used to predict the end in real time to confirm whether to make the next call, that is, AI predictive call. If the intelligent outbound robot dials the next call in advance, the customer service can confirm whether to take over the intelligent outbound robot based on the conversation content between the intelligent outbound robot and the second user, and continue to talk to the second user, so as to realize the collaborative operation of the customer service agent and the intelligent outbound robot, that is, to realize the collaboration between the agent and AI predictive outbound call.

Please refer to FIG. 6 for a schematic diagram of the structure of an AI-based intelligent voice call prediction device 200 provided in an embodiment of the present application. The embodiment of the present application provides an AI-based intelligent voice call prediction device 200, including:

The user data receiving module 210 is used to receive the call data of the user in the current call;

The detection module 220 is used to detect and process the user's call data to obtain a call detection result; the detection process includes at least one of emotion detection, audio detection, speech speed detection, keyword detection and dialogue flow detection;

The call prediction module 230 is used to predict the end time of the current call based on the call detection result, and determine whether to make the next call based on the end time of the current call.

Optionally, based on any embodiment, in the AI-based intelligent voice call prediction device 200, the user call data includes user call audio data and/or user call text data; the call detection result includes an emotion detection result; the detection module 220 is also used to input the user call data into a preset emotion detection model to obtain an emotion detection result; the emotion detection model is obtained by training the preset emotion data and the emotion labels corresponding to the emotion data.

Optionally, based on any embodiment, the AI-based intelligent voice call prediction device 200, the call detection result includes an audio detection result, and the detection module 220 is also used to perform audio detection on the user call audio data in the user call data to obtain the audio detection result; the audio detection includes tone detection and/or decibel detection.

Optionally, based on any embodiment, the AI-based intelligent voice call prediction device 200, the call detection result includes the call step detection result, the detection module 220, and is also used to obtain customer service call data in the current call; the customer service call data and the user call data are input into a preset call process model to obtain the call step detection result; the call process model obtains step features by extracting features from the question and answer data corresponding to each step in the call process; adding corresponding step labels to the step features; and training the step features and the labels corresponding to the step features.

Optionally, based on any embodiment, the AI-based intelligent voice call prediction device 200, the call prediction module 230 is used to obtain the end duration of the current call according to the call detection result using a preset fitting prediction function.

Optionally, based on any embodiment, the AI-based intelligent voice call prediction device 200, the call detection result includes an emotion detection result, an audio detection result, a speech speed detection result, a keyword detection result, and a call step detection result, and the fitting prediction function includes:

Among them, y is the end time of the current call, k is the first parameter, emotion.pt is the emotion detection result, a is the emotion detection parameter, Tone.pt is the audio detection result, b is the audio detection parameter, Speed.pt is the speech speed detection result, c is the speech speed detection parameter, Key.pt is the keyword detection result, d is the keyword detection parameter, Process.pt is the dialogue process detection result, and e is the dialogue process detection parameter.

Optionally, based on any embodiment, the AI-based intelligent voice call prediction device 200 also includes a call collaboration module, which is used to call the next call and use the intelligent outbound call robot to talk to the second user corresponding to the next call; perform real-time voice recognition on the call voice data of the second user to obtain the user's real-time call text and the time of the user's real-time call text; obtain the intelligent outbound call robot's real-time call text and the time of the intelligent outbound call robot's real-time call text; the intelligent outbound call robot's real-time call text is generated according to the preset text and/or the call voice data of the second user; sort the user's real-time call text and the intelligent outbound call robot's real-time call text in chronological order to obtain the conversation content between the intelligent outbound call robot and the second user; display the conversation content so that the customer service can confirm whether to take over the intelligent outbound call robot and continue to talk with the second user based on the conversation content between the intelligent outbound call robot and the second user.

Optionally, based on any embodiment, the AI-based intelligent voice call prediction device 200 also includes: a call summary module, which is used to obtain conversation data of the current call; the conversation data includes conversation data between the intelligent outbound call robot and the user and/or conversation data between the customer service and the user; the conversation data is input into a preset text summary model to obtain a call summary; the text summary model is obtained by performing LORA fine-tuning on the pre-trained model; performing LORA fine-tuning on the pre-trained model, including: loading the pre-trained model and keeping the preset weight parameters of the pre-trained model unchanged; adding a trainable layer to the pre-trained model; training the trainable layer using a preset conversation sample data set through a low-rank adaptive method; updating the pre-trained model using the trained trainable layer to obtain a text summary model.

Optionally, based on any embodiment, the AI-based intelligent voice call prediction device 200 also includes: a speech recommendation module, which is used to identify the user call audio data in the user call data to obtain the user call text data; predict punctuation marks on the user call text data, and divide the user call text data according to the punctuation mark prediction results to obtain the target user sentences; use the preset speech recommendation model to obtain the corresponding reply speech according to the target user sentences; the speech recommendation model obtains the knowledge base data and the pre-built large language model; integrates the large language model and the language chain tool; and uses the language chain tool to vectorize the knowledge base data.

It should be understood that the device corresponds to the above-mentioned AI-based intelligent voice call prediction method embodiment and can execute the various steps involved in the above-mentioned method embodiment. The specific functions of the device can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system (OS) of the device.

Please refer to the structural diagram of the electronic device provided by the embodiment of the present application shown in Figure 7. An electronic device 300 provided by the embodiment of the present application includes: a processor 310 and a memory 320, the memory 320 stores machine-readable instructions executable by the processor 310, and the machine-readable instructions are executed by the processor 310 to perform the above method.

An embodiment of the present application further provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above method is executed.

Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a part of the module, program segment or code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and/or flowchart, and the combination of boxes in the block diagram and/or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

The above description is only an optional implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims (12)

1. An AI-based intelligent voice call prediction method, characterized by comprising: Receive call data of the user in the current call; Performing detection processing on the user call data to obtain a call detection result; the detection processing includes at least one of emotion detection, audio detection, speech speed detection, keyword detection, and dialogue flow detection; Based on the call detection result, the end time of the current call is predicted, and based on the end time of the current call, it is determined whether to make a next call. 2. The method according to claim 1, characterized in that the user call data includes user call audio data and/or user call text data; the call detection result includes an emotion detection result; performing detection processing on the user call data to obtain the call detection result comprises: The user call data is input into a preset emotion detection model to obtain the emotion detection result; the emotion detection model is obtained by training the preset emotion data and the emotion labels corresponding to the emotion data. 3. The method according to claim 1, wherein the call detection result includes the audio detection result, and the detection and processing of the user call data to obtain the call detection result comprises: Performing audio detection on the user call audio data in the user call data to obtain the audio detection result; the audio detection includes tone detection and/or decibel detection. 4. The method according to claim 1, wherein the call detection result includes a call step detection result, and the call detection result is obtained by performing detection processing on the user call data, comprising: Obtain customer service call data in the current call; The customer service call data and the user call data are input into a preset call process model to obtain the call step detection result; the call process model obtains step features by extracting features from the question and answer data corresponding to each step in the call process; corresponding step labels are added to the step features; and the step features and the labels corresponding to the step features are trained to obtain the features. 5. The method according to claim 1, characterized in that predicting the end duration of the current call based on the call detection result comprises: The end duration of the current call is obtained according to the call detection result by using a preset fitting prediction function. 6. The method according to claim 5, characterized in that the call detection result includes an emotion detection result, an audio detection result, a speech rate detection result, a keyword detection result and a call step detection result, and the fitting prediction function includes: Among them, y is the end duration of the current call, k is the first parameter, emotion.pt is the emotion detection result, a is the emotion detection parameter, Tone.pt is the audio detection result, b is the audio detection parameter, Speed.pt is the speech speed detection result, c is the speech speed detection parameter, Key.pt is the keyword detection result, d is the keyword detection parameter, Process.pt is the dialogue process detection result, and e is the dialogue process detection parameter. 7. The method according to claim 1, characterized in that after confirming the next call, the method further comprises: Call the next call, and use the intelligent outbound calling robot to talk to the second user corresponding to the next call; Performing real-time voice recognition on the call voice data of the second user to obtain the real-time call text of the user and the time of the real-time call text of the user; Acquire the real-time call text of the intelligent outbound calling robot and the time of the real-time call text of the intelligent outbound calling robot; the real-time call text of the intelligent outbound calling robot is generated according to the preset text and/or the call voice data of the second user; Sort the real-time call text of the user and the real-time call text of the intelligent outbound call robot in chronological order to obtain the conversation content between the intelligent outbound call robot and the second user; The conversation content is displayed so that the customer service can confirm whether to take over the intelligent outbound call robot and continue talking with the second user based on the conversation content between the intelligent outbound call robot and the second user. 8. The method according to claim 1, characterized in that the method further comprises: Acquire the conversation data of the current call; the conversation data includes the conversation data between the intelligent outbound call robot and the user and/or the conversation data between the customer service and the user; The conversation data is input into a preset text summary model to obtain a call summary; the text summary model is obtained by performing LORA fine-tuning on a pre-trained model; the LORA fine-tuning on the pre-trained model includes: loading the pre-trained model and keeping the preset weight parameters of the pre-trained model unchanged; adding a trainable layer to the pre-trained model; training the trainable layer using a preset conversation sample data set through a low-rank adaptive method; and updating the pre-trained model using the trained trainable layer to obtain the text summary model. 9. The method according to any one of claims 1 to 8, characterized in that the method further comprises: Identifying the user call audio data in the user call data to obtain the user call text data; Predicting punctuation marks on the user call text data, and dividing the user call text data according to the punctuation mark prediction results to obtain target user sentences; Using a preset speech recommendation model, the corresponding reply speech is obtained according to the target user's sentence; the speech recommendation model obtains knowledge base data and a pre-built large language model, and integrates the large language model with a language chain tool; and the knowledge base data is vectorized using the language chain tool. 10. A computer program product, comprising computer program instructions, wherein when the computer program instructions are executed by a processor, the method according to any one of claims 1 to 9 is performed. 11. An electronic device, comprising: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the method according to any one of claims 1 to 9 is performed. 12. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is executed.