WO2025176402A1 - Method for adapting an infotainment system, infotainment system, and motor vehicle - Google Patents

Abstract

The invention relates to a method for adapting an infotainment system (35) comprised in a motor vehicle (32) to user preferences. For this purpose, a contextual trigger (15) is detected and a time (16) at which a user (33) is receptive to a notification (11) is determined. Subsequently, the notification (11) is formulated to the user (33) on the basis of the detected contextual trigger (15) and the determined time (16), wherein the notification (11) aims to determine user preferences of the user (33). User preferences are then determined on the basis of the reaction (23) of the user (33) to the notification (11). A user profile is then created on the basis of the determined user preferences. Settings (38) of the infotainment system (35) are then derived from the determined user preferences and are applied.

Description

Method for adapting an infotainment system and infotainment system and motor vehicle

DESCRIPTION:

The invention relates to a method for adapting an infotainment system (information entertainment system) included in a motor vehicle to user preferences.

A modern vehicle system, such as a speech dialogue system in a motor vehicle, often depends on user-initiated inputs and is therefore static. If a dialogue initiated by the speech dialogue system, e.g., to determine user preferences, does not immediately lead to success, the user must repeatedly answer the same question or manually enter an input to answer this question.

US 2021/0 326 344 A1 discloses a virtual assistance system for a user with a vehicle computing unit that has a function for situation-based generation of audiovisual questions. The audiovisual questions are issued according to a stored user profile and a current user situation determined by sensors.

US 2022/0 051 669 A1 discloses an interactive assistance device of a vehicle navigation system for a user with a proactive voice output function. To supplement a user profile, the voice output function asks the user situation-dependent questions based on previous recommendations and/or determined function usage behavior. The invention is based on the object of proactively and situation-appropriately suggesting to a user functions of a vehicle system that are rarely or never used.

The problem is solved by the subject matter of the independent patent claims.

Advantageous further developments of the invention are described by the dependent claims, the following description and the figures.

The invention provides a method for adapting an infotainment system included in a motor vehicle to user preferences. The method comprises the following steps:

First, one or at least one contextual trigger is detected. Such a trigger could, for example, be an environmental condition, such as an outside temperature below a predefined threshold or within a predefined value interval. For example, an outside temperature of -5 degrees Celsius to 15 degrees Celsius can be represented as a contextual trigger. Another contextual trigger could be a specific day of the week and/or month and/or a time of year and/or a specific location and/or a network status, in particular a battery status of a device or system and/or a lighting condition and/or a specific weather condition, such as snowfall. If it is determined that it is currently snowing and/or the outside temperature is, for example, -5 degrees Celsius, then the vehicle heating, for example, to 20 degrees Celsius, is automatically activated or suggested via the infotainment system, to give one example.

The determination of such a contextual trigger can be realized with a sensor device, such as a temperature sensor and/or a snow depth sensor and/or a light sensor and/or a global navigation satellite system (GNSS), which can be taken from the state of the art. For example, to determine temporal variables such as time and/or day of the week and/or month, the Infotainment system may have a digital calendar system and/or a Real-Time Clock (RTC) module.

Next, a point in time is determined at which a user is receptive to a hint. A hint can be a (proactive) question and/or a (proactive) suggestion from a voice dialogue system. Such a point in time can be, for example, when the user is currently stuck in a traffic jam and/or the driving speed of their vehicle is within a predetermined range, such as 0 km/h to 30 km/h. At such a point in time or moment, the user's attention is therefore preferably not heavily demanded. A speed sensor can be used for this purpose to signal such a point in time. Additionally or alternatively, such a point in time can be a subsequent one. If the interior noise of the vehicle is, for example, below a predetermined threshold, such as 10 to 65 decibels, this can be seen as a point in time at which the user is receptive to a hint.

Subsequently, once at least one contextual cue and a time at which the user is receptive to a cue have been identified, the cue is formulated for the user. The cue aims to determine the user's preferences. For example, the voice dialog system can issue a cue such as: "What is your preference for the seat massage?" or "Would you like to try the 'Wave' massage function?" The voice dialog system can thus use the cue to suggest activating a function that the user might want to use.

Depending on the user's reaction to the hint, user preferences are then determined. The speech dialogue system can therefore interpret a user's reaction, for example, by applying speech recognition. Additionally or alternatively, an emotion recognition device can be included in the motor vehicle, which is configured to determine a user's reaction and/or forward it to the infotainment system. Thus, if the user suggests a If a user requests a function of the speech dialog system, such as the massage function mentioned above, the speech dialog system initiates the execution of this function. The speech dialog system can then receive some kind of feedback from the user to obtain a rating for the function performed. If the user likes the function, they can communicate this manually or verbally to the speech dialog system, so that the resulting user preferences are saved.

These user preferences can consist of at least one of the following information: the contextual trigger and/or the time that caused the voice dialog system to introduce and present the function via a notification to the user and/or preferred day or time of day for using the function and/or frequency of requesting the function and/or duration of use of the function and/or feedback in the form of a star rating and/or a verbal comment made by the user after using the function and/or reaction time between the suggestion of the function by the voice dialog system and the confirmation or rejection by the user and/or emotion or mood captured from the user’s voice and/or cancellation or switching to another function during the use of the suggested function and/or an improvement suggested by the user. Function can, for example, mean B may mean: seat heating and/or an adaptive lighting system and/or a car radio and/or automatic climate control and/or the sunroof and/or a parking aid and/or an automatic start-stop system and/or a WLAN (Wireless Local Area Network) hotspot.

A user profile containing these user preferences is then created for the user. Infotainment system settings are derived or learned based on the user preferences. Alternatively, only one setting can be derived. For example, if the user preferences include the information that the user activates the seat heating when the outside temperature drops below 10 degrees Celsius, this function can be be started automatically when such an outside temperature is measured.

Settings of the infotainment system are therefore derived from the determined user preferences, in particular using the user profile, and these settings are applied.

The advantage of the invention is that the user's sense of comfort can be increased because settings can be automatically made based on their user preferences. Another advantage of the invention is that a user can become familiar with new and/or unused vehicle functions when they are clearly receptive to them. This can positively influence the user experience. Furthermore, the learning phase for creating the user profile can be shortened because the voice dialog system does not have to passively observe the user's behavior, but can actively ask about their preferences and save them as user preferences.

The invention also includes embodiments which provide additional advantages.

A further development provides that the notification refers to a function that the user has not yet activated at the time of receiving the notification and/or the user's frequency of use for this function falls below a predetermined threshold. This threshold can, for example, be in a range from 1 to 10. This can be achieved, for example, using a profiling tool that is part of the infotainment system. This allows a user to become familiar with a function that is not yet used or is rarely used.

A further development provides for the frequency of use to be determined using a histone, whereby the histone quantifies which functions of the vehicle and/or how often these have been activated in the past while the user was registered as a passenger. A state-of-the-art recognition device, such as a facial recognition system and/or key recognition, can be used to identify the user as a passenger. The histone is therefore a record or log of which functions of the vehicle were used by the user and how often.

A further development provides that in step a), the contextual trigger includes an estimated travel time and/or a planned destination and/or the current outside temperature and/or the user's usage profile. As already described, such information, such as the estimated travel time, can be read by a sensor, in particular by a global navigation satellite system. If the estimated travel time is, for example, more than ten minutes, the user can be asked, if there is also a time at which they are receptive to a notification, whether they would like to listen to a podcast, for example.

A further development provides that, in step b), a time is determined depending on a driving speed and/or the interior noise of the motor vehicle and/or a determined number of occupants in the motor vehicle and/or the surrounding traffic density. If the infotainment system detects, for example, via an environmental sensor, in particular via an ultrasonic sensor and/or radar sensor, that the user is in a traffic jam, the voice dialogue system can be prompted to formulate a notification to the user. This has the advantage that the user is only provided with a notification in at least slightly distracting situations.

A further development provides that step c) is based on an "Active Question Answering" voice dialogue, which is initiated by a voice dialogue system and carried out with the user. The voice dialogue system is designed to receive spoken and/or text-based and/or menu-based feedback from the user. To implement the voice dialogue, the expert can refer to the Publication by Buck, Christian et al. “Ask the Right Questions: Active Question Reformulation with Reinforcement Learning.” orientate.

For this purpose, a further development provides for the speech dialogue system to have an artificial neural network trained using reinforcement learning on data sets (experience data sets) consisting of question-answer pairs to formulate instructions depending on at least one contextual trigger and/or a determined point in time. The term "artificial neural network" refers to the software that enables the speech dialogue in the speech dialogue system.

A further development provides that steps b) to e) are skipped as soon as the artificial neural network signals that a predefined threshold for the contextual trigger has been reached. For example, if the neural network has learned a certainty or confidence of 80 to 90% in percentage points regarding a contextual trigger, steps b) to e) are skipped. For example, the neural network can have linked a confidence of 80% for a specific outside temperature, such as -5 to 5 degrees Celsius, as a contextual trigger, so that the user no longer needs to be asked whether the seat heating should be activated, as this now happens automatically. If the user then manually deactivates the seat heating, for example, this learned contextual trigger can be discarded, at least iteratively. This means that the confidence would be less than 80% the next time that contextual trigger occurs.

A further development provides that step c) involves reformulating the question if no user preferences could be determined and/or if a user gesture of confusion is detected. Reformulating involves asking a question that is positioned directly after the initially formulated question according to an ordered probabilities list based on the results of the speech dialogue. The reformulation of the question is therefore based on an ordered list of probabilities derived from the results of the previous speech dialogue. The new question is placed immediately after the original question. This can ensure that misunderstandings are reduced and/or user preferences are reliably determined.

A further development provides that the helplessness gesture comprises detecting, by means of image-based and/or radar-based and/or lidar-based and/or microphone-based user monitoring, that the user shrugs his shoulders and/or frowns and/or groans and/or performs a predetermined gesture to request help in response to a notification issued by the infotainment system.

A further development provides that the following response is recorded: verbal and/or haptic feedback and/or a gesture, e.g., captured by a camera, such as a nod of the head, and/or a period of time that has passed, i.e., ignoring the hint. This period can be, for example, 10 to 30 seconds. Additionally or alternatively, a change in user settings and/or an emotional reaction can be recorded as a response. A change in user settings can, for example, be muting or reducing the volume, e.g., by 3 to 10 units or 10 to 20 decibels, of the infotainment system. An emotional reaction can be recorded by a biometric sensor, such as a skin conductivity sensor and/or facial recognition software for identifying microexpressions and/or a thermal camera for measuring temperature changes in the face.

Additionally or alternatively, the voice pitch can be captured, for example, via a microphone in such a way that frequency changes of the voice at the end of words (e.g., the distinction between "rising,""falling,""constant") are fed into the neural network as input data. The combination of facial expressions and voice pitch provides the particular advantage of capturing audiovisual signals to interpret the user's emotional reaction. A further development provides that activation or deactivation for deriving at least one setting is controlled depending on a manual input or feedback. In other words, the user can specifically control or deactivate a learning process of the infotainment system. If the user initiates the start of such a learning process, it can be provided that user preferences are determined depending on a contextual trigger that has already been recorded or is to be recorded. The activation of this learning process can, for example, be set such that the infotainment system should record user preferences for the next half hour or for the next two hours, or (initially) without a time limit. Likewise, it can be provided that no user preferences should be determined for the next half hour or for the next two hours, or (initially) without a time limit. This gives the user control over when user preferences should be recorded. Furthermore, concerns about data protection can be mitigated. During a journey with multiple occupants, for example, different styles of music can be played. In order to prevent the user from being assigned other people's music preferences as user preferences, it is therefore useful to be able to deactivate the learning process manually.

A further development provides that transparency of the derived settings is ensured, at least in part, by displaying them in an AI menu or a user interface of the infotainment system. In other words, for a setting, it can be disclosed from which user preferences this setting was achieved. Transparency can be ensured, for example, by displaying a histone or a protocol from which the user can, for example, read the user frequency for a function and/or a contextual trigger. This allows the user to (better) understand their own habits and/or preferences. If an "incorrectly" learned setting is found, i.e., a setting that the user does not like, the cause can be checked. By ensuring such transparency, it is more likely that the user will trust the Infotainment system developed, improving user experience.

For use cases or application situations that may arise during the method and which are not explicitly described here, it may be provided that, in accordance with the method, an error message and/or a request to enter user feedback is issued and/or a default setting and/or a predetermined initial state is set.

The invention also includes an infotainment system for a motor vehicle. The infotainment system can have a data processing device or a processor device configured to carry out an embodiment of the method according to the invention. For this purpose, the processor device can have at least one microprocessor and/or at least one microcontroller and/or at least one FPGA (Field Programmable Gate Array) and/or at least one DSP (Digital Signal Processor). In particular, a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), or an NPU (Neural Processing Unit) can be used as the microprocessor. Furthermore, the processor device can have program code configured to carry out the embodiment of the method according to the invention when executed by the processor device. The program code can be stored in a data memory of the processor device. The processor device can be based, for example, on at least one circuit board and/or on at least one SoC (System on Chip).

The invention also includes further developments of the infotainment system and/or motor vehicle according to the invention that have features already described in connection with the further developments of the method according to the invention. For this reason, the corresponding further developments are not described again here. The motor vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger car or truck, or as a passenger bus or motorcycle.

As a further solution, the invention also encompasses a computer-readable storage medium comprising program code which, when executed by a computer or computer network, causes the computer or computer network to execute an embodiment of the method according to the invention. The storage medium can be provided at least partially as a non-volatile data memory (e.g., as a flash memory and/or as an SSD - solid state drive) and/or at least partially as a volatile data memory (e.g., as a RAM - random access memory). The storage medium can be arranged in the computer or computer network. However, the storage medium can also be operated, for example, as a so-called app store server and/or cloud server on the Internet. The computer or computer network can provide a processor circuit with, for example, at least one microprocessor. The program code can be provided as binary code and/or as assembly code and/or as source code of a programming language (e.g., C) and/or as a program script (e.g., Python).

The invention also encompasses combinations of the features of the described embodiments. The invention therefore also encompasses implementations that each comprise a combination of the features of several of the described embodiments, unless the embodiments are described as mutually exclusive.

Exemplary embodiments of the invention are described below. Shown are:

Fig. 1 shows a system for carrying out the method according to the invention; Fig. 2 shows a technical implementation for carrying out the method according to the invention; and

Fig. 3 shows an embodiment of the invention in a motor vehicle.

The exemplary embodiments explained below are preferred embodiments of the invention. In the exemplary embodiments, the described components of the embodiments each represent individual features of the invention that can be considered independently of one another, each of which also develops the invention independently of one another. Therefore, the disclosure is intended to encompass combinations of the features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

In the figures, the same reference symbols designate elements with the same function.

Fig. 1 shows a system for implementing the idea described above. This system can consist of two parts: one is an existing part, geni:OS 14 (see also https://www.semvox.de/technologien/genios/), and the other is an integrated question-answering model 13. A geni:OS IQA agent 1 can be integrated into geni:OS 14, which interacts with the following components. These components can be previously described sensors 6 and/or a dialog histone or interaction history or protocol 4 and/or missing information 12. This missing information 12 can be extracted from a central user profile 3. In other words, if missing information 12 is detected, this missing information 12 can be extracted from the central user profile 3. For example, for formulating a hint 11, the information about which music a user 33 likes to listen to could be relevant, so this information is extracted from the central user profile 3. Each of these components 6, 4, 12 can be connected to an encoder network 2, 2' or 2", respectively, creating so-called hidden states. From this, a concatenation 8 can then be performed, so that these hidden states are summarized. Additionally or alternatively, it can be provided that the geni:OS IQA agent 1 performs a language style adaptation 9 from the dialogue histone 4 and/or the missing information 12, e.g., using a neural style transfer model. This language style adaptation 9 can be added to the concatenation 8, so that the content of the concatenation 8 is adapted by the language style adaptation 9. This content can then be fed to a decoder network 10. Overall, it is provided that the question answering model or the artificial neural network 13, as already described, is trained by means of reinforcement learning on data sets (experience data sets) consisting of question-answer pairs from a question corpus 5, to detect clues or formulate questions 11 depending on at least one contextual trigger 15 and/or a determined time 16. A generated hint 11 can be sent to the geni:OS IQA Agent 1, which then reproduces it as a voice output. The geni:OS IQA Agent 1 can therefore refer to a specific voice dialog system.

Fig. 2 shows a technical implementation of the idea. A previously described contextual trigger 15 can, for example, be determined and/or signaled by haptic and/or verbal interaction. Furthermore, a time 16, also described, at which a user 33 is receptive to a hint 11 can be determined by various sensors 6 already mentioned above. These sensors 6 can determine and/or evaluate an emotion and/or a stress and/or a fatigue level of the user 33. A data fusion 17 can be performed based on the determined time 16 and by adding user preferences from the central user profile 3. This data fusion 17 can comprise the dialog histone 4 and/or a rule set 18 for implementing this data fusion 17. A result 19 can result therefrom. It can be provided that the result 19 is subjected to a completeness criterion, so that completeness or incompleteness is signaled. This completeness criterion can, for example, be an achieved confidence, whereby this confidence can be calculated or read from the dialogue history 4. If this confidence is below a predetermined threshold value, the speech dialogue system 1 can be prompted to formulate a hint 11. For this purpose, for example, an existing question can be taken and/or asked from a question corpus 5 comprising question-answer pairs, so that a hint 11 is formulated. The aforementioned language style adaptation 9 can be carried out on this hint 11. The result of this can be a reformulated question 24, after which a question-answer dialogue 25 can be conducted. This question-answer dialogue 25 can then be subjected to an evaluation 28.

However, if the confidence is above a predetermined threshold, this may mean that an already learned action 21 should be carried out. A confirmation prompt 22 can be sent to the user 33. A reaction 23 from the user 33 can then be determined or recorded. Following this, an evaluation of the interaction 28 can be carried out. If this evaluation 28 contains a confidence below a threshold, the result is considered incomplete and such a signal is sent to a reformulation model 30 in order to create a reformulated question 31. This reformulated question 31 can be, for example, an explanation, recommendation or follow-up question. Based on this reformulated question 31, a question-answer dialogue 25 can be carried out again, which is then subjected to another evaluation 28. The question-and-answer dialogue 25 can be based on an NLU (Neural Language Understanding) corpus 26 and/or on emotion recognition 27 and/or on a dialogue history or interaction history 4. If the result is considered complete, the interaction can be learned or refined as an action, or even discarded, so that an adaptation of the AI interaction 29 is performed. Fig. 3 shows the user 33 in a motor vehicle 32. This motor vehicle 32 comprises an infotainment system 35 having a processor device 36. The infotainment system 35 also includes the voice dialogue system 1. Furthermore, the motor vehicle 32 has a temperature sensor 6. Symbolically represented is an outside temperature 37 of -5 degrees Celsius. This outside temperature 37 can be detected as a contextual trigger 15 via the temperature sensor 6 in the infotainment system 35. Next, a time 16 at which the user 33 is receptive to a hint 11 can be determined. If such a time 16 is determined, the hint 11 can be formulated to the user 33 based on the detected contextual trigger 15 and the determined time 16, wherein the hint 11 aims to determine user preferences of the user 33. This hint 11 is symbolically depicted as a question mark. Depending on the user's 33 reaction 23 to the notification 11, user preferences can be determined. A user profile can be created based on the user preferences. Settings 38 of the infotainment system 35 can then be derived and applied from the determined user preferences.

Overall, the examples show how an Inverse Question Answering (IQA) speech dialogue system can be provided.

List of reference symbols

1 speech dialogue system

2 Encoder Block 1

2‘ Encoder Block 2

2" Encoder Block 3

4 dialogue histones

5 question corpus

6 Sensor device

7 Edge endpoint

8 Concatenation

9 Language style adaptation

10 decoders

11 Note

12 missing information

13 Inverse QA Model

14 geni:OS Kl Interaction

15 contextual triggers

16 Time

17 Data fusion

18 Standard rate

19 results

20 Inverse QA

21 Learned Action

22 Confirmation prompt

23 Reaction

24 reformed question

25 Question-Answer Dialogue

26 NLU corpus

27 Emotion recognition

28 Evaluation

29 Adaptation of the AI interaction

30 Reformulation model

32 Motor vehicle 33 users

35 Infotainment system

36 Processor setup

37 Outside temperature 38 Setting

Claims

PATENT CLAIMS: 1 . Method for adapting an infotainment system (35) included in a motor vehicle (32) to user preferences, the method comprising: a) detecting a contextual trigger (15), b) determining a time (16) at which a user (33) is receptive to an indication (11), c) formulating the indication (11) to the user (33) based on the detected contextual trigger (15) and the determined time (16), the indication (11) aiming to determine user preferences of the user (33), d) depending on the reaction (23) of the user (33) to the indication (11): determining the user preferences, e) creating a user profile based on the determined user preferences, f) deriving settings (38) of the infotainment system (35) from the determined user preferences and applying these settings (38). 2. Method according to claim 1, wherein the indication (11) refers to a function which the user (33) has not yet activated by the time (16) of receiving the indication (11) and/or the frequency of use of the user (33) for this function falls below a predetermined threshold. 3. The method according to claim 2, wherein the frequency of use is determined by means of a history (4), wherein the histone (4) quantifies which functions of the motor vehicle (32) and/or how often these have already been activated in the past while the user (33) was registered as an occupant. 4. Method according to one of the preceding claims, wherein in step a) the contextual trigger (15) comprises: an expected travel time and/or a planned travel destination and/or the current outside temperature (37) and/or the usage profile of the user (33). 5. Method according to one of the preceding claims, wherein in step b) the determination of a time (16) as a function of a driving speed and/or an interior noise of the Motor vehicle (32) and/or a determined number of occupants in the motor vehicle (32) and/or a surrounding traffic density. 6. Method according to one of the preceding claims, wherein step c) is based on an "Active Question Answering" voice dialogue (25), which is initiated by a voice dialogue system (1) and carried out with the user (33). 7. The method according to claim 6, wherein the speech dialogue system (1) comprises an artificial neural network and is trained by means of reinforcement learning on data sets consisting of question-answer pairs to formulate instructions (11) depending on at least one contextual trigger (15) and/or a determined time (16). 8. The method according to claim 6 or 7, wherein steps b) to e) are skipped as soon as the artificial neural network signals that a predetermined threshold value for the contextual trigger (15) is reached. 9. Method according to one of the preceding claims, wherein step c) includes reformulating the question if no User preferences could be determined and/or if a gesture of perplexity of the user (33) is determined, wherein the reformulating comprises asking a question which is positioned directly after the initially formulated question according to an ordered probability list based on results of the voice dialogue. 10. The method according to any one of the preceding claims, wherein the gesture of helplessness comprises detecting by image-based and/or radar-based and/or lidar-based and/or microphone-based user monitoring (6) that the user (33) shrugs his shoulders and/or frowns and/or groans and/or performs a predetermined gesture to request help in response (23) to an indication (11) issued by the infotainment system (35). 11. Method according to one of the preceding claims, wherein at least one of the following is detected as reaction (23): a verbal and/or haptic feedback, a gesture, a period of time passed, a change in user preferences (38) an emotional reaction. 12. Method according to one of the preceding claims, wherein activation or deactivation for deriving at least one setting (38) is controlled in dependence on a manual input. 13. Method according to one of the preceding claims, wherein transparency of the derived settings (38) is ensured at least partially by displaying them in an AI menu of the infotainment system (35). 14. Infotainment system (35) according to one of the preceding claims, wherein the infotainment system (35) comprises a processor device (36) which comprises program instructions which, when executed by the processor device (36), cause it to carry out a method according to one of the preceding method claims. 15. Motor vehicle (32) comprising an infotainment system (35) according to claim 14.