TR2022005022A2 - INTERACTIVE DOCTOR EXAMINATION SYSTEM AND METHOD - Google Patents

Abstract

The invention is related to an interactive doctor examination system and method that provides automatic clinical examination report generation, smart contract-based data sharing, doctor and patient emotion analysis, speech-to-text conversion and blockchain-based secure storage systems.

Description

DESCRIPTION INTERACTIVE DOCTOR EXAMINATION SYSTEM AND METHOD Technological Field: Invention, creation of online interactive meeting platforms for use in the health sector, emotional analysis of doctor-patient interviews, data labeling by transcribing the voice from speech to text during the examination, and the creation of a structured multi-purpose examination report. making medical data accessible based on the patient's consent, preserving privacy with a high-security common cloud platform (SaaS) by using blockchain technology and encryption protocols, developing a unique system where a different doctor can access the interview records with common permission or public decisions, It is about a system and method that makes access records to this content reportable, enables the doctor to provide visual and audio information to the patient during the examination, or to share interactive content by collecting information from the patient. With the system developed within the scope of the invention subject to the description, both doctor, hospital, patient examination and management processes will be completely digitalized, the individual can be observed independently thanks to patient and disease follow-up data storage, and in case the communication and operational problems experienced during the examination process turn into a public lawsuit, all parties and the public will be provided with impartial concrete information. will be able to provide data. Doctors will be provided with the opportunity to review and report through easy retrospective scanning during the patient follow-up process. In addition, it may provide a basis for the stored data to be used as concrete evidence in lawsuits between the patient and the doctor. Within the scope of the invention, there is a module that includes the conversion of speech into text using the audio data of the conversation between the doctor and the patient, the creation of a clinical examination report using the heterogeneous data obtained during this interactive examination, speaker recognition and speaker emotion analysis components. Known Status of the Technology: In recent years, due to the impact of the pandemic in the world, work and meetings have been carried out remotely in most sectors. The epidemic period has proven that many jobs can be done remotely. As a result of the increasing population and developing technology, it has been observed that some health examinations can be performed remotely. The invention subject to the description describes a method and system that enables the examination between patients and doctors to be carried out safely, efficiently and effectively in a virtual environment. There are many video calling solutions available, but recording e-health video connections between patients and doctors causes operational and legal difficulties and does not meet legal requirements. Records of remotely administered treatments contain private patient data and must be processed and stored accordingly. In many European countries, medical data cannot leave the country in question and records must be stored securely in a cloud solution located in that country. Since most providers of video conferencing solutions used today are US-based companies, it has been observed that an external cloud service does not meet the legal requirements. In our country, hospital infrastructures are not suitable and sufficient for recording treatments and archiving them securely. Since the existing solutions were developed for the purpose of online interviews and meetings, they lack the tools that will enable the medical examination processes to be carried out, the doctor to provide an interactive medical service to the patient, and the record tracking with retrospective data labeling method. E-Health and Telemedicine procedures are treatment methods recognized by European health legislation. With the issue of transparency gaining importance in medical treatments, it is aimed to increase the quality of services offered to patients. In this process; Local treatments and treatments with remote imaging must be recorded. Records are also important from a legal perspective. Patients who are not satisfied with the treatment and complain can take legal action. In such cases, medical records can be used as evidence in lawsuits. The Council of Europe has been working on the protection of personal data since the 1970s and considers individuals' health data as privacy. Accordingly, most countries are planning to make it mandatory to record all treatments performed in hospitals or remotely. Therefore, it is expected that the legal changes to be made in Europe and in our country in the near future will accept the records as legal evidence and make it mandatory to archive these records in hospitals and health centers for decades. However, the current infrastructures of today's hospitals are not suitable and sufficient for recording all treatments and archiving them securely. There are a wide variety of services for speech-to-text conversion. The main ones are "Google Web Speech API" developed by Google, "Watson" created by IBM and "Azure Speech Services" created by Microsoft. Python's "SpeechRecognition" library supports different APls for speech recognition and voice-to-text conversion, including Google, IBM and Microsoft Services. They also have the ability to adapt to a different voice accent or sound pattern. Although significant progress has been made in the languages of developed countries in terms of automatic speech recognition systems, this is not yet valid for the Turkish language. The most important reason for this is the difficulties in creating data and the lack of sufficient research to process this data specifically for the relevant field. Most automatic speech recognition systems rely on artificial neural networks-based hybrid hidden Markov models (HIVIM) to provide acoustic and language models. A dynamic programming decoder, however, uses these probabilities to find and extract the most likely sequence of words for a given input audio signal. In the light of the field research and analyzes carried out during the design phase of the invention, the following user needs were identified that will form the basis for the invention: Patients have difficulties in accessing doctors and health services due to reasons such as geographical conditions, physical disabilities of the individual and economic inadequacy, Patients' difficulty in accessing doctors and health services for various reasons (working individuals not being able to spare the necessary time). , problems in transportation of disabled and elderly people to hospitals, fear of catching diseases from hospitals, etc.), refraining from going to health institutions, delaying their treatment or not receiving treatment at all, and the resulting bigger health problems, Recently, due to the effect of the pandemic, video conferencing software used for remote doctor-patient meetings. They are not designed with health data security in mind, no interactive medical communication can be established between the doctor and the patient, the interview is insufficient to provide health care, the privacy of health data is not ensured, public lawsuits between the doctor and the patient are conducted on the basis of oral statements and there is no concrete supporting document, the doctor and the patient Privacy methods are open to improvement within the scope of protecting patient data in accordance with the Personal Data Protection Law (KVKK) and the European Union Personal Data Protection (GDPR) standards. In the literature research, various platforms provided by Mayo Clinic Digital Health Care, SteadyMD, LiveHealth Online, Myhealth Clinic and Vidyo Telemedicine companies were encountered. Although the platforms provide remote patient monitoring, video appointments, secure communication, online patient services, they do not include automatic clinical examination report creation, smart contract-based data sharing, doctor and patient emotion analysis, speech-to-text conversion and blockchain-based secure storage systems. As a result, there is a need for an interactive doctor examination system and method that exceeds the known state of the technique and eliminates its disadvantages. Brief Description of the Invention: The invention is an interactive doctor examination system and method that exceeds the known state of the technique, eliminates its disadvantages, and includes additional advantages. The purpose of the invention is to introduce an interactive doctor examination system and method that provides automatic clinical examination report generation, smart contract-based data sharing, doctor and patient emotion analysis, speech-to-text conversion and blockchain-based secure storage systems. Within the scope of the interactive examination system and method subject to the invention, there are two main solutions and many sub-solutions under these main solutions. The first main solution provided by the invention is to extract multi-purpose reports and information from the data received from the doctor-patient interview with speech-to-text conversion techniques, and the second main solution is a structure in which cryptology techniques, distributed file systems, blockchain technology, smart contracts and zero knowledge proof algorithms are used to achieve security and reliability goals. is to reveal. First Main Solution and Sub-Solutions: Within the scope of the invention, developing a text to text conversion application as a basis for developing an interactive doctor examination system by customizing it according to doctor-patient conversation data, creating a structured and summarized clinical examination report from the speech-text conversion file, emotional analysis between the doctor and the patient. It is ensured that information is performed and information that will contribute to the patient's health literacy is extracted. According to various studies conducted around the world, the time that doctors can devote to direct interaction with patients constitutes less than 13% of their days. It was also observed that the average time per week patients spent writing, preparing, or editing clinical documents in different clinical settings was approximately 12 hours for doctors, approximately 11 hours for nurses, and approximately 11 hours for therapists. Validation of transformer language models and improved training for cross-sentence modeling for the development of an existing application for converting the audio data received from the video conversation between the doctor and the patient into text, in terms of automatic speech recognition for the interactive doctor examination system targeted in the invention, by customizing it according to the data in the doctor-patient conversation. Studies are being carried out to implement the and attention-based language model on real data. In the invention, variable length contexts are processed with LSTM-RNN and Transformer language models, and these models are trained in different sequence length ranges created by combining multiple sentences and a hybrid structure is achieved. With the invention, a model that is trained to perform well on single-domain test data focused especially on conversations between doctor and patient is obtained. Although the conversion accuracy in terms of the percentage of similarity in the speech-to-text conversion process is generally independent of gender, using data from different genders in the data preparation phase to guarantee the accuracy of the system to be developed within the scope of the study, although real-time speech-to-text conversion is not targeted, training of the instructor is important in terms of increasing word accuracy. The aims of the invention are to construct the structure accordingly and to create a data set that includes different areas of expertise, especially for the interactive doctor examination system. For these purposes, within the scope of the method, the Sequence-to-sequence attention model will consist of an encoder, an attention mechanism and a decoder, both forward and reverse information will be combined with bidirectional LSTM-RNNs, and as an acoustic model, the main signal of the encoder will be separated from environmental noise. The attention mechanism will be learned more easily, a data set will be created for the recognition of doctor-patient speech, designs will be made with real word scenarios, a hybrid approach that is more robust against noisy data and does not require as much data cleaning as possible will be taken, based on doctor-patient interviews. A customized speech-to-text conversion module will be developed. Although speaker recognition based on the detection of changes and features in the voice is a long-known study, an emotion analysis model will be developed to detect the emotions of speakers by using the speech received as a voice signal together with the text file obtained by speech-text conversion. Thus, the emotions of both the patient and especially the doctor in their relationship with the patient can be monitored from the conversation between the doctor and the patient. This situation can be reported by detecting the continuity of positive or negative emotions between a doctor's relationships with all his patients, especially in large environments such as hospitals. Although the emotions of the parties will be basically classified as positive, negative and neutral within the scope of the emotion analysis part, within the scope of the invention, it will be tried to support the doctor or patient's approaches containing violence, insults and anger by removing certain features in the voice data. Second Main Solution and Sub-Solutions: In the invention, a structure is developed in which cryptology techniques, distributed file systems, blockchain technology, smart contracts and zero knowledge proof algorithms are used to achieve security and reliability goals. The patient and the doctor must give consent for the interview report and video recording. This will happen through smart contracts. Anonymization methods will be used for text data to avoid personal data in the records. Confirmation of personal data will occur through zero-knowledge proof algorithms. Session records will be encrypted (symmetrically) using the session key. Session recordings will be saved on the cloud. In these processes, it is aimed to create a record using the hash value of the file and IPFS technology, which guarantees the immutability of the file. The URL address information and session key, which will show the immutability of this record, will be recorded in the blockchain and will also be delivered to the patient. In order to securely deliver the session key to the patient, the patient's public key can be used, or secure data transmission can be achieved through the institution's certificates. In addition to the E-health applications and patient tracking systems developed so far in order to achieve the aims of the invention; Examination video recordings can be taken, when the content recorded with the speech to text method is searched with the relevant keyword within the records, you can be directed to the relevant time, the doctor can provide an effective health service by sharing interactive content with the patient during the interviews, the records can be stored in a distributed file system (lnterPlanetary File System - A system that ensures privacy and consent-based access will be developed by using cloud storage (IPFS), blockchain technology, zero knowledge proof algorithms and encryption protocols. In this way, traceability and high level of data protection during examinations will be made possible. Explanation of Drawings: The invention will be explained by referring to the attached figures, so that the features of the invention will be understood more clearly. However, this is not intended to limit the invention to these particular embodiments. On the contrary, it is intended to cover all alternatives, modifications and equivalences that can be included within the scope of the invention as defined by the attached claims. It should be understood that the details shown are for the sole purpose of illustrating preferred embodiments of the present invention and are presented for the purpose of providing the most useful and easily understandable description of both the embodiment of the methods and the rules and conceptual features of the invention. In these drawings; Figure-1 is the representative view of the interactive doctor examination system that is the subject of the invention. The figures that will help understand this invention are numbered as indicated in the attached picture and are given below with their names. Disclosure of References: . Video conferencing and audio recording platforms 11. Patient 12. Doctor 13. Media server. Consent based data sharing system. Secure storage service 31 . Storage space 32. Institution certificate 40. Conversion of voice to text and indexing module 41. Media recording file 42. Voice recording 43. Heterogeneous data 44. Pre-processing 45. Speech-to-text conversion software 46. Customization module 47.8server 48. Indexing and speech-to-text text conversion file 50.Analysis and reporting module 51. Algorithms 52. Analysis and report outputs Disclosure of the Invention: In this detailed explanation, the interactive examination system and method of the invention are explained only for a better understanding of the subject, with examples that will not create any limiting effect. The specification describes an interactive examination system and method that allows patients (11) to be examined by doctors (12) with remote access. In the invention subject to the description, audio and video transmission to the interactive examination of the doctor (12) and the patient (11) is achieved through video conferencing and audio recording platforms (10) consisting of the patient's (11) smart device, the doctor's (12) smart device and a media server (13). is provided. WebRTC technology will be used in video conferencing and audio recording platforms (10). A media server (13) (SFU media server unit) is used to manage the audio and video traffic in the video call between the patient and the doctor, and the traffic with this server is encrypted with SSL. In the invention, consent-based data sharing systems (20) are used. The patient (11) and the doctor (12) must give consent for the interview report and video recording. This is achieved through smart contracts. Anonymization methods are used for text data to avoid personal data in the records. Confirmation of personal data takes place through zero-knowledge proof algorithms. Doctor-patient consent is required to access these records later. The conditions for this are defined in the smart agreement according to the relevant protocol. When these conditions are met, the smart contract can transmit the URL information that provides access to the relevant record and the session key used in symmetric encryption; The aim of the invention is to provide preview on the web using this information. The invention includes a secure storage service (30) through which secure data sharing of lPFS storage areas (31) is made through institution certificates (32). lPFS and cryptology are used as distributed file systems in the secure storage service (30). Session recordings are saved on cloud storage areas (31). It is aimed to create a record on lPFS, especially for reports and analysis documents. The URL address information and session key, which will show the immutability of this record, are recorded in the blockchain and are also delivered to the patient. (via email or user account). In order for the session key to be delivered to the patient securely, the patient must have a public key. In case there is no such key; Secure transmission can be ensured through the institution's certificates (32). The interactive system subject to the invention includes a voice-to-text conversion and indexing module (40). This module (40) takes the examination media recording file (41) between the patient (11) and the doctor (12), combines the voice recording (42) in it with the heterogeneous data (43) known before the examination, pre-processes it (44) and creates a conversation. It converts it to text with text conversion software (45). Speech-to-text conversion basically consists of statistical models that enable the mapping of audio sequences into recognizable text output in the relevant language. Therefore, automatic speech recognition systems include a language model, a pronunciation model and an acoustic model. Once models are appropriately trained on speech data from multiple speakers and language models with expanded vocabularies, the accuracy of speech-to-text conversion increases. Statistically, the performance or accuracy of speech-to-text conversion is generally measured by the word error rate. The most important consideration when training a speech recognition model is the quality of the dataset, voice, and dictionary. Especially during the speech-to-text conversion of doctor-patient conversations, the following basic stages can be followed: o The sound of the doctor-patient conversation is recorded from the interactive doctor examination system, recorded on the server and uploaded to the relevant platform. In the speech recognition (speech-to-text conversion) engine, the doctor analyzes the acoustic model sounds improved with patient voice data, - The dictionary model improved with the converted text data matches the sounds with the correct words, o In the customization module (46) the results are configured with the special language model and all words are given a confidence score, A raw text file with the user ID and timestamp is obtained. The invention includes an analysis and reporting module (50) that automatically generates analyzes and reports after the meeting. In this module (50), the indexing and speech-text conversion file (48) obtained by indexing studies is passed through algorithms (51) to produce analysis and reports such as information extraction, text summarization, patient health literacy report, clinical examination reports in different formats, doctor-patient emotion analysis. (52) is prepared as output. In the invention, a video conferencing platform will be created for the patient and the doctor to have an interactive online conversation. A server (SFU media server unit) will be used to manage the audio and video traffic in the video call, and the traffic with this server will be encrypted with SSL. Video conference calls can include doctors (12), patients (11), doctor's assistants and similar participants, and the IP camera image at the end of a medical device, in addition to computer and phone cameras, can also be included in the conversation over the network with the RTSP protocol. For example; While examining a bedridden patient at home, an assistant doctor will be able to show images of the patient's mouth and throat online to his professor at the conference with a simple endoscopic device. Here, it is possible to include a second non-USB camera without the need for a medical device. In the consent-based data sharing system (20) of the invention, when more than two participants participate in video conference calls, for example; When a third person, such as a doctor's assistant, needs to be involved in the interview, the doctor authorized as the moderator (12) must obtain approval from the patient (11) who is the participant. In this platform, even if (1) patient's consent is given, the doctor's assistant and the patient will be prevented from seeing each other during the meeting. The consent of the doctor (12) and the patient (11) will be required to access these records later. The conditions for this will be defined in the smart agreement according to the relevant protocol. When these conditions are met, the smart contract can transmit the URL information that will provide access to the relevant record and the session key used in symmetric encryption; The aim of the invention is to provide preview on the web using this information. After the meeting between the doctor (12) and the patient (11), the conversations will be converted into text. Speech-to-text conversion basically consists of statistical models that enable the mapping of audio sequences into recognizable text output in the relevant language. Therefore, automatic speech recognition systems include a language model, a pronunciation model and an acoustic model. Once models are appropriately trained on speech data from multiple speakers and language models with expanded vocabularies, the accuracy of speech-to-text conversion increases. Statistically, the performance or accuracy of speech-to-text conversion is generally measured by the Word Error Rate. The most important point when training a speech recognition model is the quality of the dataset, voice and dictionary. Due to local dialect differences and the difficulty in pronouncing medical words, both the doctor and the patient are asked after the interview, "Is this what you meant here?" It is aimed to increase the accuracy rate in converting voice to text by removing confirmation screens such as: In order to index the words on the text by matching them with the voice data, the sound recording of the doctor-patient conversation is taken from the interactive doctor examination system, recorded on the server and uploaded to the relevant platform. In the speech recognition (speech-to-text conversion) engine, the doctor analyzes the acoustic model sounds improved with the patient voice data, and improves the sound with the converted text data. The dictionary model matches the sounds with the correct words, the results are structured with the custom language model and a raw text file is obtained where all words have a confidence score, user ID and timestamp. What is meant by indexing here is that when the word migraine is mentioned in the doctor-patient examination conversation, when one of the parties wants to listen to this video recording again and only needs the part where migraine is mentioned, they will be able to quickly access the parts of the recording where this word is mentioned by searching for the word migraine. In the interactive doctor examination system, after converting the voices of the doctor (12) and the patient (11) conversation into text, classification, summarization and so on are performed on the resulting text, speech-text conversion, information summary extraction, information extraction, creation of clinical examination reports, and also the patient or doctor's Sentiment analysis will be provided. It is aimed to provide clues to the doctor by analyzing some special medical words in the text. For example; Expressions such as "headache", "nausea", "dizziness" uttered by the patient during the interview will be classified and clue expressions such as the definition of pain will be shown to the doctor. After the interview, the words frequently used by the patient or the doctor will be classified as positive, negative or neutral in order to conduct an emotional analysis and it will be determined whether the person is angry or stressed at that moment. At the same time, information about the emotional state will be obtained by measuring the decibel level of the sound by obtaining amplitude and frequency information. TR TR TR